Jenkinsfile - Skip Release

If you trigger your Jenkins builds after every commit in git, then after a release, which does a git commit with the new version number, Jenkins will pickup the commit and release the project again, which of course leads to another Jenkins build/release cycle ... infinity. To work around this you have to have a 'when' expression to your Jenkins file to verify that the last commit was not due to a release and also that it contains files from your directory so it doesn't perform a release again (see the stage('Release') below). Initially, I thought this was an issue with the release plugin so I raised an issue on github. Here's how I worked it out Here's what the git log looks like immediately after a release:

$ git log --oneline -n 2
e45e8db (HEAD -> dragon, origin/dragon, origin/HEAD) [Gradle Release Plugin] - new version commit:  'utils-0.0.15.APTNG-SNAPSHOT'.
a3d19fc (tag: utils-0.0.14.APTNG) [Gradle Release Plugin] - pre tag commit:  'utils-0.0.14.APTNG'.

This is the Jenkinsfile I used to reason if a release is required.

#!/usr/bin/env groovy

def dir='services/common/utils'
def version
// Only update the description with the release version number if it performs a release
def isRelease = false

pipeline {
    agent {
        label 'build_slave_lable'
    }
    stages {
        stage('Compile') {
            steps {
                script {
                    // Added this to update the build description with the version number from the gradle.properties
                    version = sh(
                            script: "cd $dir && grep version gradle.properties | sed 's/version=//g'",
                            returnStdout: true
                    ).trim()
                    currentBuild.description = "$version"
                }
                sh "cd $dir && ./gradlew assemble
            }
        }
        stage('Unit Test') {
            steps {
                sh "cd $dir && ./gradlew test
            }
        }
        stage('Release') {
            when {
                expression {
                    def branchName = sh(script: 'git rev-parse --abbrev-ref HEAD', returnStdout: true).trim()
                    // # GIT log command will only print out the file paths from the last commit
                    // # See https://stackoverflow.com/a/17583922/529256
                    // # Example:
                    // $ git log -n 1 --name-only --pretty=format:
                    // services/common/utils/gradle.properties
                    def lastCommitFiles = sh(script: 'git log -n 1 --name-only --pretty=format:',
                                             returnStdout: true).trim()
                    def hasFiles = lastCommitFiles.contains(dir)
                    // Verify that the last commit was not a release of this module
                    def lastCommitMessage = sh(script: 'git log -n 1 --oneline', returnStdout: true).trim()
                    def wasNotRelease = !lastCommitMessage.contains("[Gradle Release Plugin]")
                    // Release only if:
                    // - This is the 'release' branch
                    // - AND last commit has files from this directory
                    // - AND last commit was a not a release of this module
                    return branchName == 'dragon' && hasFiles && wasNotRelease
                }
            }
            steps {
                sh "cd services/common/utils && ./gradlew release -Prelease.useAutomaticVersion=true"
                script {
                    isRelease = true
                }
            }
        }
    }
    post {
        success {
            script {
                if (isRelease) {
                    // Update the build description to the release version number if the release was performed
                    def description = "$version".replaceAll("-SNAPSHOT", "")
                    currentBuild.description = description
                }
            }
        }
    }
}

Cleaning up with SVN

So I had this great idea the other day when I was working with svn to remove all unversioned files from my working directory. Kind of like make clean, but without having to specifically list the files you want to keep in the Makefile. The way I did it is using the svn status command as follows:

svn status | grep ^? | cut -c8- | xargs rm -rf
svn status --no-ignore | grep ^I | cut -c8- | xargs rm -rf 

The first line removes all unversioned files, but it misses all files which svn ignores (back up files or flies that start with '.'). The second line takes care of the files svn ignores. This is a pretty dangerous command to use if you haven't added any of your newly created source files. So that I don't do that I added this to a script which has a confirmation dialog. This will list all the files that will be deleted and then asks you to confirm you want to delete them.

#!/bin/bash
true=0;
false=1;
yesNo=$false;
confirmYesNo() {
  validAnswer=$false
  until [ $validAnswer == $true ]; do
    echo -n "$1";
    read -a yesNo;
    if [ -z "$yesNo" ]; then
      validAnswer=$false;
    else
      yesNo=`echo $yesNo | tr [:upper:] [:lower:]`;
      yesNo=`expr substr $yesNo 1 1`;
      if [ $yesNo = y ] || [ $yesNo = n ]; then
        validAnswer=$true;
      else
        echo "  '$yesNo' is not an option";
        echo;
        validAnswer=$false;
      fi
    fi
  done
}

###############
# Main Method #
###############
files=`svn status | grep ^? | cut -c8-`;
files="$files`svn status --no-ignore | grep ^I | cut -c8-`";
if [ -z "$files" ]; then 
  echo "No unversioned files, exiting....";
else 
  echo "Warning! This will erase the following files:"
  echo;
  svn status | grep ^?
  svn status --no-ignore | grep ^I
  echo;
  confirmYesNo "Are you sure? [y] or [n]: "
  if [ $yesNo == y ]; then
    svn status | grep ^? | awk '{print $2}' | xargs rm -rf
    svn status --no-ignore | grep ^I | awk '{print $2}' | xargs rm -rf
    echo "Done!"
  fi
fi

FYI, this will delete folders as well as files. Its particularly useful when your using programs like Xilinx's ISE or Altera's Quartus II synthesis tools, which generate a lot of flies when you use them.

P.S. If you like the nifty syntax highlighting that I used in this post, then see Alex Gorbatchev's SyntaxHighlighter page.
P.S.S. If you want to use it with blogger then see this nice guide "easy syntax highlighting for blogger" on Carter Cole's blog.

Creating .pdf and .eps files from xfig drawings in batch

So I've been using xfig a lot recently, and I thought I would share a little of what I've discovered. In case you didn't know, xfig is a simple drawing utility that comes with most linux distributions. Its open-source so you can also get ports to Mac and Windows (I prefer to use the cygwin version of it).

As you can see its pretty "old school", but its perfect for making simple figures. If you want to make vector graphics, then I'd suggest something like Inkscape. The only problem I've ever had with xfig is creating .pdf or .eps from the command line. I've figured out how to do that using the fig2dev program, which comes with xfig. The man page for it is pretty complete, but here's a small example of how to create both a .pdf and .eps file.

For .pdf you do:

$ fig2dev -L pdftex -p portrait fig/figure1.fig > fig/figure1.pdf

Where -L pdftex, tells fig2dev to create a latex compatible pdf drawing in portrait mode. If you prefer landscape mode, then just remove the -p portrait option.

For a .eps you do:

$ fig2dev -L eps fig/figure1.fig > fig/figure1.eps

Pretty straight forward what everything means, but it should be pointed out that .eps files and .pdf files behave differently regarding orientation. Without the -p portrait parameter, the .pdf generated file is rotated 90 degrees compared with the .eps version.

---

In order to use this with my LaTeX files, I created a make file that first checks to see if the .fig files have been updated, and then generates new .pdf or .eps files depending on the type of paper (.ps or .pdf) I'm generating. Here's a copy of it:

$ cat Makefile
PAPER=file_name

BIB_DB=${PAPER}.bib
TEX=latex

BIBTEX=bibtex

FIG=fig/figure.fig \
fig/figure1.fig \
fig/figure2.fig \
fig/figure3.fig

all: ${PAPER}.ps

${PAPER}.ps: ${PAPER}.tex ${PAPER}.bbl ${FIG:%.fig=%.eps}
        ${TEX} ${PAPER}.tex
        ${TEX} ${PAPER}.tex
        dvips -o ${PAPER}.ps ${PAPER}.dvi

${PAPER}.pdf : TEX = pdflatex
${PAPER}.pdf : ${PAPER}.tex ${PAPER}.bbl ${FIG:%.fig=%.pdf}
        ${TEX} ${PAPER}.tex
        ${TEX} ${PAPER}.tex

${PAPER}.bbl : ${PAPER}.aux ${BIB_STYLE} ${BIB_ABBR} ${BIB_DB} ${PAPER}.tex
        ${BIBTEX} ${PAPER}

${PAPER}.aux : ${PAPER}.tex ${FIG:%.fig=%.pdf}
        ${TEX} ${PAPER}.tex

fig/%.pdf: fig/%.fig
        fig2dev -L pdftex -p portrait $^ > $@

fig/%.eps: fig/%.fig
        fig2dev -L eps $^ > $@

pdf: ${PAPER}.pdf

clean: distclean
        rm -rf ${PAPER}.pdf ${PAPER}.ps ${PAPER}.dvi ${FIG:%.fig=%.eps} ${FIG:%.fig=%.pdf}

distclean:
        rm -rf *~ *.aux *.bbl *.blg *.log *.mpx mpx*.dvi mpxerr.tex thumb* *.core *.out ${PAPER}-dvi.dvi ${PAPER}.[0-9]* \
        ${PAPER}-dvi.tex ${PDFMPFIGURES}

The lines to pay attention to are the:

${PAPER}.ps: ${PAPER}.tex ${PAPER}.bbl ${FIG:%.fig=%.eps}

and the

${PAPER}.pdf : ${PAPER}.tex ${PAPER}.bbl ${FIG:%.fig=%.pdf}

Where I've added the dependency ${FIG:%.fig=%.pdf} which expends the list of figures and replaces .fig with .pdf for each figures name.

The generic targets:

fig/%.pdf: fig/%.fig

and

fig/%.eps: fig/%.fig

To create a .pdf or .eps file in the fig directory there is a .fig file of the same name. The command is executed on the $^ variable, which stands for the name of the dependency (fig/%.fig), and the $@ variable, which stands for the target (fig/%.pdf or fig/%.eps)

find and execute

So I was trying to remove all the old backups created by emacs, and I knew there was a way to do this using the find command. If you want to just list all the backup files, then you simply type:

$ find . -name "*~"
./example.java~
./build/test/define.c~

and it will list all your backup files, or files that end in a '~'. Now if you want find to execute a command on each file that it finds, you have to use the -exec command. Here's a clip from the man page for find.

-exec command ;
       Execute command; true if 0 status is returned.  All
       following arguments to find are taken to be
       arguments to the command until an argument
       consisting of ‘;’ is encountered.

This is great, but from the description it seems like this is what you need to do:

$ find . -name "*~" -exec echo ;

I expect that this will simply print each file's name on standard output. However, it doesn't instead it returns:

$ find . -name "*~" -exec echo ;
find: missing argument to `-exec'

In order to get it to work the way you expect, you have to escape the semi-colon by adding a '\' in front of it. Here's the output you get:

$ find . -name "*~" -exec echo \;

All this does is print a blank line on standard output. If you want it to print the file's name, then you have to use '{}' braces and the file name will be inserted for each invocation of the command. Here is the final version, with two exec commands, the first prints out what we're going to do and the second executes the command.

$ find . -name "*~" -exec echo "rm {}" \; -exec rm {} \;
rm ./libpath.sh~
rm ./classpath.sh~
rm ./test/Makefile~

Another way to do this is to use a for loop like this:

$ for fileName in $( find . -name "*~" ); do \
  echo "rm $fileName"; \
  rm $fileName; \
done

I think this approach is a little more flexible if you have more complicated commands. You can also pipe the output of find to other commands in order to filter the results. For example:

$ for fileName in $( find . -name "*~" | grep -v java ); do \
  echo "rm $fileName"; \
  rm $fileName; \
done

This will exclude all files that have the word java in their name.

Hello

Well I thought I would create a new blog about my trials and tribulations while programming. I had thought about creating a project on some topic and hosting in on a site like Google Code, or Source Forge, or trying out GitHub. However, creating a new project is hard. The central question is: What do you want to do a project/application on? Once that's decided, you have to do some concepts and some coding. I mean you don't want your project to be just an empty shell with no code. You have to create a skeleton to work on before you get started. Don't you? So then there's the question of: When should you put your code up? and How much time do I want to devote to it once I do? and Will people find it useful? or Has it been done before? and What if they laugh at me? I have a few ideas for a project, but thats all they are right now. Ideas, and maybe I'll blog a bit about those too. Most of the work I do right now is for my PhD so my code is neatly locked away on my schools svn server. Of course most of that work is not very "useful" to anyone except me and my small corner of the scientific community. So I've decided that this blog will be a place where I can post small examples of particularly difficult problems I've overcome. I'd also like to put comments about simple things that aren't well documented, which others might find useful (by "others" I mean you the reader and myself too!). Well I'll try to think up a good topic for my next entry. Cheers!