CI/CD Pipeline with Github-SBT-TravisCI-Codacy-Nexus/Artifactory

This is a “HelloWorld CI Pipeline”!

This blog describes the steps taken to establish a CI / CD pipeline using SBT, Travis CI, Codacy and Nexus or Artifactory.

Github Private Repository

I created a HellowWorld Scala project in a github private repository. (You can create a public repo)

It is recommended that you set your github account for Two Factor Authentication.

SBT

For those who are new to Scala and SBT, here is some information. You need to create a build.sbt. Similar to ant, maven or any other build tool you can have “tasks” that sbt will perform. Eg – clean, compile, test, publish.

Publish is what we want to do. Publish where? If we want to publish the build to the Nexus Repository, this is how the build.sbt will look like:

name := "hello-scala"
version := "1.4"
organization := "fakepath.test"
scalaVersion := "2.11.1"
libraryDependencies += "org.scalatest" %% "scalatest" % "2.1.6" % "test"
publishTo := {
val nexus = "https://" + System.getenv("NEXUS_IP") + ":8081/nexus/"
if (isSnapshot.value)
Some("snapshots" at nexus + "content/repositories/snapshots")
  else
Some("releases"  at nexus + "content/repositories/releases")

System.getenv("NEXUS_USER"), System.getenv("NEXUS_PASSWORD"))

Note the publishTo section. The nexus IP, User and password are not pushed to the repo and are taken from the environment variables.

Travis CI

I have used the hosted Travis CI. For private repositories Travis CI has a SaaS offering at – https://magnum.travis-ci.com.

Travis docs suggest that they ensure our builds and artifacts from the private repo are secure and the space is not shared by any other application. Their security statement: https://billing.travis-ci.com/pages/security

Signup up with Travis CI using your github account. To start with, a webhook needs to be activated for your private repo, manually, as a one time task. Then a .travis.yml file needs to be created. There are lot of things you can do by scripting the .travis.yml file properly.

The .travis.yml would be added to your repo. Our file looks as below:

language: scala
jdk: oraclejdk7
sudo: false
scala:
- 2.11.1
before_install: umask 0022
script:
- sbt clean compile test
- sbt publish
notifications:
  slack:
    secure:
    rooms:
      secure:
email:
env:
  global:
  - secure:
  - secure:
  - secure:

The environment variables are added to this file by encrypting them. The enviroment variables used in this case are: NEXUS_IP, NEXUS_USER, NEXUS_PASSWORD. Notifications can also be set. The notifications set in this case are slack notifications to a single person or to a channel.

You can encrypt this info as:

travis encrypt NEXUS_USER=someuser --add

Note – that we run the “sbt publish” from Travis CI. This will generate an artifact. For a scala project, the artifact is a jar file. The artifact can also be a docker image. When “sbt publish” is run by Travis CI, it will use the build.sbt’s publishTo description to publish the artifact to the Nexus repository.

Note – We can (and we should) have a task like “sbt promote”, or we can write scripts in Travis CI itself which will “promote” a build. We would want to publish every build to Nexus. But we will not want to “promote” every build. The promoted build is deployed to production. This is typically a manual step with what is known as “one-click” deployment. This can be completely automated too, depending on the project. Infact, thats the difference between “Continuous Deployment ” and “Continuous Delivery”.

Note – We can read the latest version number of the build and then increment the number of the next build. This can be done programmatically by scripts.

Codacy

Codacy is a code review and analysis tool. You can signup with codacy.com using your github account and enabling the webhooks as documented by them.

Codacy also has support for private repository. You can play with the UI to see and set different metrics. However, to automate, you will need some scripts. Eg – if we want to ensure we do not build if code coverage metrics are less than 95% then these settings need to be done in codacy. To enable codacy for a private repo, again there are some web hooks. We can have codacy analyse the code per commit, per PR, per branch etc.

Nexus

I have made use of Nexus to act as an “Artifact Manager”. So Nexus would be the tool which will store all the builds. Each build will be numbered and we will also have a build named “latest” which will be like a pointer to the latest promoted build. When a decision is made to promote a build, Travis CI will publish a build to Nexus & will also have a script that will update the “latest” pointer to point to this build.

Alternatives to Nexus

There are couple of repository management solutions available. Some offer hosted services.

  1. Artifactory http://www.jfrog.com/open-source/#os-arti
  2. Bintray http://www.jfrog.com/open-source/#os-bin

Configuration Management / Deployment

I have written a simple bash script which downloads the artifact from Nexus repo or Artifactory and simply executes it. We can run this script as a cron job.

Here is the script for fetching newly published artifacts from Artifactory repo:

#!/usr/bin

# Artifactory location
server=http://myorg.artifactoryonline.com//myrepo
repo=plugins-releases-local

# Maven artifact location
name=hello-scala_2.11
artifact=hello-scala
path=$server/$repo/$artifact/$name
version=`curl -s $path/maven-metadata.xml | grep latest | sed "s/.*<latest>\([^<]*\)<\/latest>.*/\1/"`
jar=$name-$version.jar
url=$path/$version/$jar

#check if jar file exists
echo $jar
if [ ! -f $jar ]; then
 echo "Downloading new artifact"
 wget -q -N $url
 echo "Executing Jar: " `date` >> hello-scala.log
 scala $jar >> hello-scala.log
fi
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s