Chef Push Jobs

What are Push Jobs?

Push Jobs work like knife-ssh. Almost. Almost because, in knife-ssh the changes are pushed from your workstation using the SSH protocol. In push jobs, the changes are pushed to the node by the Chef Server.

Chef is based on the “pull” and has a reason for that – to keep the server “thin”. But the changing challenges demand that there is a need for a push model. So chef has introduced push jobs by keeping the server is thin!

“Chef push jobs is an extension of the Chef server that allows jobs to be run against nodes independently of a chef-client run” – that’s how push jobs are defined. A job, in this context, is a set of commands that need to be run on the target node.

Difference between Push Jobs and knife-ssh

Push Jobs knife-ssh

Use message bus (zeromq)

Parallel ssh

Claims to attack the scalability issue

SSH Protocol is slow and CPU hungry at scale

Deployment status is relayed back

Feedback on deployment status is not as easy

Newly introduced

Been in the market for long

Complex at the moment, ready with just the basic foundation

Easy to use

Configuring Chef Push Jobs Server

You need either Enterprise Chef or Chef Server 12. It relies on the ACL system that was open sourced with Chef Sever 12. Also, the install command was introduced with Chef Server 12.

Push Jobs does not work with Open Source Chef Server 11. 

Can be setup as standalone or as HA

Run the following commands on Chef Server:

chef-server-ctl install opscode-push-jobs-server
opscode-push-jobs-server-ctl reconfigure
chef-server-ctl reconfigure

Setup Workstation

  • Install knife push plugin
    Gem install knife-jobs
  • Download push-jobs cookbook
    Push jobs cookbook would be used. So download it from the site or git clone the cookbook. You would have to fetch its dependency cookbooks as well.
    knife cookbook site download push-jobs
  • Extract and save the cookbook to your cookbook path
  • Edit the attributes file (push-jobs/attributes/default.rb)
    Update the attributes to add the push jobs package URL and checksum as mentioned.
    default[‘push_jobs’][‘package_url’] = ‘https://opscode-private-chef.s3.amazonaws.com/ubuntu/12.04/x86_64/opscode-push-jobs-client_1.1.5-1_amd64.deb’

    default[‘push_jobs’][‘package_checksum’] = ‘d659c06c72397ed2bc6cd88488349857f1958538‘

  • Upload the push-jobs cookbook to your ChefServer

Create Groups

Create the pushy_job_writers and pushy_job_readers on the organization of the Chef server and add your workstation user to that group.

Setup Node

Simply run the chef client with the recipe:

sudo chef-client –r “recipe[push-jobs]”

Run the knife node status commands to check the node status. It will just show the status “available” at this stage which confirms that the node is prepared for push events.

knife node status
knife node status <node-name>

Run Push Jobs

Run chef-client as:

knife job start ‘chef-client –r recipe[git]’ <node-name>

Run your commands/script as:

knife job start ‘my_script.sh’ <my_node>

Advertisements

DevOps (Part 2): Continuous Integration

Code Review, Build and Test can be automated to achieve Continuous Integration.

Code Review Tools

Every project’s repository is usually managed by some versioning tool. A choice for the versioning tool can be made, we can assume Git for now since its most popular. When the developer pushes his change, a build would be triggered. If the build is successful, then test job would be triggered. Its only after the tests pass, that this commit should be merged to the central repository. Typically the developer’s commit would also need manual review.

As a design principle for DevOps, the central repo should not have push/write access. He will commit to an “authoritative repository”.

Key Considerations

  • A decision from DevOps perspective must be made to choose the right versioning tool, managing of user’s commits to an authoritative repo and a code review tool. This will depend on the project’s requirement. Popular tools to be evaluated are Git, Gerrit, Teamcity.
  • A DevOps engineer will have to setup and configure the tools. Eg – Git-Gerrit integration needs to be installed, setup, configured

Eg: Gerrit

Gerrit is a web-based code review tool built on top of the git version control system. It is intended to provide a lightweight framework for reviewing every commit before it is accepted into the code base. Changes are uploaded to Gerrit but don’t actually become a part of the project until they’ve been reviewed and accepted. In many ways this is simply tooling to support the standard open source process of submitting patches which are then reviewed by the project members before being applied to the code base. However Gerrit goes a step further making it simple for all committers on a project to ensure that changes are checked over before they’re actually applied.

Gerrit can be integrated with several Build Automation tools like Jenkins. It can also be integrated with Issue Tracking systems like RedMine. Eg: When a user commits his change for a bug #123 in RedMine, the bug in RedMine will get updated.

More Reads

What is Gerrit? https://review.openstack.org/Documentation/intro-quick.html

Git-Gerrit configuration: http://www.vogella.com/tutorials/Gerrit/article.html

Implementing Gitflow with TeamForge and Gerrit – http://blogs.collab.net/teamforge/implementing-gitflow-with-teamforge-and-gerrit-part-i-configuration-and-usage

Build Automation

Generically Build Automation is referred to as writing scripts to automate tasks like compiling, packaging, running automated tests, deploying to production and creating documentation. This section however talks about simply building your code.

Key Considerations

  • Most projects already have build tools for them. Ant, Maven, Gradle.
  • There might be a need for distributed builds. A build automation tool must be able to manage these dependencies in order to perform distributed builds.
  • A DevOps engineer may have to write configuration scripts to build artifacts

Eg: Gradle

Gradle can be integrated with Github. What we can achieve by this is GitHub would recognize Gradle build scripts and provide nice syntax highlighting.

Gradle can be integrated with any CI server. There is a good Jenkins plugin for Gradle. It can be integrated with TeamCity – an extensible build server. So essentially what we achieve is – “a user’s commit triggers a job in Jenkins, which uses Gradle to build the repository”.

Gradle can be integrated with some Repository Managers like Nexus. So if gradle builds an artifact successfully, it will have to be transferred to some remote location, artifacts of older builds need to be maintained, maintain common binaries across different environments and provide secure access to the artifacts. This is the role of a Repository Manager.

More Reads

Integrating Gradle with Jenkins – https://wiki.jenkins-ci.org/display/JENKINS/Gradle+Plugin

Integrating Gradle with TeamCity – http://confluence.jetbrains.com/display/TCD8/Gradle

What is Nexus? http://www.sonatype.com/nexus/why-nexus/why-use-a-repo-manager

What is a Repository Manager – http://blog.sonatype.com/2009/04/what-is-a-repository/#.VFY5s_mUd8E

Test Automation

When user commits code and its successfully built and deployed to a test environment, the actual test jobs need to be started on that environment. The test jobs include unit tests as well as integration tests. The testing would most likely involve creating test VMs and cleaning them up after every tests. The test results would have to be relayed back to the developers and others at stake.

Key Considerations

  • From DevOps perspective we dont have a “test automation tool”. What we have is an automation framework, which will involve test automation. Hence this is one of the most important aspects of deciding on a DevOps automation tool.
  • There are several CI servers, most popular being Jenkins. Travis and BuildHive are hosted services offering some additional options. The choice of a CI server will have to be made depending on several factors.
  • The frequency of commits need to be estimated. Will you run tests after each commit?
  • There are some tests that would run nightly
  • A DevOps engineer will have to write configuration scripts which will trigger test jobs, create VMs, give back feedback, etc.

Eg: CI Server – Jenkins

Jenkins can be configured to trigger jobs that run tests. It can spawn VMs, clusters where the tests run. Depending on the tests, data volumes, you may have to consider using open source Jenkins or the Enterprise version.

Jenkins can be integrated with Git/Gerrit. So every push can trigger a build & test job.

Jenkins can be integrated with Code Analysis tools like Checkmarx.

Jenkins can be integrated with Repository Managers like Nexus.

Jenkins can be integrated with Issue Tracking tools like RedMine.

More Reads

DevOps and Test Automation – http://www.scriptrock.com/blog/devops-test-automation

Case Study: Deutsche Telekom with Jenkins & Chef – http://www.ravellosystems.com/customer-case-studies/deutsche-telekom

Git, Gerrit Review, Jenkins Setup – http://www.infoq.com/articles/Gerrit-jenkins-hudson

Gerrit Jenkins Git – https://wiki.openstack.org/wiki/GerritJenkinsGit

Checkmarx – https://www.checkmarx.com/glossary/code-analysis-tools-boosting-application-security/

DevOps (Part 1): Introduction to DevOps

What Is DevOps?

The answer to this question can be given philosophically and technically. It is important to know the philosophy behind DevOps. But you will find this explanation in plenty of sites so I will skip this part.

Today almost everything is getting “automated”. Repetitive tasks are replaced with machines / code. Methods are being devised to address and minimise the defects and bugs in any system. Methods to minimise human error are being devised. The issues in the software development cycle are being addressed. One major issue in the SDLC was the process of how the developed code moved to production. DevOps addresses these issues. DevOps is an umbrella concept which deals with anything to smooth out the process from development to deployment into production.

DevOps Objectives

  • Introduce Code Review System
  • Automate Build
  • Automate Testing
  • Automate Deployment
  • Automate Monitoring
  • Automate Issue Tracking
  • Automate Feedbacks

These objects can be achieved by setting up a Continuous Integration pipeline and a Continuous Deployment/Delivery process. Post delivery, a process for Continuous Monitoring is set up.

Points to be considered while setting up a CI/CD pipeline

  • Developers push lot of code (many commits)
  • With every commit a build would be triggered
  • Automated tests should be run in production-clone environment
  • Builds should be fast
  • Generate feedbacks (to developers, to testers, to admins, to open source community, etc)
  • Publish latest distributable
  • Distributable should be deployable on various environments
  • Deployment process should be reliable and secure
  • One click deploys

A Simplified CI/CD Pipeline

DevOps for Kohls

Chef Knife plugin for Windows Azure (IAAS)

Chef is an open-source systems management and cloud infrastructure automation framework created by Opscode. It helps in managing your IT infrastructure and applications as code. It gives you a way to automate your infrastructure and processes.

Knife is a CLI to create, update, search and delete the entities or manage actions on entities in your infrastructure like node (hosts), cloud resources, metadata (roles, environments) and code for infrastructure (recipes, cookbooks), etc. A Knife plug-in is a set of one (or more) subcommands that can be added to Knife to support additional functionality that is not built-in to the base set of Knife subcommands.

The knife azure is a knife plugin which helps you automate virtual machine provisioning in Windows Azure and bootstrapping it. This article talks about using Chef and knife-azure plugin to provision Windows/Linux virtual machines in Windows Azure and bootstrapping the virtual machine.

Understanding Windows Azure (IaaS):

A complete deployment for a virtual machine in Azure looks as below.

Windows Azure IaaS deployment model

To deploy a Virtual Machine in a region (or service location) in Azure, all the components shown described above have to be created;

  • A Virtual Machine is associated with a DNS (or cloud service).
  • Multiple Virtual Machines can be associated with a single DNS with load-balancing enabled on certain ports (eg. 80, 443 etc).
  • A Virtual Machine has a storage account associated with it which storages OS and Data disks
  • A X509 certificate is required for password-less SSH authentication on Linux VMs and HTTPS-based WinRM authentication for Windows VMs.
  • A service location is a geographic region in which to create the VMs, Storage accounts etc

The Storage Account

The storage account holds all the disks (OS as well as data). It is recommended that you create a storage account in a region and use it for the VMs in that region.
If you provide the option –azure-storage-account, knife-azure plugin creates a new storage account with that name if it doesnt already exist. It uses this storage account to create your VM.
If you do not specify the option, then the plugin checks for an existing storage account in the service location you have mentioned (using option –service-location). If no storage account exists in your location, then it creates a new storage with name prefixed with the azure-dns-name and suffixed with a 10 char random string.

AZURE VIRTUAL MACHINE

This is also called as Role(specified using option –azure-vm-name). If you do not specify the VM name, the default VM name is taken from the DNS name( specified using option –azure-dns-name). The VM name should be unique within a deployment.
An Azure VM is analogous to the Amazon EC2 instance. Like an instance in Amazon is created from an AMI, you can create an Azure VM from the stock images provided by Azure. You can also create your own images and save them against your subscription.

Azure DNS

This is also called as Hosted Service or Cloud Service. It is a container for your application deployments in Azure( specified using option –azure-dns-name). A cloud service is created for each azure deployment. You can have multiple VMs(Roles) within a deployment with certain ports configured as load-balanced.

OS Disk

A disk is a VHD that you can boot and mount as a running version of an operating system. After an image is provisioned, it becomes a disk. A disk is always created when you use an image to create a virtual machine. Any VHD that is attached to virtualized hardware and that is running as part of a service is a disk. An existing OS Disk can be used (specified using option –azure-os-disk-name ) to create a VM as well.

Certificates

For SSH login without password, an X509 Certificate needs to be uploaded to the Azure DNS/Hosted service. As an end user, simply specify your private RSA key using –identity-file option and the knife plugin takes care of generating a X509 certificate. The virtual machine which is spawned then contains the required SSH thumbprint.

Install knife-azure plugin

You can either install via rubygems or build it from the latest source code.

Gem Install

Run the command: gem install knife-azure

Install from Source Code

To get the latest changes in the knife azure plugin, download the source code, build and install the plugin:

1. Uninstall any existing versions

$ gem uninstall knife-azure

Successfully uninstalled knife-azure-1.2.0

2. Clone the git repo and build the code

$ git clone https://github.com/opscode/knife-azure
$ cd knife-azure
$ gem build knife-azure.gemspec
WARNING: description and summary are identical
Successfully built RubyGem
Name: knife-azure
Version: 1.2.0
File: knife-azure-1.2.0.gem

3. Install the gem

$ gem install knife-azure-1.2.0.gem
Successfully installed knife-azure-1.2.0
1 gem installed
Installing ri documentation for knife-azure-1.2.0…
Building YARD (yri) index for knife-azure-1.2.0…
Installing RDoc documentation for knife-azure-1.2.0…

4. Verify your installation

$ gem list | grep azure
knife-azure (1.2.0)

To provision a VM in Windows Azure and bootstrap using knife,  Firstly, create a new windows azure account: http://manage.windowsazure.com and secondly, download the publish settings file from https://windows.azure.com/download/publishprofile.aspx?wa=wsignin1.0
The publish settings file contains certificates used to sign all the HTTP requests (REST APIs).

Azure supports two modes to create virtual machines – quick create and advanced.

Azure VM Quick Create

You can create a server with minimal configuration. On the Azure Management Portal, this corresponds to the “Quick Create – Virtual Machine” workflow. The corresponding sample command for quick create for a small Windows instance is:

knife azure server create
–azure-publish-settings-file ‘/path/to/your/cert.publishsettingsfile’
–azure-dns-name ‘myservice’
–azure-source-image ‘windows-image-name’
–winrm-password ‘jetstream@123’
–template-file ‘windows-chef-client-msi.erb’
–azure-service-location “West US”

Azure VM Advanced Create

You can set various other options in the advanced create including service location or region, storage-account, VM name etc. The corresponding command to create a Linux instance with advanced options is:

knife azure server create
–azure-publish-settings-file “path/to/your/publish/settings/file”
<strong>–azure-connect-to-existing-dns
</strong>–azure-dns-name ‘myservice’
–azure-vm-name ‘myvm02’
–azure-service-location ‘West US’
–azure-source-image ‘source-image-name’
–ssh-user ‘jetstream’
–ssh-password ‘jetstream@123’

List Available Images

knife azure image list

List currently available virtual machines

knife azure server list

Delete and cleanup a virtual machine

knife azure server delete –azure-dns-name myvm02 ‘myservice’ –chef-node-name ‘myvm02’ –purge

knife azure server delete –azure-dns-name myvm02 ‘myservice’ –chef-node-name ‘myvm02’ –purge