Chef Push Jobs

What are Push Jobs?

Push Jobs work like knife-ssh. Almost. Almost because, in knife-ssh the changes are pushed from your workstation using the SSH protocol. In push jobs, the changes are pushed to the node by the Chef Server.

Chef is based on the “pull” and has a reason for that – to keep the server “thin”. But the changing challenges demand that there is a need for a push model. So chef has introduced push jobs by keeping the server is thin!

“Chef push jobs is an extension of the Chef server that allows jobs to be run against nodes independently of a chef-client run” – that’s how push jobs are defined. A job, in this context, is a set of commands that need to be run on the target node.

Difference between Push Jobs and knife-ssh

Push Jobs knife-ssh

Use message bus (zeromq)

Parallel ssh

Claims to attack the scalability issue

SSH Protocol is slow and CPU hungry at scale

Deployment status is relayed back

Feedback on deployment status is not as easy

Newly introduced

Been in the market for long

Complex at the moment, ready with just the basic foundation

Easy to use

Configuring Chef Push Jobs Server

You need either Enterprise Chef or Chef Server 12. It relies on the ACL system that was open sourced with Chef Sever 12. Also, the install command was introduced with Chef Server 12.

Push Jobs does not work with Open Source Chef Server 11. 

Can be setup as standalone or as HA

Run the following commands on Chef Server:

chef-server-ctl install opscode-push-jobs-server
opscode-push-jobs-server-ctl reconfigure
chef-server-ctl reconfigure

Setup Workstation

  • Install knife push plugin
    Gem install knife-jobs
  • Download push-jobs cookbook
    Push jobs cookbook would be used. So download it from the site or git clone the cookbook. You would have to fetch its dependency cookbooks as well.
    knife cookbook site download push-jobs
  • Extract and save the cookbook to your cookbook path
  • Edit the attributes file (push-jobs/attributes/default.rb)
    Update the attributes to add the push jobs package URL and checksum as mentioned.
    default[‘push_jobs’][‘package_url’] = ‘https://opscode-private-chef.s3.amazonaws.com/ubuntu/12.04/x86_64/opscode-push-jobs-client_1.1.5-1_amd64.deb’

    default[‘push_jobs’][‘package_checksum’] = ‘d659c06c72397ed2bc6cd88488349857f1958538‘

  • Upload the push-jobs cookbook to your ChefServer

Create Groups

Create the pushy_job_writers and pushy_job_readers on the organization of the Chef server and add your workstation user to that group.

Setup Node

Simply run the chef client with the recipe:

sudo chef-client –r “recipe[push-jobs]”

Run the knife node status commands to check the node status. It will just show the status “available” at this stage which confirms that the node is prepared for push events.

knife node status
knife node status <node-name>

Run Push Jobs

Run chef-client as:

knife job start ‘chef-client –r recipe[git]’ <node-name>

Run your commands/script as:

knife job start ‘my_script.sh’ <my_node>

Cloud Attribute when using Chef’s knife-ssh in VPC mode

Error: FATAL: 1 node found, but does not have the required attribute to establish the connection. Try setting another attribute to open the connection using –attribute.

You need to set the –attribute to have the *name* of the cloud attribute. In AWS case most likely you would be looking for this: “ipaddress”.

So your knife ssh command would look like –

knife ssh <name> -x <user> “sudo chef-client” -a ipaddress

Knife-cloud Gem: Introduction & Knife Plugin Development Using It

Reposted from – Clogeny, An Msys Company

Chef Software, Inc. has released knife-cloud gem. This article talks about what is the knife-cloud gem and how you can use it to develop your custom knife-cloud plugin.

Knife is a CLI tool used for communication between local chef-repo and the Chef Server. There are a couple of knife subcommands supported by Chef, e.g., knife bootstrap, knife cookbook, knife node, knife client, knife ssh, etc. Knife plugin is an extension of the knife commands to support additional functionality. There are about 11 knife plugins managed by Chef and a lot more managed by the community.

The concept of knife-cloud came up as we have a growing number of cloud vendors, and therefore a number of knife plugins, to support the cloud specific operations. The knife-cloud plugins use cloud specific APIs to provision a VM and bootstrap it with Chef. These plugins perform a number of common tasks, such as connection to the node using SSH or WinRM and bootstrapping the node with Chef. The knife-cloud (gem) has been designed to integrate the common tasks of all knife cloud plugins. As a developer of a knife cloud plugin, you will not have to worry about writing the generic code in your plugin. More importantly, if there is any bug or change in the generic code of the knife plugin, the fix would be done in knife-cloud itself. Today we need to apply such changes across all the knife plugins that exist.

Knife-cloud is open source available at: https://github.com/opscode/knife-cloud.
You may refer to https://github.com/opscode/knife-cloud#writing-your-custom-plugin about the steps to write your custom knife cloud plugin.

Clogeny Technologies has written a knife-cloud scaffolder (https://github.com/ClogenyTechnologies/knife-cloud-scaffolder) to make your job even simpler. The scaffolder generates the stub code for you with appropriate TODO comments to guide you in writing your cloud specific code.

To use the knife-cloud-scaffolder:
– git clone https://github.com/ClogenyTechnologies/knife-cloud-scaffolder
– Update properties.json
– Run the command: ruby knifecloudgen.rb E.g., ruby knifecloudgen.rb ./knife-myplugin ./properties.json

Your knife-myplugin stub will be ready. Just add your cloud specific code to it and you should be ready to use your custom plugin.

Chef Knife plugin for Windows Azure (IAAS)

Chef is an open-source systems management and cloud infrastructure automation framework created by Opscode. It helps in managing your IT infrastructure and applications as code. It gives you a way to automate your infrastructure and processes.

Knife is a CLI to create, update, search and delete the entities or manage actions on entities in your infrastructure like node (hosts), cloud resources, metadata (roles, environments) and code for infrastructure (recipes, cookbooks), etc. A Knife plug-in is a set of one (or more) subcommands that can be added to Knife to support additional functionality that is not built-in to the base set of Knife subcommands.

The knife azure is a knife plugin which helps you automate virtual machine provisioning in Windows Azure and bootstrapping it. This article talks about using Chef and knife-azure plugin to provision Windows/Linux virtual machines in Windows Azure and bootstrapping the virtual machine.

Understanding Windows Azure (IaaS):

A complete deployment for a virtual machine in Azure looks as below.

Windows Azure IaaS deployment model

To deploy a Virtual Machine in a region (or service location) in Azure, all the components shown described above have to be created;

  • A Virtual Machine is associated with a DNS (or cloud service).
  • Multiple Virtual Machines can be associated with a single DNS with load-balancing enabled on certain ports (eg. 80, 443 etc).
  • A Virtual Machine has a storage account associated with it which storages OS and Data disks
  • A X509 certificate is required for password-less SSH authentication on Linux VMs and HTTPS-based WinRM authentication for Windows VMs.
  • A service location is a geographic region in which to create the VMs, Storage accounts etc

The Storage Account

The storage account holds all the disks (OS as well as data). It is recommended that you create a storage account in a region and use it for the VMs in that region.
If you provide the option –azure-storage-account, knife-azure plugin creates a new storage account with that name if it doesnt already exist. It uses this storage account to create your VM.
If you do not specify the option, then the plugin checks for an existing storage account in the service location you have mentioned (using option –service-location). If no storage account exists in your location, then it creates a new storage with name prefixed with the azure-dns-name and suffixed with a 10 char random string.

AZURE VIRTUAL MACHINE

This is also called as Role(specified using option –azure-vm-name). If you do not specify the VM name, the default VM name is taken from the DNS name( specified using option –azure-dns-name). The VM name should be unique within a deployment.
An Azure VM is analogous to the Amazon EC2 instance. Like an instance in Amazon is created from an AMI, you can create an Azure VM from the stock images provided by Azure. You can also create your own images and save them against your subscription.

Azure DNS

This is also called as Hosted Service or Cloud Service. It is a container for your application deployments in Azure( specified using option –azure-dns-name). A cloud service is created for each azure deployment. You can have multiple VMs(Roles) within a deployment with certain ports configured as load-balanced.

OS Disk

A disk is a VHD that you can boot and mount as a running version of an operating system. After an image is provisioned, it becomes a disk. A disk is always created when you use an image to create a virtual machine. Any VHD that is attached to virtualized hardware and that is running as part of a service is a disk. An existing OS Disk can be used (specified using option –azure-os-disk-name ) to create a VM as well.

Certificates

For SSH login without password, an X509 Certificate needs to be uploaded to the Azure DNS/Hosted service. As an end user, simply specify your private RSA key using –identity-file option and the knife plugin takes care of generating a X509 certificate. The virtual machine which is spawned then contains the required SSH thumbprint.

Install knife-azure plugin

You can either install via rubygems or build it from the latest source code.

Gem Install

Run the command: gem install knife-azure

Install from Source Code

To get the latest changes in the knife azure plugin, download the source code, build and install the plugin:

1. Uninstall any existing versions

$ gem uninstall knife-azure

Successfully uninstalled knife-azure-1.2.0

2. Clone the git repo and build the code

$ git clone https://github.com/opscode/knife-azure
$ cd knife-azure
$ gem build knife-azure.gemspec
WARNING: description and summary are identical
Successfully built RubyGem
Name: knife-azure
Version: 1.2.0
File: knife-azure-1.2.0.gem

3. Install the gem

$ gem install knife-azure-1.2.0.gem
Successfully installed knife-azure-1.2.0
1 gem installed
Installing ri documentation for knife-azure-1.2.0…
Building YARD (yri) index for knife-azure-1.2.0…
Installing RDoc documentation for knife-azure-1.2.0…

4. Verify your installation

$ gem list | grep azure
knife-azure (1.2.0)

To provision a VM in Windows Azure and bootstrap using knife,  Firstly, create a new windows azure account: http://manage.windowsazure.com and secondly, download the publish settings file from https://windows.azure.com/download/publishprofile.aspx?wa=wsignin1.0
The publish settings file contains certificates used to sign all the HTTP requests (REST APIs).

Azure supports two modes to create virtual machines – quick create and advanced.

Azure VM Quick Create

You can create a server with minimal configuration. On the Azure Management Portal, this corresponds to the “Quick Create – Virtual Machine” workflow. The corresponding sample command for quick create for a small Windows instance is:

knife azure server create
–azure-publish-settings-file ‘/path/to/your/cert.publishsettingsfile’
–azure-dns-name ‘myservice’
–azure-source-image ‘windows-image-name’
–winrm-password ‘jetstream@123’
–template-file ‘windows-chef-client-msi.erb’
–azure-service-location “West US”

Azure VM Advanced Create

You can set various other options in the advanced create including service location or region, storage-account, VM name etc. The corresponding command to create a Linux instance with advanced options is:

knife azure server create
–azure-publish-settings-file “path/to/your/publish/settings/file”
<strong>–azure-connect-to-existing-dns
</strong>–azure-dns-name ‘myservice’
–azure-vm-name ‘myvm02’
–azure-service-location ‘West US’
–azure-source-image ‘source-image-name’
–ssh-user ‘jetstream’
–ssh-password ‘jetstream@123’

List Available Images

knife azure image list

List currently available virtual machines

knife azure server list

Delete and cleanup a virtual machine

knife azure server delete –azure-dns-name myvm02 ‘myservice’ –chef-node-name ‘myvm02’ –purge

knife azure server delete –azure-dns-name myvm02 ‘myservice’ –chef-node-name ‘myvm02’ –purge