Vagrant, Jenkins, and CoreOS quickstart

I recently started exploring how to allow teams to self service their own infrastructure for jenkins. What I mean by this is somehow allowing them to spin up new nodes whenever they wanted to by just including some metadata about their projects, and then pushing to git and having some automation take care of the rest.

I don’t have a full solution worked out yet, but I used this as an excuse to play with docker and CoreOS.

These are just preliminary notes, I will post more on how successful I am when I get something more solid in place.

Check out the repo for more details.

~ 20 Apr 2014

Vagrant, chef-solo, and databags learning experience

I’ve never had to use databags before in chef-solo, but this past week I needed to do some exploration with the opscode squid cookbook. Aside from the opscode databag docs, I couldn’t find much good information on it, so here goes.

Vagrant

This part is straightforward. Just add a directory where your Vagrantfile lives and then include a databags_path in your provisioning block.

config.vm.provision :chef_solo do |chef|
  chef.data_bags_path = "data_bags"

  chef.run_list = [
    "recipe[rubygems_proxy::default]"
  ]
end

Databag structure

Databags must be in a directory with a name matching the databag. Each individual databag must be a json file, the only requirement being a id element. The (truncated for clarity) directory structure looks like:

.
|-- Berksfile
|-- cookbooks
|   `-- squid
|-- data_bags
|   |-- squid_acl_actions
|   |   `-- rubygems.json
|   `-- squid_hosts
|       |-- everythingelse.json
|       `-- rubygems.json
`-- Vagrantfile

Databag items

Databag items themselves, as mentioned above, only need to contain an id element to be considered valid.

{                                                                                  
  "type": "dstdomain", 
  "id": "everythingelse",
  "net": [
    "all"
  ]
} 

Using Databags

In the squid cookbook, databags are used to describe ACL rules and hosts which are retrieved using a library. To access them in your recipes (or libraries or resources), use the data_bag() method. In a chef server and chef client environment, you can leverage search(), but this is not available in chef-solo out of the box.

def squid_load_host_acl
  host_acl.push [group['id'],group['type'],host]
      end
    end
  rescue
    Chef::Log.info "no 'squid_hosts' data bag"
  end
  host_acl
end

I hope this helps get started with databags, chef-solo, and vagrant.

~ 01 Nov 2013

Chef Testing Stratagies Compared

MiniTest/chefspec:

Minitest is an integration testing tool that allows one to make assertions about the final state of the machine after the cookbook run. This requires a ‘clean’ machine to run your cookbooks on to get accurate results.

  • Pros: The most logical way to automate tests that you would generally do manually. Gives you the freedom to refactor a cookbook or library and still confirm that it performs your intended actions.

  • Cons: Slow, you must spin up a machine using test-kitchen, vagrant, or some other virtualization technology. LXC is a great candidate for running minitests, but that is bound to a single platform.

  • Link: https://github.com/calavera/minitest-chef-handler https://github.com/acrmp/chefspec

  • Pipelineable? Yes. This framework was designed with pipelines in mind.

Example

describe 'mysql::server' do
  it 'runs as a daemon' do
    service(node['mysql']['service_name']).must_be_running
  end
end

This example will check if mysql is running, provided that attribute is populated.

Cucumber Chef:

Cucumber chef is an integration testing tool that uses an easy to read and write syntax to describe testing scenarios and infrastructure.

  • Pros: Allows you to use gherkin syntax to describe your infrastructure and test the outcome of your recipes. A natural choice for doing true TDD on your cookbooks.

  • Cons: Many people don’t like gherkin as it is very verbose. It makes some serious assumptions about your workflow and intentions. Not necessarily a bad thing, but it is very opinionated. By default it sets up an EC2 instance that your tests will run inside using LXC. This takes a while and requires you to be online.

  • Speed: Cucumber itself is fast enough, but this process feels like a kludge. Needing to be online and spending (marginal) amounts of money when I have a linux computer in front of me feels bizarre. I understand the cross platform reasoning behind it, but I would prefer some options.

  • Link: https://github.com/Atalanta/cucumber-chef/wiki

  • Pipelineable? Not a natural fit for a pipeline due to it’s reliance on EC2, but could be done.

Example

 Scenario: Chef-Client is running as a daemon
  When I run "ps aux | grep [c]hef-client"
    Then I should see "chef-client" in the output
    And I should see "-d" in the output
    And I should see "-i 1800" in the output
    And I should see "-s 20" in the output

Test-Kitchen:

Test kitchen simply runs your cookbooks against a few clean virtualbox nodes running different versions of different operating systems. It does not run any external verification, but you can leverage minitest to accomplish this.

  • Pros: Really easy to understand the results. Great replacement for manual testing. Can spin up a bunch of different platforms for testing your cookbook on different operating systems.

  • Cons: Time Consuming. Needs to build a bunch of VMs using vagrant and run against it.

  • Link: https://github.com/opscode/test-kitchen

  • Pipelineable? Debatable, at great length. It can be, but the length of time it takes, and the difficulty of capturing useful output limits its effectiveness.

chef whyrun:

Whyrun is a mode that in chef 10.14 or higher will give you an idea of what chef would have done on the node, had you run without it. This is similar to puppet’s dry-run mode, but has a few gotchas, such as it always assumes the positive outcome of only_if and not_if statements. It also does not cope well with interdependencies between cookbooks, as it has no way of verifying if a previous run was successful.

  • Pros: Better than nothing, good for debugging single cookbooks and recipes.

  • Cons: Very verbose output, hard to parse for correctness with anything other than trained eyeballs.

  • Link: http://www.opscode.com/blog/2012/09/07/chef-10-14-0-released/

  • Pipelinable? It is, but you would need to parse and capture the output and visualize it in an information radiator, which would not be impossible. It belongs on the left-ish.

Foodcritic:

Foodcritic is a linting tool to ensure Consistency and enforce some best practices in your cookbooks. There are two additional sets of rules that you can add to it, etsy and customink. This is run simply with foodcritic -I $location_of_custom_rules $cookbook_path/$cookbook_name

  • Pros: Enforces Cons:istency and a clean style for your cookbooks. Fast. Easy to use.

  • Cons: none.

Pipelineable? Definitely. This belongs on the far left of the pipeline, or ideally run pre-commit.

Example:

stillinvisible ~/Corp/CorpChef $ foodcritic -I foodcritic/* cookbooks/CorpBuildTools 
CINK001: Missing CHANGELOG in markdown format: cookbooks/CorpBuildTools/CHANGELOG.md:1
FC033: Missing template: cookbooks/CorpBuildTools/recipes/phantomjs.rb:13

This output means that my CorpBuildTools cookbook violates a few rules, and FC033 would cause my cookbook to fail to converge. Absolutely valuable, time-saving output.

knife cookbook test:

This is merely a syntax checker. Handy, but not nearly as comprehensive as it should be. As long as your recipies parse properly, this will pass. That means that you can have something like

execute "clean yum cache" do
  command "yum clean all"
  command "yum install ruby"
  mode "744"
  action :nothing
end

There are two glaring mistakes in that execute statement, but knife cookbook test won’t catch them, despite the fact that the cookbook won’t even compile. Don’t depend on this too heavily.

  • Pros: Will catch syntactic errors and prevent you from checking in stupid mistakes.

  • Cons: Fails to catch errors that will cause the cookbook to not compile. Pipelineable?: Definitely. Belongs on the WAY left side of the pipeline, or better yet, pre-commit.

Fast and dirty debugging

Iterating with chef-server is a pain if you’re pinning your cookbook versions (which you should be). A way around this is to use chef-solo where you can. I am definitely not advocating this workflow, but if you’re looking at why a machine converged a certain way, it can be helpful to try this.

Create a file with the following contents:

solo.rb

file_cache_path "/var/chef-solo"
cookbook_path ["/var/chef/cache/cookbooks/"]

Create another file with whatever attributes you need to use, and a run_list:

solo.json

{ 
  "run_list": [ 
  "recipe[CorpBuildTools::buildgems]"
  ] 
}

Run chef-solo -j solo.json. Observe the results, and make some changes. This can be useful when debugging a recipe on a disposable machine.

Tying it all togther:

Strainer is a tool that allows you to setup a workflow of testing tools and then run them against a cookbook while you are developing. My Colanderfile looks like this:

knife test: bundle exec knife cookbook test $COOKBOOK
foodcritic: bundle exec foodcritic -I foodcritic/* cookbooks/$COOKBOOK
chefspec: bundle exec rspec cookbooks/$COOKBOOK

I keep it in the same directory as the cookbook’s metadata.rb. From the top level run bundle exec strain cookbook_name and this will run knife test, foodcritic with any custom rules indicated by the -I flag, and then run the specs. This saves a bunch of typing, and can be automated with watchr or similar if you desire. Why-run could also be used as part of this workflow.

stillinvisible ~/work/Corp/CorpChef $ tree cookbooks/CorpBuildTools 
cookbooks/CorpBuildTools
|-- attributes
|   `-- default.rb
|-- Colanderfile
|-- files
|   `-- default
|       `-- tests
|           `-- minitest
|-- Gemfile
|-- Gemfile.lock
|-- metadata.rb
|-- README.md
|-- recipes
|   |-- buildgems.rb
|   |-- bundler.rb
|   |-- default.rb
|   |-- foodcritic.rb
|   |-- git.rb
|   |-- phantomjs.rb
|   `-- ruby.rb
|-- spec
|   |-- buildgems_spec.rb
|   |-- bundler_spec.rb
|   |-- default_spec.rb
|   |-- git_spec.rb
|   `-- phantomjs_spec.rb
|-- templates
|   `-- default
|       |-- CorpSoftware-phantomjs.repo.erb
|       `-- Gemfile-build.erb
`-- test
`-- kitchen
`-- Kitchenfile

Your cookbooks will begin to look more like real software when tests are added. If you’re hosting them internally, perhaps it’s time to start treating them that way and giving each one it’s own git repo.

Conclusion:

We’re using a combination of chefspec, minitest, foodcritic and vagrant for our pipeline. We haven’t completed it yet, but will publish details when we do. Local development is done using foodcritic, chefspec, and vagrant as well. I’m open to suggestions and would be interested in hearing what others are doing.

  • Pipelineable is a real word in the resource extraction business, so I thought it was ok to use here.

~ 27 Nov 2012

Opscode Community Summit 2012

The opscode community summit was a two day, community organized, unconferences. It was organized like many other unconferences with no set schedule or talks, merely a bunch of people sharing the same interests and then scheduling around open spaces. This format worked really well for the size of the summit, which was around 250 people.

The tl;dr summary was that almost every discussion eventually touched on the idea that our chef workflow is very broken. There were tons of differing opinions on testing, development, and management of modules. There is progress being made on all of them, but because there is no one recommended or set path for doing cookbook testing, the tools around them are constantly changing and getting abandoned.

Quick summary of some of the talks I participated in:

Berkshelf - Jamie of RiotGames

Berkshelf is a tool that one can use to manage dependencies in cookbooks and keep a sane record of what versions of cookbooks are running. Philosophically, each unit of your infrastructure cookbooks, such as nginx and MySQL are unimportant until packaged with your application, so why do we manage them this way?

Chef server is merely an artifact repository for software, and each cookbook is an individual piece of software, therefore it deserves it’s own git repo and test suite. Berkshelf can take some of the pain out of this by allowing you to specify a source and a version for each cookbook in a Berksfile, much like a Gemfile. The source can be a git repo, a filesystem, or a chef-server api endpoint.

Jamie recommends writing cookbooks that can be used in chef solo to make manual testing significantly easier and automated testing possible. At RiotGames they use Berkshelf, minitest, and test-kitchen for cookbook testing.

Jamie also reimplemented roles as cookbooks, because roles are dangerous and should not be used as they cannot be versioned, and that makes them useless. Use wrapper cookbooks.

Instant Infrastructure - Chris of Opscode

The idea behind instant infrastructure is two fold.

One is to give people the tools to describe their real world problems so that things can be done about them. The tool he has chosen for this task is Gherkin.

The other is to give non-technical people a way to deploy a sane, managed infrastructure. The plan for this is to deploy a usb stick that will wipe a laptop or desktop, replace it with ubuntu running a chef workstation, with some example cookbooks and the ability to create more usb keys.

The idea of getting people to write acceptance critera using gherkin seems like a great idea. Giving non technical people the ability to describe their problems in a way that a developer can address directly is a good one.

MotherBrain: Orchestration with Chef - Jamie of RiotGames

(coming to RiotGames (Riot Games) · GitHub soon)

MotherBrain has yet to be released, but it’s a tool that allows one to add layers of protection to your chef runs by interacting with the chef-server API. MotherBrain files contain logic about what services must be notified before making changes. For example, they allow the user to explain to motherbrain that before bringing up the application servers, the database server configuration must be complete.

It also allows you to stop convergence of specific parts of your recipes while you debug or troubleshoot servers that are configured to converge via a cron job. This feature sounds incredibly useful for real life failover and redundancy orchestration.

Tools to watch:

Cucumber-chef 2 Chef 11 !!! (mid nov release date, apparently will converge orders of magnitude faster) minitest test-kitchen rspec (library testing) fauxhai Berkshelf MotherBrain

Cookbooks to look at for inspiration:

Application cookbook by opscode (how to tie a bunch of library cookbooks together properly)

Redis by miah_ (has some tests)

mysql by opscode (also has good minitest suite)

Upcoming books:

Test Driven infrastructure with Chef (yes, again, written by the same guy so… buyer beware)

Books to read anyway:

The cucumber book

Metaprogramming ruby

I went to more talks, mostly about TDD with chef and other testing tools. If you interested in chatting about this stuff, or have any questions hit me up! We had interesting discussions about how to isolate cookbooks in your pipeline, but no real decisions were made. Many ideas, but we’re really not there yet.

~ 02 Nov 2012

What are we doing again?

Recently I’ve been reading and thinking quite a bit about what ‘we’ as developers are really doing, or attempting to accomplish. I’m sure this has happened to tons of other people and it’s gone away or gotten worse, depending on the individual. I’m think I’m done trying to make money. It’s has proved to be pretty easy, and I don’t want to be wealthy, or own things I can’t afford, or go on trips that I can’t pay for. So now what? The solution for me is to strive for excellence in everything I do, and to attempt to help people at the same time. I think this is my future.

~ 01 Oct 2012

Previously