Leaving Atlanta The Agile Way

I’m leaving Atlanta after a series of bad things happened and one good one. The good one is that I was hired by ThoughtWorks to be an awesome devOps consultant. This means that I have to learn the Thoughtworks way of doing things, which means learning agile. I’ve been staffed on a couple projects already, so have been on agile teams, but not managed my own project. I have a ton to do, so I’m going to organize it using Pivotal Tracker and applying agile methodologies as well as possible for a team of one. There will be solo standups at 10:10 daily, I’m going to walk the wall and talk to myself as often as needed!

Obviously some things I’m not going to be emulate simply because this is not a software project. I’ll try and make that clear.

You can track my progress, watch me shuffle cards around and see how I’m doing. I’ll be adding cards and scoring stories tonight, so it should be pretty filled out by the time you read this.

I’m also reading the PragProg Thoughtworks book of essays, so a ton of project management stuff is coming at me right now. I’m still forming my own ideas about how to manage this.

Fellow TWers, I wanted to use Mingle for this, but I’m working out some Licensing issues with the Studios team

~ 26 Apr 2012

CI with puppet, ec2, Jenkins, git-flow, Tomcat, and gitolite

These are notes I took while doing an assignment for a friend. I was on a pretty serious time constraint, so I cut some corners, which I will address at the end of the article. I’m not advocating that this is the best way to structure a project like this, but I think it’s pretty good given the situation.

We’re writing a tomcat app in Eclipse, on Windows boxes. As it stands, we’re not doing any kind of automated testing, we have version control (svn) , but we’re not particularlly attached to it, and we would like to be able to single-click deploy to a number of environments. We have a small team of developers that wear different hats, but we have dedicated QA people. Can you build us something that would make our dev and deploment process smoother?

The most interesting part of this setup is the gitolite server. It’s not using ‘true’ git-flow because one of the stipulations was that the developers were using eclipse on windows to develop their tomcat application. This means that they can’t use the git-flow scripts until someone writes a git flow plugin for eclipse. I may do this in the near future if this comes up often enough.

My solution is going to look like this:

  • Puppet for installing and maintaining configuration
  • Jenkins for builds and deployment of environments
  • git for version control, the repo broken into feature develop QA release hotfix and master(prod), al-la git-flow

The server setup

For fun, I used geppetto to write and test my modules. I thought that I might deploy this stackhammer, but didn’t wind up doing that in the end. Both of them are cool projects. I think for something like this, geppetto was a little overkill, but it was interesting to use a new tool regardless.

This is only a single server, so I’m going to do everything in puppet standalone “mode”. I think that this is an often overlooked and incredibly useful way to develop modules and avoid repeating mistakes one has the tendency to make with shell scripts.

What I mean by this is that I wrote a module then wrote a test and ran puppet apply --modulepath=/etc/puppet/modules/ --verbose ${modulename}/tests/init.pp. Doing it this way means that if it were necessary, we could deploy these modules and configurations to many machines and have an idea about how it would work.

First thing I did was run apt-get update followed by apt-get upgrade. Just to get a baseline of security updates and make sure the system is up to date.

Puppet

Nothing really special here, I just added the apt repo from puppet labs, and added their key.

echo -e "deb http://apt.puppetlabs.com/ lucid main\ndeb-src http://apt.puppetlabs.com/ lucid main" >> /etc/apt/sources.list.d/puppet.list
apt-key adv --keyserver keyserver.ubuntu.com --recv 4BD6EC30

There were some dependency issues with libaugeas-ruby1.8, and I saw that it was not in the hardy repo. I added the lucid apt repo and got it from there.

echo -e "deb http://us.archive.ubuntu.com/ubuntu/ lucid main restricted" >>/etc/apt/sources.list.d/lucid.list
echo -e "deb-src http://us.archive.ubuntu.com/ubuntu/ lucid main restricted" >>/etc/apt/sources.list.d/lucid.list

apt-get update 
apt-get -y install puppet git git-core apt curl

At this point. I removed the “rogue” repos, because I plan on administrating repositories with puppet.

For the rest of the package management and configuration I want to automate, I’m going to be using puppet.

If this were a production box, I would probably create an image or snapshot from this point. I worked this exercise out using virtualbox, on my local machine so I took a snapshot.

Jenkins

At work, I had just used an excellent puppet module for installing and configuring jenkins, so I hit that one up first.

I had to make some modifications for ubuntu 8, namely including the sun jdk instead of the included openjdk. I built a little module for installing the sun jdk and jre, and then included them as requirements for jenkins. Jenkins also needed keys and it’s gitconfig setup. I put those in it’s puppet module. This is available on my github.

additional configuration
  • Installing the git plugin (done with web interface)
  • Building the ant file for different environments

Jenkins is checking out different branches of the project and building and deploying them upon checkin. There are 4 branches in the repo and jenkins can read all of them. Jenkins checks out each branch, builds it, and then runs an ant task (qa prod dev stage). The ant tasks will compile, war, and deploy to their respective places in the tomcat server, then restart it. Each is accessible by it’s name and environment, and wiped clean every time the jenkins runs the build.

http://hostname:8180/fightingChickenDev
http://hostname:8180/fightingChickenQA
http://hostname:8180/fightingChickenStage
http://hostname:8180/fightingChickenProd

Gitolite server

I built a module for installing and configuring gitolite. The users still have to be managed by hand. I wound up using a package from the natty repo for this, which I installed using an apt module that I’ve used in the past. The users for gitosis could be managed using puppet templates, I chose not to for the sake of saving time. In production, I would probably manage this by hand anyway, or if the team was big enough and growing, externalize this with LDAP.

the git server is accessible via

git clone git@hostname:fightingChicken
git clone git@hostname:gitolite-admin
additional configuration

Users are managed by adding their keys to gitolite-admin/keydir directory in the admin repo. The filename of the key has to match the username. Permissions are managed by the gitolite-admin/conf/gitolite.conf file. I did something interesting in the gitolite.conf file:

There are 4 branches in the repo, development, qa, stage, master.

gitolite.conf

    repo    gitolite-admin
            RW+     =   adminkey
            RW+     =   admin

    repo    fightingChicken
            RW+           =   adminkey
            RW+           =   admin
            R             =   jenkins
            RW+   develop =   @all
            RW+   qa$     =   @developers
            RW+   stage$  =   @qa
            RW+   master$ =   @ops

The development process would go like this:

Developers (and everyone else) can push to dev and qa. Once the code arrives at QA, it is deployed to the qa environment. If the code is satisfactory, it can only be pushed to stage by the QA team members. If it is not, it is pushed back to the development branch for more work. QA can push to stage, where the ops team looks at it and makes sure that it will deploy without issues. Once that is verified, ops can push to the master, where it is deployed to production.

All of the builds are automated, but in reality, I would probably make that last push a manual one, or at least on that wasn’t on a cron job. If it were more than 2 steps, I would automate it, but I think forcing members of the ops team to actually decide to deploy the application after it is in the master repo (and tagged so we can roll back!) is OK.

Post Mortem writeup

What commands to execute to deploy the application?
  • Clicking on the build now in jenkins will build and deploy the application. You can also do it from the command line by launching ant from the root of the application directory.

If I had more time:

  • The ops side of my brain is screaming because this box doesn’t exist because it’s not monitored, and the basics (ntp, dns, ip, hostname, tomcat, etc..) are unmanaged. I could have included a bunch of modules that I use regularly, but I think that would just confuse things for this project.
  • Redmine. Setting up redmine by hand is really easy. Doing an automated redmine deployment and integrating it with jenkins is not. I would have set it up by hand ‘for show’, but I came to the conclusion that it didn’t make any sense to do so. I think having a ticket tracking system is essential to any good software project, and if you’re not going to pay for JIRA (free tools only in the project description) then redmine is the way to go.
  • I would write an egit wrapper for git-flow. git-flow makes so much sense to me, but windows users can’t use it.
  • My restart tomcat on every build paradigm is clumsy. I would manage each container separately so that restarts on the qa environment would not effect anyone else. Doing this would probably necessitate managing tomcat through puppet, which would be the right way of deploying this in production. As I said above, I would have built modules for all of the components of the system that required configuration. Every once in a while, a build fails because /etc/init.d/tomcat tomcat fails to execute. This could be fixed by siloing each environment and just reloading instead of restarting.
  • Many people have problems with branched development because of merge hell. I’m on the fence about this for this project. For projects with more than a few developers, or developers that change frequently, than single tree is the only way to go. For this project I wanted to explore what could be done witha a git-flow like development environment. We’ll see how the team does with this. It’s nice that this is flexible, and the team can change it if their needs do.
Why were certain tools selected:
  • Jenkins because I think it is the best tool for the job when it comes to continious integration. It’s mature, stable, and easy to deploy. There are a ton of plugins, and it scales well. Authentication can be added pretty easily,
  • Puppet because I’m most familiar with it. I could have used any number of configuration management tools, but this is the one I know best.
  • Git because I think it’s the best dvcs there is. It is a pain that windows users have to use a graphical interface but I think most windows developers are used to that.
What is your recommendation for future work if time allows?
  • I would build modules for the ‘moving parts’ of the server, as I mentioned above, and figure out why the application did not deploy properly.

~ 23 Mar 2012

Wasting time with .vimrc

This may be just about my process, but my .vimrc is an unbelievable time suck for me.

When I sit down to play with or learn a new language or framework, the first thing I do is go grab syntax highlighting files, snippets, and whatever vim plugins that can do some hand-holding and prevent me from doing stupid things. For coffeescript and clojure, this turned into a battle that caused me to initially drop vim for learning them. I tried to get an integrated REPL for both langauges to work in different ways, and in then end I discovered that they don’t matter when you’re getting started. Use whatever is comfortable, even if it’s known to be non-optimal.

When you’re learning something, don’t bother with getting the tooling just right. Just jump in and do it. Forget snippets and plugins. You can mess with them when you find that you need them.

~ 19 Mar 2012

design fail snowball at the airport

The Atlanta airport (supposedly the busiest on the planet), has an incredibly bizarre problem that due to a recent personal flurry of travel has become more apparent and irritating to me.

There is a train that takes one from terminal to terminal, as there is in any large, modern, American airport. This train has a audio prompting system, so in the event you can’t see or read the signs, you know what terminal you are in, and what direction the train is going.

The female voice that tells you which station the train is in, is so poorly synthesized that it’s nearly impossible to differentiate between B, C, D, and E unless you are giving it your fullest attention. I imagine they let this go in production for a while, or maybe they realized it during testing and debugging. They exasperated the issue instead of fixing it by having the train-robot repeat at maximum volume “This train is arriving at TERMINAL A. A as in alpha.” The distance between each stop (very short) and volume of the robot make this an extremely unpleasant ride. The robot is talking the entire way. Every time I make it back to baggage claim, which is your destination when coming inbound, I have to take a deep breath and adjust myself and remind myself ehat I’m only in a huge rush and feeling anxious because I’ve been shouted at by a hostile, barely-literate robot for the past few minutes.

This whole situation could easily be avoided in numerous ways:

  • Using a recording instead of using the synthesized voice.
  • Using large, obvious visual prompts on the inside of the train.
  • Lowering the volume and letting people ask one another.

There are many more. Instead of fixing the problem they allowed it to snowball into a UX nightmare. I imagine these things are not hard to change, someone just has to want to.

~ 27 Feb 2012

ruby web-based LDAP password changer

(I was just turning my notes into a blog post, so if it seems weird, rambling, and contains too many shell commands, that’s why.)

I noticed that at my company when someone uses the change password tool in Pidgin, they get an access denied, and no one can set their own passwords. Pain in the ass. Every time a user forgets, or their pidgin client somehow loses the password, they have to ask us for it, and we have to generate a password that they must remember. How frickin’ prehistoric. I set out to figure out why this was, and how to fix it.

The fixing it part, not so much. Turns out that our chat server, openfire, treats the LDAP database as read only. I can imagine why, but that’s pretty annoying. So there needed to be another way around this. I figured, why not use this as an opportunity to learn some new things! I banged out a password changer in python in about 20 minutes. It worked, but I wanted to see how I could do this in ruby. Turns out the whole batteries included bragging rights of the python community really means something. I’m not blaming anyone, I’m merely illustrating that getting something up and running from zero in ruby on Debian Lenny is not as easy as you would hope.

Here’s what I spent the first hour or so doing with ruby:

No ruby, installed. I’ll take care of that!

apt-get install ruby
ruby --version

worked!

gem install sinatra

doh. no gem. I’ll just install that right quick…

apt-get install libgemplugin-ruby
gem install sinatra

Nice, it worked. Now I need a ruby ldap module. That should be a solved problem a million times over. This one looked stable enough to me, and coincidentally was the first result on google for ruby ldap.

git clone https://github.com/alexey-chebotar/ruby-ldap.git

Oh, no git.

apt-get install git

Hmmm WTF is gnuit? Ah, it’s like Norton Commander, an OFM! Awesome! I love those. Except, it’s not the git I know, love, and need. Screw it, I’ll skip that one. Whatever. I’m not going to send upstream patches from this box anyway, I’ll just get the zip

wget https://github.com/alexey-chebotar/ruby-ldap/zipball/master
unzip master 

Oh, no unzip. ugh… this is getting old…

apt-get install unzip
unzip master 
cd alexey-chebotar-ruby-ldap-71302b3/
ruby extconf.rb --with-openldap2

It’s complaining about mkmf. I don’t know what that is, but I know I need it to build this.

gem install mkmf
gem search mkmf

nothing. Quick google around, mkmf is in ruby1.8-dev

apt-get install ruby1.8-dev
ruby extconf.rb --with-openldap2
cat mkmf.log 

ok, now we’re missing some ldap libraries.

apt-cache search ldap | grep libraries
apt-get install libldap-2.4-2
sudo updatedb
locate ldap.h

Still no ldap.h, guess I’ll google around for that one too.

apt-get install libldap2-dev
ruby extconf.rb --with-openldap2

nice! now just to compile…

saslconn.c:19:23: error: sasl/sasl.h: No such file or directory

ok, need that one too.

locate sasl.h

nothing… back to google.

apt-get install  libsasl2-dev
updatedb
locate sasl.h
make clean
make 
make install

Great. I got it up and running. Now after playing around with it, I get my first connection object working and then… hit a wall. I pass it my credentials and I get a protocol error. Weird. I change the order, even though it looks right, then decide to do the right thing. Turn on maximum logging, and try again. My log says:

Jan 26 17:58:40 v2 slapd[17786]: send_ldap_result: err=2 matched="" text="historical protocol version requested, use LDAPv3 instead" 
Jan 26 17:58:40 v2 slapd[17786]: send_ldap_response: msgid=4 tag=97 err=2 

So this is where my knowledge ends. I google around again for answers about if ruby-ldap works with LDAPv3, and if so, how. I wind up reading the source. Ruby does generate some awesome docs.

LDAP::Conn.set_option(LDAP::LDAP_OPT_PROTOCOL_VERSION, 3 )

Fine, it does, awesome. Now to mess around in irb until I get it right. After about 45 minutes of typing and getting things wrong, I had my command line password changer for LDAP.

#/usr/bin/ruby -w

#arguments UserName CurrentPassword NewPassword 
username        = ARGV[0].to_s
currentpassword = ARGV[1].to_s
newpassword     = ARGV[2].to_s

if ARGV.length < 1
  puts "clichangepass.rb UserName CurrentPassword NewPassword"
  exit
end

require 'ldap'

$HOST =    'localhost'
$PORT =    LDAP::LDAP_PORT

LDAP::Conn.set_option(LDAP::LDAP_OPT_PROTOCOL_VERSION, 3 )


def changepass(username, currentpassword, newpassword)
  conn = LDAP::Conn.new($HOST, $PORT)
  conn.bind("uid=#{username}, ou=people, dc=test, dc=com","#{currentpassword}")
  changepw=[LDAP::Mod.new(LDAP::LDAP_MOD_REPLACE, 'userPassword', ["#{newpassword}"]),]
  begin
    conn.modify("uid=#{username}, ou=people, dc=test, dc=com", changepw)
  rescue LDAP::ResultError
    conn.perror("modify")
    exit
  conn.unbind
  end
end

puts "#{username}"
puts "#{currentpassword}"
puts "#{newpassword}"

changepass("#{username}", "#{currentpassword}", "#{newpassword}")

Supposedly that was the hard part. Getting a ruby application to go from the command line to a web application is supposed to be CAKE from what I’ve heard. I had my doubts. I looked into rails for this, but didn’t really want to get all that involved. I don’t intend on doing anything other than submitting a form with this app, so rails is overkill. Sinatra seemed like the natural choice. I installed it and got to messing with it. Seemed really really easy at first. Got the hello world up in a matter of seconds. Got my /form route done even faster. In all the tutorials, it looks like we’re using erb, which I know rubyists despise in favor of any other templating engine, so I skipped it entirely and went with haml.

HAML was a really nice surprise. Concise, easy, to the point. I was trying to do some not-so-fancy stuff with inheritance and couldn’t figure that out immediately, but getting the application working is my first priority, and I’ll worry about how to make the code look good later.

The script itself looks alot like the one above. I just changed the names of the username currentpassword and newpassword variables and was on my way. The sinatra side of things is below.

require 'rubygems'
require 'sinatra'
require 'changepass'

get '/form' do
  username        = params[:username]
  currentPassword = params[:currentPassword]
  newPassword     = params[:newPassword]
  haml :form
end

post '/form' do
  changepass("#{params[:username]}", "#{params[:currentPassword]}", "#{params[:newPassword]}")
end

I have the application working now, and have to worry about deploying it now. It’s just a few lines of ruby, and I would like to run it from sinatra and WebBRICK, but I know that’s really not the right thing to do if the server is already running an apache instance, which is was.

I jumped through all the (admittely easy) hoops of installing installing passenger and getting it up and running. The thing that screwed me up was that I didn’t know that you had to have an empty directory named /public under the application root. That piece of information was integral, and really frustrating. It actually was the most time consuming part of this whole project. After I installed and got passenger running, I debugged this issue for about 2 hours. I know, what a waste, I should have just known I needed it. Oh well. Now I know. Woot.

I learned quite a bit, and am looking forward to coding in ruby more often, as I plan on building some web apps in the very near future.

~ 01 Feb 2012

Previously