Rapid scaling with auto-generated AMIs using Puppet

I wrote a blog about automatically updating your EC2 AMIs using Puppet on the Puppet Labs blog:  http://puppetlabs.com/blog/rapid-scaling-with-auto-generated-amis-using-puppet/

Automatically Updating Your Puppet Masters’ Modules

One common problem with managing a puppet infrastructure is updating the modules on the Puppet Masters. If you're managing your Puppet modules in Version Control such as subversion or git (you ARE doing this, right?), you can have the Version Control System tell your Puppet Masters to update their checkout of whatever enviornment you’re currently checking code in to.

The mcmasterupdate module will install a MCollective agent called puppetupdate on whatever nodes you declare the mcmasterupdate class on. The mcmasterupdate module also contains an example SVN hook. Let's take a look at how this all works.

Requirements:

The following modules will be required:

• mcollective
• mcmasterupdate

You can download the mcmasterupdate and mcollective modules from the Puppet Labs github account.
http://www.github.com/puppetlabs/puppetlabs-mcmasterupdate
http://www.github.com/puppetlabs/puppetlabs-mcollective

Configuring the agent

The agent is managed from the mcmasterupdate class. Only declare this class on your Puppet masters. You'll also need the mcollective class to be declared. Here's an example on your Puppet masters.

class { 'mcollective':
  server          => false,
  client          => true,
  version         => '1.2.0',
  stomp_port      => 6163,
  stomp_server    => 'your.stomp.server',
  manage_packages => true,
  manage_plugins  => true,
}

class { 'mcmasterupdate': }


Configuring the hook

In the ext directory of the mcmasterupdate module, there is a post-commit.svn file for subversion. The only configuration necessary is to create the ENVIRONMENTS hash at the top of the file. The keys of the hash will be the names of your environments as configured in the puppet.conf file on your masters. The value for each key will be the regular expression used to match the files being committed to the appropriate environment. For this example, let's say you three environments, development, testing, and production. Your hash will look something like this:

ENVIRONMENTS = {
  'development' => /puppet\/trunk/,
  'testing'     => /puppet\/branch\/testing/,
  'production'  => /puppet\/tags/,
}

Put this file in the hooks directory of your subversion repository and name it 'post-commit'.

Now whenever you commit new Puppet code, the post-commit script will update the
checkouts on all the Puppet Masters for each environment you checked code in to.

Templating Puppet Resource Type Documentation

Most are familiar with the puppetdoc command. It prints out documentation about various aspects of your puppet environment in markdown format. I've always been interested in having a way to collect the documentation on our puppet resource types (including ones we build) and automatically generating Confluence pages for each of them. Unfortunately Confluence doesn't support Markdown and puppetdoc doesn't support passing in an erb file. So we'll have to just do it ourselves. All we have to do is load up all of our puppet types, then take each one and run it through an ERB template. From there we can do whatever we want.



Simple enough? And here's our ERB:



That will generate Confluence friendly documentation. Of course you can modify it to do whatever you like. For those interested, here's my entire code I use to generate my Confluence pages. You'll probably want to modify it to suit your needs. I modified this module some else wrote Confluence4r

Running MCollective with RabbitMQ on CentOS

Most MCollective examples use ActiveMQ as the queueing system. However, I find ActiveMQ to difficult to manage. This is because of my two main hates: Java and XML. To resolve this I setup RabbitMQ as my queueing system on CentOS.

Step 1)
Install RabbitMQ:

root@server:~ rpm -i http://www.rabbitmq.com/releases/rabbitmq-server/v2.3.1/rabbitmq-server-2.3.1-1.noarch.rpm

Step 2)
Install MCollective

root@server:~ yum install mcollective-client mcollective

or get the RPMs from here: http://www.puppetlabs.com/downloads/mcollective/

Step 3)
Install stomp and amqp

The ruby gems works just fine

root@server:~ gem install stomp amqp

Step 4)
Configure MCollective

Put the following in /etc/mcollective/server.cfg replacing the identity with your server's FQDN. Skip if you already have mcollective configured.

# main config
topicprefix = /topic/mcollective
libdir = /usr/libexec/mcollective
logfile = /var/log/mcollective.log
daemonize = 1
keeplogs = 1
max_log_size = 10240
loglevel = debug
identity = server.mydomain.com
registerinterval = 300
classesfile = /var/lib/puppet/state/classes.txt

# connector plugin config
connector = stomp
plugin.stomp.host = server.mydomain.com
plugin.stomp.port = 6163
plugin.stomp.user = mcollective
plugin.stomp.password = marionette

# facts
factsource = facter
plugin.yaml = /etc/mcollective/facts.yaml

# security plugin config
securityprovider = psk
plugin.psk = abcdefghj

Step 5)
Configure RabbitMQ

Step 5.1)
Setup rabbitmq.config

Create the file /etc/rabbitmq/rabbitmq.config with the following contents

[ {rabbit_stomp, [{tcp_listeners, [6163]} ]} ].

Step 5.2)
Add mcollective user

The following command will create the mcollective user with password marionette.

root@server:~ rabbitmqctl add_user mcollective marionette

Step 5.3)
Set mcollective permissions

root@server:~ rabbitmqctl set_permissions -p / mcollective "^amq.gen-.*" ".*" ".*"

Step 6)
Install rabbitmq stomp plugin

root@server:~ cd /usr/lib/rabbitmq/lib/rabbitmq_server-2.3.1/plugins
root@server:~ wget http://www.rabbitmq.com/releases/plugins/v2.3.1/amqp_client-2.3.1.ez
root@server:~ wget http://www.rabbitmq.com/releases/plugins/v2.3.1/rabbit_stomp-2.3.1.ez

Step 7)
Start RabbitMQ

root@server:~ service rabbitmq-server start

Use external nodes to define puppet resources

A technique I like to use when making my puppet modules is to split out the data from the logic. This allows me to create site generic modules that work anywhere. This also allows me to change configurations on the fly without having to go through a dev/test/prod cycle with my puppet code. Extending on that, I often create virtual resources outside of my modules and then realize those resources in the classes. This way I can add/remove/modify resources per site and it 'just works.' For example, one manufacturing site might employ 5 lifters that are controlled by a server that is then managed by puppet. Another site might only have 2. Futhermore, if a new lifter is added to the second site, you don't want to have to modify your puppet classes.

To do this, I would do something like this:

site.pp:

import "resources/*.pp"

Then I would have a resource file named lifters.pp:



Then in my lifter class, I just do:

Lifter<||>

This works, but as you can imagine it gets ugly. What would be nicer is to be able to manage these in an external nodes classifer such as puppet-dashboard. Luckily with puppet 2.6, we can. We're going to make use of puppet's new pure ruby manifests. The class is going to take all the variables in puppet, including ones created by external nodes, find ones that are both hashes and have a key called :type, and turn those in to resources.

First we need to make a module called externalresources. In the manifests directory, we're going to create a init.rb file instead of a init.pp file. It's important that the init.pp file does not exist. In the init.rb file, we're going to put the following code:



In your site.pp file add this line:

include externalresources

Now we can define parameters in puppet-dashboard and have their values be hashes that look almost like puppet resources. The only difference is we're going to use symbols and we have to add a parameter called :type. Let's create a file resource.

In puppet-dashboard, create a parameter on a host you manage with the dashboard. Name the parameter the title of the file resource. For our example, /tmp/test. For the value, we'll create a hash that looks like this:

{ :type => 'file', :content => "My file's content", :ensure => 'present' }

Now when we run puppet on that host, the file /tmp/test will be created with the content My file's content. For extra fun, we can add the parameter :virtual => 'true' and externalresources will make it a virtual resource that can be realized in other classes.

Finding which LVM volume is assigned for XenServer VDI

There are times when you want to determine which LVM volume on a Storage Repository (SR) in XenServer is assigned to a VM. All you need to do this is the SR's uuid and the VDI's uud. The path for the logical volume is /dev/VG_XenStorage-${sr-uuid}/VHD-${vdi-uuid}. Here's how we get it.

NOTE: If you're using StorageLink on, for example, EqualLogic arrays, the name of the LUN is in the format XenStorage${sr-uuid}${vdi-uuid}


First log in to the xenserver as root. For this example, we're assuming our VM is named 'web01'.

Get the UUID of the VM with the following commond:

[root@xenserver ~]# xe vm-list name-label=web01 params=uuid
uuid ( RO) : 39ccc7b9-9a7b-e13c-d381-b5e81fca5394

List all the VBDs assigned to the VM and their associated VDIs using the VM's uuid.

[root@xenserver ~]# xe vbd-list vm-uuid=39ccc7b9-9a7b-e13c-d381-b5e81fca5394 params=vdi-uuid,vdi-name-label
vdi-uuid ( RO) : 9c124393-bd7d-4223-a208-dc38ae6afdad
vdi-name-label ( RO): web01-root


vdi-uuid ( RO) : 2f8af49b-2cc1-4af8-80f2-15bee4de67bf
vdi-name-label ( RO): web01-swap

Notice the VDI that is assigned as the root disk for the VM, take note of the vdi-uuid value. Now we can see what SR the VDI is assigned to using the vdi-uuid value.

[root@xenserver ~]# xe vdi-list uuid=9c124393-bd7d-4223-a208-dc38ae6afdad params=uuid,sr-uuid
uuid ( RO) : 9c124393-bd7d-4223-a208-dc38ae6afdad
sr-uuid ( RO): 720183de-96e7-9b38-a97d-102324838b97

Using the sr-uuid and vdi-uuid, we can determine which LVM volume the root disk of the VM is assigned to. The path is /dev/VG_XenStorage-${sr-uuid}/VHD-${vdi-uuid} which you can verify by listing all the LVM volumes.

[root@xenserver ~]# lvdisplay
--- Logical volume ---
LV Name /dev/VG_XenStorage-720183de-96e7-9b38-a97d-102324838b97/VHD-9c124393-bd7d-4223-a208-dc38ae6afdad
VG Name VG_XenStorage-720183de-96e7-9b38-a97d-102324838b97
LV UUID a38GYk-1Bzg-2Syu-3Rfn-BVYE-DaW2-cVhPuM
LV Write Access read/write
LV Status available
# open 1
LV Size 260.52 GB
Current LE 66692
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 251:1

Fixing a corrupt mobile account in Snow Leopard

It's rare, but sometimes mobile accounts (accounts that sync to an Active Directory/LDAP entry) can become corrupt and bad things happen as a result. Common symptoms are shells not being able to start, applications claiming you don't permission to do this-or-that, etc. I still haven't found a good reason for this happening, but have discovered the corruption exists in /Local/Default/Users/<user> in the local Directory Service on the mac. Since the data is synced to Active Directory/LDAP, the easiest thing to do is simply delete the entry in Directory Service using dscl. Then when you log in with the account (using the Other option in the Login Window,) it will create a new mobile account and take over the home directory for the old account that you DIDN'T delete. Since the uid is synced, the files in the home directory are owned by new mobile account. To delete the entry in Directory Service, run the following command replacing <user> with the username of the account to be removed.

dscl . -delete /users/<user>