Automatic Cluster Membership with Puppet

Overview

Recently, I was tasked with setting up a HA solution for our monitoring servers. I decided to use Linux HA and set up an active/passive cluster, one per environment. We use Puppet for configuring and deploying our services, so a Puppet module that would bootstrap, install, and configure multiple, different Linux HA clusters was needed. After looking at existing Puppet modules, I decided to write my own. The driving factors for writing my own Puppet module were that I wanted the cluster membership to be stored in a variable, so it could be used in templates or manifest logic. Also, I did not want to have to maintain the list manually.  Finally, I wanted other Puppet modules and nodes to be able to generate and use the membership list as well, for configuring things like Nagios checks or to enable metric collection. In this blog post, I am going to talk about a custom Puppet function I wrote that exploits stored configs and exported resources to automatically generate a list of nodes and assigns them to a variable.

Stored configs got me close to what I was looking for but did not get me all the way there. I was interested in the list of nodes themselves and not the resources they were using. Puppet provides the ‘common::concatfilepart’ function that would allow the hostname to be put into a file and then concatenated with other file fragments to generate a complete file. I felt, however, that breaking up files into fragments made it harder to read and maintain. Also, the list of nodes could not be used in the manifests. So I looked at the Puppet code for querying stored configs and wrote a custom Puppet function that would query the stored configs database and return a list of the nodes, rather than the resource. I used existing Puppet ruby methods rather than query the database directly to try and make the function portable to newer versions of Puppet.

Implementation

The first thing I tackled was identifying the resource to export and use it to determine each cluster’s membership. In our case, we were using pacemaker for the Linux HA cluster resource layer, so each cluster would have a unique cib.xml file. That file was also under Puppet’s control, so I decided to export that resource. Any node that had that file in its catalog would, by inclusion, be participating in the Linux HA cluster.  Within the Puppet class that installed and configured pacemaker I added:

class clusterlabs (
$cluster = 'undefined'

) {

@@file { "/var/lib/pacemaker/cib/${cluster}-${fqdn}-cib.xml":
ensure => file,
path => "/var/lib/pacemaker/cib/${cluster}-cib.xml",
mode => '0644',
owner => 'hacluster',
group => 'root',
tag => "$cluster",
content => template("clusterlabs/$cluster.cib.erb"),
}

...

} # end class clusterlabs

I did not want to overwrite the existing cib.xml of the running cluster so I wrote it to a different path. (The manifest has other logic to determine if the cluster is being bootstrapped, and if so, it copies the ${cluster}-${fqdn}-cib.xml into place.)  Since the file was being exported, the resource name had to be unique, so I included the host’s fully qualified domain name in filename. Now when a host included the class ‘clusterlabs’ and specified the cluster, the file resource would be exported and placed in the stored configs database.

Now that the resource was exported, I needed to write a Puppet function to query the stored configs database looking for that file, for a specified cluster, and return all the hosts that exported that
resource. What I ended up with was a function I called ‘clustermembership’ that took a cluster name as a parameter and looked for any file resource where the title was in the form of
“/var/lib/pacemaker/cib/${cluster}-cib.xml”

# Begin Code
require "puppet"
require "puppet/rails"
require "socket"
module Puppet::Parser::Functions
newfunction(:clustermembership, :type => :rvalue ) do |args|
theCluster = args[0]
class Hosts < Puppet::Rails::Host; end
printtype = "name"
query = "( resources.restype = \'File\' AND resources.title like \'/var/lib/pacemaker/cib/" + theCluster + "-%-cib.xml\')"
Hosts.find(:all,
:include => [ :resources ],
:conditions => query
).map { |host| host.send(printtype) }
# End Code

The final step was to bring everything together. The Puppet class ‘clusterlabs’ is used to configure the cluster messaging and resource layer. So I added the line

$nodes = clustermembership($cluster)

to the class. That would then populate the variable $nodes with a list of nodes that exported the cib.xml for the specified cluster. That variable could then be used within the templates to create the cluster configuration files. The template could be a single file, which has logic to iterate over the list and generates the correct lines in the correct place, making the configuration template
easy to read and follow. Other manifests and Puppet nodes could also get the list of nodes simply by calling the Puppet function and passing in the cluster in question.

Future Direction

Going forward, I hope to generalize the function more and add more validation and error handling. It works well for what we are currently using it for, but I see a future where whenever, for example, I add a new syslog server or Cassandra node, the clients will query and dynamically include the new servers, without having to explicitly tell them to. Just by having a node include a Puppet class, the right thing is done.

I also plan on looking at this work with Puppet DB going forward and removing the need to explicitly export resources to identify cluster membership.

 Continue the conversation by sharing your comments here on the blog.