Server Provisioning Automation – Part 1

In the next few blog posts I’ll describe how we automated several parts of our server provisioning process.  In this first blog post of the series I cover the pre-provisioning phase, in which servers are prepared so that they can be remotely accessed and kickstarted.  Next, I’ll illustrate the provisioning phase and show how we use Puppet to automatically kickstart machines, both physical and virtual.  The setup is completed by the system configuration phase, which is mostly a matter of running Puppet on the host.

Our Environment

While most of this post describes how we automated the out-of-band interface configuration, I should note a couple of points that determined the configuration details below.  First, most of our 1u servers only use only one Ethernet cable.  This was a deliberate decision we’ve made:  it greatly simplifies cable and switch management and lowers cost by requiring less networking hardware. The out-of-band and switch port configurations take the one-cable constraint into account.

Next, this setup is further made simple due to most of our machines being from a single hardware vendor.  We use mostly Dell hardware and therefore refer to the out-of-band management interface as iDRAC (integrated Dell Remote Access Card).  The infrastructure described is therefore iDRAC-centric, but a similar process could be used for any vendor, as long it provides configuration tools that can be run from the OS.

The Challenge

When an order of servers arrives, the Datacenter staff will rack them, wire them with power and networking, configure the out-of-band interface, and finally hand over the inventory of the new equipment, including serial numbers and rack location to our Datacenter Managers, who in turn put the data in an inventory system.  One way of configuring an out-of-band interface is to do it manually, on each server, before the machine boots, and from the console, similarly to how one can configure the BIOS.   When, in the face of explosive growth, we started tackling server-provisioning automation it became quickly clear that this method would not scale. We needed a easy process for the Datacenter staff to follow with as little room for error as possible.

Fortunately, our hardware vendor provides a set of Linux management tools for configuring the iDRACs from a running OS.  We’ve taken advantage of this and built a minimal Linux live in-memory environment with the Dell tools necessary to configure an iDRAC.  To make it crazy easy for the Datacenter guys we’ve packaged it all up into a PXE bootable image, so that all they need to do after racking is power on the machines and push F12 to force them to PXE boot.  I’ll describe the PXE boot image, its creation process, and what stars need to align as pre-requisites before this all works in some detail below.

Configuring the Out-of-Band Interface

After the machine PXE boots and the OS and Dell utilities are loaded, a simple startup script configures the iDRAC:  the network port on the host is switched to shared mode (we’re using only one Ethernet cable per machine), VLANs are turned on, the appropriate VLAN ID is configured, and DHCP is enabled.   The iDRAC then resets with the new settings and obtains an IP number from the iDRAC DHCP server.  DHCP, in turn, interacts with dynamic DNS, which registers iDRAC names.   Typically, an iDRAC registers a name like idrac-SERIAL, but we’ve found that some older iDRAC versions use rac-SERIAL, so in the iDRAC configuration script we force a consistent name based on the serial number of the host.

This provides an easy way to identify each machine’s iDRAC by the machine’s serial number.  In order to have a more human-friendly name that can be used to access the iDRAC a DNS alias is also created, e.g. idrac.HOSTNAME.DOMAIN -> idrac-SERIAL.IDRACDOMAIN.  This happens in the kickstart phase of provisioning when the host name is known and assigned to the machine.

The PXE Boot Image

We create the PXE boot image by starting with a minimal kickstart configuration.  It just installs a base CentOS, the minimal Dell OpenManage RPMs needed to configure an iDRAC and any pre-requisites they need.  The image starts the Dell services upon boot and executes /etc/rc.local.  The script enables the iDRAC to use VLANs, sets the appropriate VLAN ID, and enables DHCP.  Finally, it prints out the obtained network information and serial number to /etc/issue so it’s available right on the console.

Here’s a sample kickstart configuration:

### Begin aleks-livecd-racadm-p2.ks

lang en_US.UTF-8

keyboard us

timezone US/Eastern

auth --useshadow --enablemd5

selinux --disabled

firewall --disabled

rootpw --iscrypted

repo --name=a-base    --baseurl=http://mirror.centos.org/centos/5/os/$basearch

repo --name=a-updates --baseurl=http://mirror.centos.org/centos/5/updates/$basearch

repo --name=a-live    --baseurl=http://www.nanotechnologies.qc.ca/propos/linux/centos-live/$basearch/live

%packages --excludedocs

bash

kernel

syslinux

passwd

policycoreutils

chkconfig

authconfig

rootfiles

comps-extras

xkeyboard-config

wget

yum

OpenIPMI

vim-minimal

vim-common

%post --interpreter /bin/bash

# Set up Dell repo

# OMSA_6.2 is known to work; the latest version, 6.3, does not work as of Nov 10, 2010.

#OMSAVER="latest"

OMSAVER="OMSA_6.2"

/usr/bin/wget -q -O /var/tmp/dellrepos.sh http://linux.dell.com/repo/hardware/${OMSAVER}/bootstrap.cgi

chmod 700 /var/tmp/dellrepos.sh

/var/tmp/dellrepos.sh

/bin/rm -f /var/tmp/dellrepos.sh

# Install racadm

/usr/bin/yum -y install srvadmin-idracadm

RACADM="/opt/dell/srvadmin/idrac/bin/racadm"

#RACADM="/opt/dell/srvadmin/bin/idracadm"

cat <<__HERE_DOC_ENDS_HERE__ > /usr/bin/querydrac

#!/bin/bash

export LD_LIBRARY_PATH=/opt/dell/srvadmin/idrac/libs

echo "iDRAC Network Settings:"

${RACADM} getniccfg | head -8

echo -n "cfgNicSelection = "

${RACADM} getconfig -g cfgLanNetworking -o cfgNicSelection

echo -n "cfgNicVLanID = "

${RACADM} getconfig -g cfgLanNetworking -o cfgNicVLanID

echo -n "cfgNicVLanEnable = "

${RACADM} getconfig -g cfgLanNetworking -o cfgNicVLanEnable

echo -n "cfgNicUseDHCP = "

${RACADM} getconfig -g cfgLanNetworking -o cfgNicUseDHCP

echo -n "cfgDNSServersFromDHCP = "

${RACADM} getconfig -g cfgLanNetworking -o cfgDNSServersFromDHCP

echo -n "cfgIpmiLanEnable = "

${RACADM} getconfig -g cfgIpmiLan -o cfgIpmiLanEnable

echo " "

echo -n "Dell Service Tag: "

${RACADM} getsvctag

__HERE_DOC_ENDS_HERE__

chmod 755 /usr/bin/querydrac

cat <<__HERE_DOC_ENDS_HERE__ >> /etc/rc.local

echo "Configuring iDRAC:"

export LD_LIBRARY_PATH=/opt/dell/srvadmin/idrac/libs

# iDRAC configuration - order matters!

# cfgNicSelection

# 0 = Shared

# 1 = Shared with Failover LOM2

# 2 = Dedicated

# 3 = Shared with Failover All LOMs (iDRAC6 Enterprise only)

echo -n "Setting cfgNicSelection... "

${RACADM} config -g cfgLanNetworking -o cfgNicSelection 3

sleep 1

echo -n "Setting cfgNicVLanID... "

${RACADM} config -g cfgLanNetworking -o cfgNicVLanID

sleep 1

echo -n "Setting cfgNicVLanEnable... "

${RACADM} config -g cfgLanNetworking -o cfgNicVLanEnable 1

sleep 1

echo -n "Setting cfgNicUseDHCP... "

${RACADM} config -g cfgLanNetworking -o cfgNicUseDHCP 1

sleep 1

echo -n "Setting cfgDNSServersFromDHCP... "

${RACADM} config -g cfgLanNetworking -o cfgDNSServersFromDHCP 1

sleep 1

echo -n "Setting cfgIpmiLanEnable... "

${RACADM} config -g cfgIpmiLan -o cfgIpmiLanEnable 1

sleep 1

echo -n "Setting cfgUserAdminPassword... "

${RACADM} config -g cfgUserAdmin -o cfgUserAdminPassword -i 2 redacted

sleep 1

${RACADM} racreset

#/usr/sbin/shutdown -h now

SLEEPTIME=90

echo "Sleeping ${SLEEPTIME} seconds while the iDRAC resets..."

sleep ${SLEEPTIME}

/usr/bin/querydrac > /etc/issue

echo "" >> /etc/issue

__HERE_DOC_ENDS_HERE__

### End minimal-drac-live.ks

From this, we build a CentOS live ISO image and then convert it into a PXE bootable image, which we install on our build server.  The livecd-creator and livecd-iso-to-pxeboot utilities used for this are provided by the livecd-tools RPM.

LANG=C /usr/bin/livecd-creator --config=aleks-livecd-racadm-p2.ks --fslabel=aleks-livecd-p2

/usr/bin/livecd-iso-to-pxeboot aleks-livecd-p2.iso

This will produce a tftpboot directory that contains everything needed to set up PXE boot on the build server.

Build Server Preparation

Build servers are a key part in our provisioning infrastructure.  They’re used to PXE boot new hosts and kickstart them.  They are also used to PXE boot the live CentOS environment described above.  We do this by setting up a default entry in the PXE boot menu – when no MAC address for the server is found and no PXE menu option is selected from the PXE boot menu, the host will automatically boot the live CentOS environment and configure the machine’s iDRAC.  This is safe even when run multiple times on the same host.

Example /tftpboot/linux-install/pxelinux.cfg/default that uses the image made above:

default aleks-live

timeout 200

[…]

label aleks-live

kernel aleks-live/vmlinuz0

append initrd=aleks-live/initrd0.img root=/aleks-livecd-p2.iso rootfstype=iso9660 rootflags=loop

[…]

Network Prerequisites

In order for all of the above to work there is some network configuration needed.  First, the iDRACs are homed to one of our iDRAC VLANs.  These are separate form any production or externally accessible network. DHCP and dynamic DNS are used in these VLANs to assign IP addresses to iDRACs and register them in an iDRAC DNS zone.  DNS delegation is also set up to facilitate DNS lookups of iDRACs from management networks.  DNS zone replication is configured so that iDRAC zone information is available from multiple DNS servers.

Second, all switch ports for newly racked servers are preconfigured in advance to be in what we call “the build network.”  This network is only used for kickstarting machines; when a machine is later kickstarted, the switch port is configured for the correct VLAN depending on the type and function of a host.  More on that in the next blog post.

In addition to being in the build network, all switch ports are also preconfigured for the iDRAC VLAN.  We do this with VLAN tagging; each port is thus typically in native mode for the main interface and uses VLAN tagging for the iDRAC.  Again, this is due to our choice to only use one network cable per machine.

Here are sample Cisco and Juniper switch port configurations.   One important fact to note is the spanning-tree portfast trunk setting.  If this is not configured, the connection to a remote console may be unstable while the machine is booting.

Cisco:

In the example below:

1001 is the host interface VLAN ID

555 is the iDRAC VLAN ID

666 is a VLAN ID of a non-existing network

cisco-switch#show run int Gi1/40

Building configuration...

Current configuration : 295 bytes

!

interface GigabitEthernet1/40

description c1-dellr410:eth0:===:B2-16:Gi1/40

switchport access vlan 666

switchport trunk native vlan 1001

switchport trunk allowed vlan 555,1001

switchport mode trunk

logging event link-status

logging event trunk-status

spanning-tree portfast trunk

end

Juniper:

user@juniper-switch# show interfaces ge-2/0/23

description p2-dellr410:eth0:===:B2-17:ge-2/0/23;

unit 0 {

family ethernet-switching {

apply-groups access-port-drac;

port-mode trunk;

native-vlan-id P2_QA_Eucalyptus;

}

}

user@juniper-switch# set protocols vstp vlan P2_idrac interface ge-2/0/23.0 edge

user@juniper-switch# set ethernet-switching-options bpdu-block interface ge-2/0/23.0

user@juniper-switch# show groups access-port-drac

interfaces {

<*> {

hold-time up 0 down 2000;

unit 0 {

family ethernet-switching {

port-mode trunk;

vlan {

members P2_idrac;

}

}

}

}

}

Quality Control

While developing this process, it became clear we needed an easy way for Datacenter Managers to verify the pre-kickstart phase is completed correctly on newly racked servers.  The goal is to catch defective hardware, wiring, or a missed step by the Datacenter guys who racked a cabinet or two full of machines.

As a quick visual feedback, machines that PXE booted and ran the iDRAC configuration display on the console the networking information the iDRAC obtained from the DHCP server along with the server’s serial number.  This serves as an indication the machine has successfully obtained an address for its iDRAC.  The Datacenter staff does this step – if a host shows an iDRAC with IP address 0.0.0.0 they know something’s gone wrong.  They then check the cabling and reboot with F12 again.

Another way our Datacenter Managers can verify if all the newly racked iDRACs are functioning is by running a script that takes as an argument a file with a list of serial numbers.  Since an iDRAC address is based on the machine’s serial number, the script is able to verify each one.   Should one serial number not check out, the Datacenter Managers can identify its location based on their inventory data and troubleshoot further.

Lessons Learned and Future

The infrastructure and process described has worked very well for us for over 1.5 years.  One issue we did run into earlier this year in one of our iDRAC VLANs is that we ran out of IPs in the assigned DHCP range.  Explosive growth is a good problem to have!  We simply added a new network range to DHCP and set up monitoring to alert when we’re low on available IPs.

Another problem is mistyped serial numbers.   Sometimes Z’s are transcribed as 5’s or vice versa.  If we’re lucky the QA step catches that, but sometimes it may not.  We’re looking into eliminating any manual input in the process; bar code scanners should help with this one.

The scope of the pre-kickstart phase was intentionally kept small and simple.  We are, however, now looking at extending and generalizing the pre-kickstart phase to include automatic BIOS updates and inventory registration that ties into the provisioning process.  In addition to exploring this homegrown solution, we’re considering solutions like Razor from Puppet Labs and EMC.

Conclusion

In this blog post I described the pre-kickstart configuration process, the setup that allows us to prepare hosts for remote access and kickstarting.  With this infrastructure and configuration in place, each host is then ready for the provisioning phase, which I’ll cover next time.