PXE Booting SmartOS from Ubuntu Server 12.04.1 LTS

My experience in setting up Ubuntu Server 12.04.1 LTS as a PXE server to boot SmartOS. These instructions should continue to work with future SmartOS releases that keep the same naming conventions.

I have had many false starts on this road. A lot of it - in my opinion - comes down to overcomplicating a very simple system. I will simplify things to the best of my ability by removing the optional parameters that complicated the tutorials I used. I will use Ubuntu Server 12.04.1 LTS (64-bit) as my PXE server. I will be booting SmartOS (64-bit) from PXE on a Dell T3500 with a Broadcom UNDI PXE ROM. I will document the straight and narrow path that I have untangled from my troubleshooting.

All steps are assumed to be run from the superuser (typically root) shell. To get the superuser shell, run the following:

sudo su -l

WARNING: Serious network disruption can result if a DHCP server is run on an office network. Make sure it is okay to run a DHCP server on the segment your Ubuntu Server is on. Do not proceed without a sandbox network or guidance/approval from your colleagues in IT/Operations.

Creating a Generic iPXE Chainloaded Boot Server

A DHCP server and a TFTP server are required for PXE. Run the following commands to install necessary packages:

apt-get install isc-dhcp-server tftpd-hpa

The TFTP server provides the means to download the iPXE config and operating system image. It needs to be configured to allow relative paths (--secure) or absolute file errors will occur when PXE tries to download. The verbose option (--verbose) will log the files requested in /var/log/syslog. Create /etc/default/tftpd-hpa with the following content:

# /etc/default/tftpd-hpa
TFTP_OPTIONS="--secure --verbose"
TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/var/lib/tftpboot"
TFTP_ADDRESS="0.0.0.0:69"

Restart the TFTP server to take the new settings:

service tftpd-hpa restart

UNDI PXE ROMs cannot directly load iPXE configuration. Chainloading by loading undionly.kpxe works around this by bootstraping to iPXE and then having iPXE load smartos.ipxe.

The DHCP server provides IP configuration and walks the hosts through the iPXE chainloading process. My home subnet is 192.168.2.0/24 and 100-200 are not handled by another DHCP server. Create /etc/dhcp/dhcpd.conf with the following content (adjust the subnet, range and domain-name as required):

# /etc/dhcp/dhcpd.conf
ddns-update-style none;
option domain-name "example.com";
option domain-name-servers 208.67.222.222, 208.67.220.220;
default-lease-time 600;
max-lease-time 7200;
log-facility local7;
subnet 192.168.2.0 netmask 255.255.255.0 {
  pool {
    range 192.168.2.100 192.168.2.200;
    next-server 192.168.2.34;
    if exists user-class and option user-class = "iPXE" {
      filename = "smartos.ipxe";
    } else {
      filename = "undionly.kpxe";
    }
  }
}

DHCP needs to run on at least one NIC (aka interface). Create /etc/default/isc-dhcp-server with the following content:

# /etc/default/isc-dhcp-server
INTERFACES="eth0"

Restart the DHCP server:

service isc-dhcp-server restart

Ben Rockwood provides a version of undionly.kpxe on his site. Run the following to get the PXE chainload binaries in place:

cd /var/lib/tftpboot/
curl http://cuddletech.com/IPXE-100612_undionly.kpxe > undionly.kpxe
chown tftp undionly.kpxe
chgrp tftp undionly.kpxe

At this point a generic PXE boot server is complete. iPXE will still expect smartos.ipxe, but that can be created with whatever content is needed. For those interested in booting SmartOS, what follows are the steps to provide SmartOS boot services on this server.

Providing SmartOS PXE Boot Services

A template iPXE config is useful both upfront and when updating to new platform releases. Create /var/lib/tftpboot/smartos.ipxe.tpl with the following content (-B smartos=true is essential, otherwise logins will fail):

#!ipxe
# /var/lib/tftpboot/smartos.ipxe.tpl
kernel /smartos/$release/platform/i86pc/kernel/amd64/unix -B smartos=true
initrd /smartos/$release/platform/i86pc/amd64/boot_archive
boot

Having a place to store releases avoids clutter. Set up a home for SmartOS platform releases:

cd /var/lib/tftpboot/
mkdir smartos
chown tftp smartos
chgrp tftp smartos

Deploy/Update to the latest SmartOS platform release

The steps in this section work for both initial deployment and upgrades as Joyent releases them.

Next get the latest SmartOS platform and massage it into a workable shape for our iPXE config:

cd /var/lib/tftpboot/smartos/
curl https://download.joyent.com/pub/iso/platform-latest.tgz > /tmp/platform-latest.tgz
cat /tmp/platform-latest.tgz | tar xz
directory=`ls | grep platform- | sort | tail -n1`
release=${directory:9}
mv $directory $release
cd $release
mkdir platform
mv i86pc platform
chown -R tftp .
chgrp -R tftp .

Process the smartos.ipxe.tpl template to create/update the iPXE smartos.ipxe file:

cd /var/lib/tftpboot/
cat smartos.ipxe.tpl | sed -e"s/\$release/$release/g" > smartos.ipxe
chown tftp smartos.ipxe
chgrp tftp smartos.ipxe

Booting your soon to be SmartOS hosts

Make sure PXE boot is enabled and that it is the first in the boot sequence. On the Dell Precision T3500 this is under Settings | System Configuration | Integrated NIC in the BIOS. I had to set it to Enable with PXE. I also had to go to Settings | General | Boot Sequence and put the Onboard Network Controller first by selecting it and hitting PgUp until it was at the top.

Conclusion

From cold boot pressing the power button to iPXE is 30s. To SmartOS initial is 2m 5s and all the way to login prompt was 3 minutes. I have not compared this to USB boot times, but I am using a pretty weak 1Gbps D-Link consumer desktop switch.

Given these instructions you should be able to have a PXE booted SmartOS within an hour including installing Ubuntu Server 12.04.1 LTS under VirtualBox.

Thanks

Thank you to Canonical and the Ubuntu User Community for posting and maintaining the Ubuntu PXEInstallServer wiki page which was instrumental for me getting this working.

Big thanks to Ben Rockwood for creating and maintaining the PXE Booting SmartOS wiki page which gave me all the steps necessary to do this at a high level.  Without that article I probably would have had a very hard time getting this working.

blog comments powered by Disqus