SmartOS on Multi-Socket and the ACPI SRAT

The ACPI SRAT describes the topology and nature of the processors and memory in your system. With it SmartOS can perform dynamic optimizations of memory allocation in a multicore, multiprocessor (multi-socket) environment.

I have been researching SmartOS intently recently.  One of the things that initially concerned me was a warning on startup about being unable to parse the ACPI SRAT Table.  This seemed a little arcane at the time, but like a doctor telling you literally what you have and not some folksy generalized term SmartOS is very clear what is wrong.

Here is what SmartOS was telling me on startup:

WARNING: Couldn't read ACPI SRAT table from BIOS. lgrp support will be limited to one group.

Without this SmartOS is not going to be able to make effective use of multiple sockets.  It is extremely important to get the ACPI SRAT available to get the most out of the servers I have at my disposal.

With the excellent help of the experienced folks on #smartos on IRC (specifically Scott McWhirter aka konobi of Cloudtone) I learned that NUMA and ACPI SRAT conflict.  Scott even pointed me directly to the BIOS option in question: Node Interleaving (which is another way of saying NUMA).

In short you need to disable NUMA/Node Interleaving in your BIOS to get the ACPI SRAT unless the BIOS has a specific option to enable ACPI SRAT.  You really aren't disabling NUMA, you are allowing the OS to handle it.

There is an excellent write-up on the ACPI SRAT table on Tech ARP.  They say it much better than I can :)