NUMA and node interleaving

Chris Linton-Ford writes:

> "Enable Node Interleaving": initially disabled, when turned on the

> operating system booted, but I got the following error message:

> "MPO disabled because memory is interleaved"

You definitely don't want "Node Interleaving."  As a BIOS option, it

wins a special award for being the most misleading misfeature since

plug-n-play.

Opteron systems have memory attached to each CPU, but a global address

space.  When you access memory that's not local to your CPU, you incur

a penalty going over the internal "hypertransport" between the CPUs.

On OSes (such as Solaris) that are aware of NUMA hardware, that's not

a problem.  The OS automatically optimizes the allocation of memory

based on where the task is running (or vice-versa) so that you get

good locality.

On OSes that are ignorant of this sort of hardware, naive testing can

produce strange results.  You run your test once and it runs fast.

Run it again, and it's slow.  Run again, it's fast again.  It all

depends on where Wind^Wthat OS puts your application, and it's

unpredictable.

This is where "Node Interleaving" comes in.  Turning that misfeature

on causes memory to be mapped page-by-page in a round-robin fashion

among the CPUs instead of being contiguous.  In other words, every 4K

boundary talks to a different CPU in the system.  This averages out

the costs, making everything run equally poorly, and making your naive

OS and benchmarks seem to be stable.

The OS can't keep track of that sort of addressing mess, so it causes

MPO to be turned off.  You don't want this on Solaris.