Matthew racked his first 8-GPU HGX H100 node, wired the power, cabled the network, and hit the button. The fans spun, the BIOS posted, Ubuntu came up clean. Then he ran nvidia-smi and every GPU showed up, so he launched a container and got a CUDA initialization error on eight healthy GPUs. Nothing was broken. The Fabric Manager service was not running, so the NVSwitch fabric never came up, and without that an HGX node has GPUs it cannot actually use together.
Standing up a multi-GPU AI server is not hard, but the failure modes are unforgiving and most of them happen before you ever run a workload. This is the practical first-boot sequence: the one fork that changes everything, the BIOS settings that decide whether the GPUs even appear, the driver and CUDA stack in the right order, how to verify the fabric is real, and the handful of mistakes that catch almost everyone. It mirrors our ASIC miner setup guide for the AI side of the rack.
Standing up a multi-GPU AI server is not hard, but the failure modes are unforgiving and most of them happen before you ever run a workload. This is the practical first-boot sequence: the one fork that changes everything, the BIOS settings that decide whether the GPUs even appear, the driver and CUDA stack in the right order, how to verify the fabric is real, and the handful of mistakes that catch almost everyone. It mirrors our ASIC miner setup guide for the AI side of the rack.
The short version
- First determine which server you have: PCIe multi-GPU (2 to 8 cards, no Fabric Manager) or an SXM / HGX baseboard (8-GPU, NVSwitch, Fabric Manager mandatory).
- In the BIOS, Above 4G Decoding and Resizable BAR must be on or the GPUs will not initialize. Set PCIe to Gen5 and boot UEFI.
- Install in order: OS, then nvidia-open driver, then CUDA Toolkit separately, then Fabric Manager (HGX only), then the Container Toolkit. Verify at each step.
- On any NVSwitch system, Fabric Manager version must match the driver exactly, or CUDA will fail to launch even though every GPU is visible.
- The fastest path for most teams is a pre-built, tested, burned-in server, deployed into hosting so the power and cooling are handled.
First, know which machine you have
Every decision downstream depends on this, so settle it before you touch anything. Multi-GPU servers come in two fundamentally different shapes, and the setup diverges completely.
PCIe multi-GPU is the common shape: two to eight cards in PCIe 5.0 slots, the kind of build using RTX 5090 blowers, RTX PRO 6000, or A100 PCIe. The GPUs talk over PCIe, or over NVLink bridge pairs on cards that support them. There is no NVSwitch and no Fabric Manager. Real-world scaling lands around 60 to 80 percent, and standard air cooling with blower cards handles the heat. This is where most pre-built GPU servers in the 2-to-8 GPU range sit.
SXM / HGX is the dense shape: eight H100, H200, or B200 GPUs mounted to a shared baseboard, fully connected by NVLink through NVSwitch. The interconnect is all-to-all, scaling reaches 80 to 90 percent, and the newest B200 baseboards are liquid-cooled and ship only in SXM form. The catch is that Fabric Manager is mandatory: it configures the NVSwitch fabric into one memory domain, and without it the GPUs register but CUDA will not initialize, per NVIDIA's Fabric Manager user guide. If you are still choosing between these, our H100 vs H200 vs B200 comparison and the GPU chooser guide cover the hardware decision; this guide assumes the box is already on your bench.
SXM / HGX is the dense shape: eight H100, H200, or B200 GPUs mounted to a shared baseboard, fully connected by NVLink through NVSwitch. The interconnect is all-to-all, scaling reaches 80 to 90 percent, and the newest B200 baseboards are liquid-cooled and ship only in SXM form. The catch is that Fabric Manager is mandatory: it configures the NVSwitch fabric into one memory domain, and without it the GPUs register but CUDA will not initialize, per NVIDIA's Fabric Manager user guide. If you are still choosing between these, our H100 vs H200 vs B200 comparison and the GPU chooser guide cover the hardware decision; this guide assumes the box is already on your bench.
Before you power on
The physical layer is where a bad first boot is usually born. Work through it deliberately.
- Power delivery. An 8-GPU node draws well past 10 kW, so it needs the right circuit: 208 to 240V, often three-phase, with redundant PSUs sized above peak draw, not at it. Every dual-slot GPU needs its own seated PCIe power connectors, not a daisy chain.
- Stage the power-on. Dense nodes have a large inrush at start. Bring the server up on a circuit that can absorb it rather than one shared with other live load, or you will trip a breaker on the first boot.
- Cooling and airflow. Blower cards exhaust out the back and belong in a closed chassis with front-to-back airflow. Open-air cards need real space. SXM and B200 baseboards need the liquid loop connected and flowing before you push load. Getting the room right is its own topic, tied to the power that makes AI sites viable. When provisioning 10-plus kW and the cooling to match is not practical on site, AI GPU hosting, or colocation, hands that problem to a facility built for it.
- Seat everything twice. Reseat each GPU, each power connector, and the baseboard cables. A card that is 1 mm proud of the slot is a card that will not appear in nvidia-smi.
The BIOS settings that make or break it
More first boots fail here than anywhere else, and the fixes are all toggles. Before installing an OS, enter firmware setup and set the following:
- Above 4G Decoding: ON. Non-negotiable for multi-GPU. Without it the system cannot map the large memory windows the GPUs expose, and they will fail to initialize or not appear at all.
- Resizable BAR: ON. Pairs with Above 4G Decoding and is required for correct GPU memory mapping on modern cards.
- PCIe link speed: Gen5. Force it rather than leaving it on Auto, which sometimes negotiates down and quietly halves your bandwidth to each card.
- Boot mode: UEFI, CSM disabled. Legacy compatibility mode interferes with large-BAR mapping and modern GPU init.
- IOMMU and ACS: on, if you will virtualize. Needed for VFIO passthrough and multi-tenant partitioning. Leave off for a single bare-metal box to avoid a small performance tax.
- Performance / latency-sensitive profile. If your vendor exposes a max-performance or latency-sensitive mode, enable it, and disable deep C-states for consistent throughput.
The software stack, in order
Install bottom-up and verify each layer before moving to the next. Debugging one broken layer is easy; debugging three at once is not.
1. Operating system. Ubuntu Server LTS is the default for AI work and what NVIDIA's stack targets first. Install it, get to a shell, confirm networking.
2. NVIDIA driver. Install the open kernel driver. On B200 the open variant is required; on everything else it is the recommended path for data-center GPUs.
sudo apt install nvidia-open # then: sudo update-initramfs -u && sudo reboot
After reboot, nvidia-smi should list every GPU. Note that the CUDA version shown there is the highest the driver supports, not the toolkit you have installed.
3. CUDA Toolkit, separately. The driver alone does not give you nvcc or the CUDA libraries. Install the toolkit matched to your driver and framework using NVIDIA's CUDA documentation and the version matrices. Avoid the generic cuda-drivers meta-package if you want to keep the open kernel modules intact.
nvcc --version # confirm the toolkit is installed and matches
4. Fabric Manager, HGX only. On any NVSwitch baseboard, install Fabric Manager at a version that exactly matches the driver, then enable and start the service. Skip this entirely on PCIe systems, where it is neither needed nor wanted.
5. NVIDIA Container Toolkit. If you run workloads in Docker, which most teams do, install the Container Toolkit so containers see the GPUs, then pull framework images from NGC that already bundle CUDA, cuDNN, and the runtime.
2. NVIDIA driver. Install the open kernel driver. On B200 the open variant is required; on everything else it is the recommended path for data-center GPUs.
sudo apt install nvidia-open # then: sudo update-initramfs -u && sudo reboot
After reboot, nvidia-smi should list every GPU. Note that the CUDA version shown there is the highest the driver supports, not the toolkit you have installed.
3. CUDA Toolkit, separately. The driver alone does not give you nvcc or the CUDA libraries. Install the toolkit matched to your driver and framework using NVIDIA's CUDA documentation and the version matrices. Avoid the generic cuda-drivers meta-package if you want to keep the open kernel modules intact.
nvcc --version # confirm the toolkit is installed and matches
4. Fabric Manager, HGX only. On any NVSwitch baseboard, install Fabric Manager at a version that exactly matches the driver, then enable and start the service. Skip this entirely on PCIe systems, where it is neither needed nor wanted.
5. NVIDIA Container Toolkit. If you run workloads in Docker, which most teams do, install the Container Toolkit so containers see the GPUs, then pull framework images from NGC that already bundle CUDA, cuDNN, and the runtime.
Ready to Start Mining?
Free worldwide DDP shipping. Professional hosting from $0.055/kWh.
Verify the fabric before you trust it
Every GPU showing up in nvidia-smi is necessary but not sufficient. On an HGX node especially, confirm the interconnect is actually live.
Confirm fabric registration (HGX). On NVSwitch systems, check that the GPUs finished registering with the fabric. Anything other than Completed / Success means CUDA will not run correctly.
nvidia-smi -q -i 0 | grep -A 2 Fabric # want: State: Completed, Status: Success
Read the topology. This one command shows how every GPU connects to every other, plus the CPU and NUMA affinity you need for placement. NV# entries are NVLink, PXB is a PCIe bridge, SYS crosses the CPU sockets.
nvidia-smi topo -m
Check the NVLinks. Confirm no link is inactive or throwing errors.
nvidia-smi nvlink -s
Respect NUMA affinity. On dual-socket servers, a workload pinned to the wrong socket pays a real bandwidth penalty. Use the affinity columns from nvidia-smi topo -m to pin processes and memory to the socket local to their GPUs.
Measure real bandwidth. Run the standard NCCL tests to confirm you are getting genuine GPU-to-GPU bandwidth, not a fabric that came up degraded, using NVIDIA's NCCL documentation.
Confirm fabric registration (HGX). On NVSwitch systems, check that the GPUs finished registering with the fabric. Anything other than Completed / Success means CUDA will not run correctly.
nvidia-smi -q -i 0 | grep -A 2 Fabric # want: State: Completed, Status: Success
Read the topology. This one command shows how every GPU connects to every other, plus the CPU and NUMA affinity you need for placement. NV# entries are NVLink, PXB is a PCIe bridge, SYS crosses the CPU sockets.
nvidia-smi topo -m
Check the NVLinks. Confirm no link is inactive or throwing errors.
nvidia-smi nvlink -s
Respect NUMA affinity. On dual-socket servers, a workload pinned to the wrong socket pays a real bandwidth penalty. Use the affinity columns from nvidia-smi topo -m to pin processes and memory to the socket local to their GPUs.
Measure real bandwidth. Run the standard NCCL tests to confirm you are getting genuine GPU-to-GPU bandwidth, not a fabric that came up degraded, using NVIDIA's NCCL documentation.
The failures that catch almost everyone
Nearly every stuck first boot traces back to this list:
- Above 4G / Resizable BAR left off. GPUs missing from nvidia-smi or failing to init. The most common single cause.
- Fabric Manager missing on an NVSwitch box. Every GPU visible, but CUDA will not launch. Matthew's exact problem. (And never install it on a PCIe box.)
- Fabric Manager version does not match the driver. Cryptic CUDA init failures and missing NVLink. The two versions must be identical.
- Wrong driver variant. B200 requires the open driver; the proprietary path will not bring it up.
- The cuda-drivers meta-package. It can pull in modules that break the open kernel stack. Install the driver and the toolkit as separate, matched pieces.
- PCIe power or lanes. An unseated connector or a card in a lane-starved slot shows as a missing or underperforming GPU.
Prove it under load, then run something
A node that idles clean can still throttle or trip under a real workload. Push it before you hand it off.
- Watch thermals and power live. Stream per-GPU temperature, utilization, and power draw while you load the box. nvidia-smi dmon -s pucm # live power, util, clocks, memory
- Set sane power limits. On dense nodes, capping power per GPU can improve stability and total throughput. Set and confirm the limit rather than assuming the default fits your circuit.
- sudo nvidia-smi -pl <watts>
- Soak it. Run a sustained load (a GPU burn-in and the NCCL all-reduce tests) long enough to reach thermal steady state, and watch for throttling or link errors.
- First real inference. Launch a container and serve a model, or run a short distributed training smoke test. If throughput scales roughly with GPU count, the fabric is doing its job. Sanity-check the numbers against the GPU and AI benchmarks so you know what good looks like for your cards.
The shortcut most teams take
Everything above is doable, and worth understanding even if you never do it by hand. But for most teams the honest calculation is that a day of BIOS toggles, driver matching, and fabric debugging is not the best use of expensive engineers, especially when a single version mismatch can cost an afternoon.
That is the case for buying the server already built. Our pre-built GPU servers and AI workstations ship assembled, tested, and burned in, with the component compatibility, power delivery, PCIe lane allocation, and stability validation already done, from 2-GPU towers through HGX H100 and H200 8-GPU nodes to GB200 NVL72 rack-scale systems. You can pair them with the right GPUs from the same catalog and see the full AI hardware range in one place.
The other half of the shortcut is where it runs. A configured server still needs the 10-plus kW per rack, the cooling, and the power rate that decide whether the economics work, which is exactly the power moat driving the public-miner AI pivot. This is the case for AI GPU hosting (colocation): deploying into professional hosting at $0.07 to $0.08 per kWh, with remote management, means the node is racked, powered, cooled, and monitored without you standing in a data center. If you are weighing the silicon itself, the NVIDIA vs AMD vs Intel comparison covers that trade-off.
The other half of the shortcut is where it runs. A configured server still needs the 10-plus kW per rack, the cooling, and the power rate that decide whether the economics work, which is exactly the power moat driving the public-miner AI pivot. This is the case for AI GPU hosting (colocation): deploying into professional hosting at $0.07 to $0.08 per kWh, with remote management, means the node is racked, powered, cooled, and monitored without you standing in a data center. If you are weighing the silicon itself, the NVIDIA vs AMD vs Intel comparison covers that trade-off.
The bottom line
Matthew's server was fine the whole time. One service, matched to one driver version, stood between eight idle GPUs and a working node, and that is the pattern of multi-GPU setup in a sentence: the hardware is rarely the hard part, the sequence is. Know which machine you have, set Above 4G and Resizable BAR, build the stack bottom-up, and verify the fabric before you trust it.
Do that and an 8-GPU node goes from racked to first inference in an afternoon. Skip a step and you will spend that afternoon reading CUDA error codes. When you would rather the box arrive already configured and tested, start with our GPU servers, or have them deployed into hosting so the power and cooling are someone else's job.
Do that and an 8-GPU node goes from racked to first inference in an afternoon. Skip a step and you will spend that afternoon reading CUDA error codes. When you would rather the box arrive already configured and tested, start with our GPU servers, or have them deployed into hosting so the power and cooling are someone else's job.
Frequently asked questions
Do I need Fabric Manager for a multi-GPU server?
Only on NVSwitch-based systems, meaning HGX and DGX 8-GPU baseboards with H100, H200, or B200 GPUs. There, Fabric Manager is mandatory: it configures the NVSwitch fabric, and without it CUDA will not initialize even though every GPU appears in nvidia-smi. On PCIe multi-GPU servers (2 to 8 cards in slots), Fabric Manager is not used at all. Its version must exactly match the installed driver.
Why do my GPUs not show up in nvidia-smi?
The most common cause is BIOS: Above 4G Decoding or Resizable BAR is disabled, so the system cannot map GPU memory and the cards fail to initialize. Enable both, set PCIe to Gen5, and boot in UEFI mode with CSM disabled. Other causes are an unseated PCIe power connector, a card not fully seated in its slot, or a GPU in a lane-starved slot. Reseat the hardware and recheck the firmware settings first.
What driver should I install for an AI server?
Install the NVIDIA open kernel driver (nvidia-open) for data-center GPUs; on B200 the open variant is required. Install the CUDA Toolkit separately and matched to the driver, rather than using the generic cuda-drivers meta-package, which can break the open kernel module stack. Verify with nvidia-smi for the driver and nvcc --version for the toolkit, and pick the CUDA and framework versions together from NVIDIA's support matrices.
How do I verify NVLink and GPU topology?
Run nvidia-smi topo -m to see how every GPU connects to every other and to the CPU, where NV entries are NVLink, PXB is a PCIe bridge, and SYS crosses sockets. It also shows CPU and NUMA affinity for placement. Use nvidia-smi nvlink -s to confirm no link is inactive or in error, and on HGX systems check nvidia-smi -q for a Fabric State of Completed. Then run the NCCL tests to measure real bandwidth.
What BIOS settings does a multi-GPU server need?
Enable Above 4G Decoding and Resizable BAR, both required for multi-GPU memory mapping. Force PCIe to Gen5 instead of Auto. Boot in UEFI with CSM disabled. Enable IOMMU and ACS only if you plan to virtualize or partition GPUs, and turn on a max-performance or latency-sensitive profile with reduced deep C-states for consistent throughput. These firmware settings decide whether the GPUs even appear before any driver is involved.
How much power and cooling does an 8-GPU server need?
A dense 8-GPU node draws well past 10 kW, so it needs a 208 to 240V circuit, often three-phase, with redundant PSUs sized above peak draw and a plan for inrush at power-on. Blower cards want front-to-back airflow in a closed chassis; SXM and B200 baseboards need a connected liquid loop. Because per-rack power and cooling at this density are hard to provision on site, many operators deploy into professional hosting instead.
What is the difference between a PCIe and an SXM (HGX) GPU server?
PCIe servers hold 2 to 8 GPUs in expansion slots that communicate over PCIe or NVLink bridge pairs, need no Fabric Manager, and scale around 60 to 80 percent. SXM/HGX servers mount 8 GPUs on a shared baseboard fully connected by NVLink through NVSwitch, require Fabric Manager, scale 80 to 90 percent, and at the B200 generation are liquid-cooled and SXM-only. The setup process differs mainly in the Fabric Manager step and the cooling requirements.
Can I set up a multi-GPU AI server myself, or should I buy it pre-built?
You can do it yourself; the process is a defined sequence of BIOS settings, a bottom-up driver and CUDA stack, and fabric verification. But version matching and firmware traps make it easy to lose a day, so many teams buy the server pre-built, tested, and burned in, then deploy it into hosting where power and cooling are handled. Pre-built removes the compatibility, power-delivery, and stability work; hosting removes the facility problem.
What operating system is best for a multi-GPU AI server?
Ubuntu Server LTS is the standard choice and what NVIDIA's driver and CUDA stack target first, with the widest framework and container support. Install it, confirm it boots to a shell and has networking, then layer the NVIDIA driver, CUDA Toolkit, Fabric Manager where applicable, and the Container Toolkit on top. Always confirm the specific OS, driver, CUDA, and framework combination against the vendor support matrices before committing a production build.
Sources and image credits
Technical procedure grounded in NVIDIA's CUDA Toolkit documentation, Fabric Manager user guide, data-center driver documentation, and NCCL documentation. Hero photograph of the Sierra GPU supercomputer by the U.S. Department of Energy (public domain) and multi-GPU rig photograph by BluberdeBlubBlub (CC BY 4.0), both via Wikimedia Commons, cropped and graded for MillionMiner. Diagrams: original MillionMiner graphics. Verify OS, driver, CUDA, and framework versions against current vendor support matrices before a production build.
Only on NVSwitch-based systems, meaning HGX and DGX 8-GPU baseboards with H100, H200, or B200 GPUs. There, Fabric Manager is mandatory: it configures the NVSwitch fabric, and without it CUDA will not initialize even though every GPU appears in nvidia-smi. On PCIe multi-GPU servers (2 to 8 cards in slots), Fabric Manager is not used at all. Its version must exactly match the installed driver.
Why do my GPUs not show up in nvidia-smi?
The most common cause is BIOS: Above 4G Decoding or Resizable BAR is disabled, so the system cannot map GPU memory and the cards fail to initialize. Enable both, set PCIe to Gen5, and boot in UEFI mode with CSM disabled. Other causes are an unseated PCIe power connector, a card not fully seated in its slot, or a GPU in a lane-starved slot. Reseat the hardware and recheck the firmware settings first.
What driver should I install for an AI server?
Install the NVIDIA open kernel driver (nvidia-open) for data-center GPUs; on B200 the open variant is required. Install the CUDA Toolkit separately and matched to the driver, rather than using the generic cuda-drivers meta-package, which can break the open kernel module stack. Verify with nvidia-smi for the driver and nvcc --version for the toolkit, and pick the CUDA and framework versions together from NVIDIA's support matrices.
How do I verify NVLink and GPU topology?
Run nvidia-smi topo -m to see how every GPU connects to every other and to the CPU, where NV entries are NVLink, PXB is a PCIe bridge, and SYS crosses sockets. It also shows CPU and NUMA affinity for placement. Use nvidia-smi nvlink -s to confirm no link is inactive or in error, and on HGX systems check nvidia-smi -q for a Fabric State of Completed. Then run the NCCL tests to measure real bandwidth.
What BIOS settings does a multi-GPU server need?
Enable Above 4G Decoding and Resizable BAR, both required for multi-GPU memory mapping. Force PCIe to Gen5 instead of Auto. Boot in UEFI with CSM disabled. Enable IOMMU and ACS only if you plan to virtualize or partition GPUs, and turn on a max-performance or latency-sensitive profile with reduced deep C-states for consistent throughput. These firmware settings decide whether the GPUs even appear before any driver is involved.
How much power and cooling does an 8-GPU server need?
A dense 8-GPU node draws well past 10 kW, so it needs a 208 to 240V circuit, often three-phase, with redundant PSUs sized above peak draw and a plan for inrush at power-on. Blower cards want front-to-back airflow in a closed chassis; SXM and B200 baseboards need a connected liquid loop. Because per-rack power and cooling at this density are hard to provision on site, many operators deploy into professional hosting instead.
What is the difference between a PCIe and an SXM (HGX) GPU server?
PCIe servers hold 2 to 8 GPUs in expansion slots that communicate over PCIe or NVLink bridge pairs, need no Fabric Manager, and scale around 60 to 80 percent. SXM/HGX servers mount 8 GPUs on a shared baseboard fully connected by NVLink through NVSwitch, require Fabric Manager, scale 80 to 90 percent, and at the B200 generation are liquid-cooled and SXM-only. The setup process differs mainly in the Fabric Manager step and the cooling requirements.
Can I set up a multi-GPU AI server myself, or should I buy it pre-built?
You can do it yourself; the process is a defined sequence of BIOS settings, a bottom-up driver and CUDA stack, and fabric verification. But version matching and firmware traps make it easy to lose a day, so many teams buy the server pre-built, tested, and burned in, then deploy it into hosting where power and cooling are handled. Pre-built removes the compatibility, power-delivery, and stability work; hosting removes the facility problem.
What operating system is best for a multi-GPU AI server?
Ubuntu Server LTS is the standard choice and what NVIDIA's driver and CUDA stack target first, with the widest framework and container support. Install it, confirm it boots to a shell and has networking, then layer the NVIDIA driver, CUDA Toolkit, Fabric Manager where applicable, and the Container Toolkit on top. Always confirm the specific OS, driver, CUDA, and framework combination against the vendor support matrices before committing a production build.
Sources and image credits
Technical procedure grounded in NVIDIA's CUDA Toolkit documentation, Fabric Manager user guide, data-center driver documentation, and NCCL documentation. Hero photograph of the Sierra GPU supercomputer by the U.S. Department of Energy (public domain) and multi-GPU rig photograph by BluberdeBlubBlub (CC BY 4.0), both via Wikimedia Commons, cropped and graded for MillionMiner. Diagrams: original MillionMiner graphics. Verify OS, driver, CUDA, and framework versions against current vendor support matrices before a production build.
