DGX Spark in My Personal AI Lab

The NVIDIA DGX Spark is excellent for running models that exceed consumer-GPU memory, but its value is capacity and flexibility rather than maximum inference speed.

That distinction has shaped how I use it in my personal lab.

Why I Chose It

The main attraction was its 128 GB of unified memory. Given what memory costs these days, I occasionally reassure myself that I bought an AI system and not simply a very expensive box of RAM.

I wanted a machine capable of running larger models locally without maintaining a multi-GPU workstation or relying entirely on cloud infrastructure. The integrated ConnectX-7 networking also made it an interesting foundation for future multi-node experiments.

My main intended workloads were:

• large-model inference;
• embeddings;
• fine-tuning experiments;
• exploring the NVIDIA AI ecosystem;
• studying modern AI architecture from an HPC perspective.

One of my first tests was running gpt-oss-120b locally. That immediately demonstrated the value of the device: it can load models that would not be practical on a typical consumer GPU.

That does not mean every model runs quickly. It means the experiment is possible on one compact system.

Capacity Is Not Throughput

The main limitation has been memory bandwidth.

LLM inference is often constrained by how quickly model weights can be moved through memory, particularly during token generation. The Spark can hold large models and large contexts, but it does not have the high-bandwidth memory subsystem of a datacentre accelerator.

I notice this with models such as Qwen 27B and 35B mixture-of-experts variants. They fit comfortably enough to be useful, but would generate tokens considerably faster on a GPU with higher memory bandwidth.

This is the most important distinction:

Fitting a model and serving it quickly are different problems.

The unified memory makes the Spark unusually flexible, but it does not turn it into a miniature datacentre GPU.

For a personal lab, I consider that a reasonable trade-off. I can experiment with more architectures, quantisations and context sizes than I could on a conventional consumer card. Maximum token throughput is not always the objective.

Integration into My Infrastructure

I already operate Nomad and Consul in production, so I integrated the Spark into a separate Nomad lab region rather than treating it as an isolated workstation.

The production and lab environments are connected through Consul WAN federation over a WireGuard tunnel. This keeps experimental workloads separate while still allowing selected services to be discovered and reached from the rest of my infrastructure.

Model serving on the Spark is handled by llama.cpp or vLLM, depending on the model, runtime requirements and purpose of the experiment.

I use llama.cpp where quantised models, predictable memory usage and lightweight OpenAI-compatible serving are the priority. vLLM is useful where batching, higher concurrency or its broader serving capabilities are a better fit.

Applications do not depend directly on either runtime. Requests are routed through Bifrost, my existing AI gateway, which provides a stable interface across local and hosted model providers.

The architecture is broadly:

application
    → bifrost ai gateway
        ├── hosted model provider
        └── llama.cpp or vllm on the dgx spark

Bifrost remains responsible for application-facing concerns such as:

• authentication;
• model routing;
• provider selection;
• observability;
• caching;
• fallback behaviour.

The Spark is responsible for model execution and experimentation.

This separation lets me change models, quantisations and serving backends without requiring applications to know where a model is running or whether it is served through llama.cpp, vLLM or an external provider.

Orchestration Limitations

Nomad integration on the Spark is not completely seamless yet.

Its ARM64 platform and the current state of NVIDIA device-plugin support do not provide the same straightforward GPU scheduling experience as a conventional discrete GPU system. I opened an issue after encountering missing ARM64 release artefacts for the Nomad NVIDIA device plugin.¹

For now, I use a pragmatic combination of Nomad-managed services and direct access to the NVIDIA environment where required. This works well for a controlled single-node lab, although it provides less complete GPU resource accounting than I would want in a shared production cluster.

It is a limitation, but not one that prevents the Spark from being useful.

Power and Sustained Use

The device has been more resilient than I expected from some of the early criticism around it.

Under sustained inference, I have seen light throttling at most, without a noticeable effect during normal use. I generally limit the GPU clock slightly:

sudo nvidia-smi -lgc 0,2000

In day-to-day use, this appears to reduce power consumption by roughly 20 to 30 watts while causing only a small performance reduction.

I have not benchmarked this rigorously, so I treat those figures as an operational observation rather than a precise measurement. The lower clock also reduces sustained heat. Since I am often away from the lab while experiments are running, I would rather sacrifice a small amount of performance than worry about thermal throttling or an unexpected reboot.

ConnectX-7 and Future Expansion

The integrated ConnectX-7 interface was another reason I chose the Spark.

I would eventually like to add a second unit and explore distributed inference and other multi-node workloads. The networking hardware makes that an interesting possibility, although the architecture of the device itself introduces practical PCIe and system-level constraints.

A high-speed interface does not guarantee that every workload can sustain its theoretical throughput. PCIe topology, memory movement, communication overhead and software design all influence the result.

I therefore see ConnectX-7 as a valuable experimental capability rather than a guarantee of datacentre-class scaling.

Who It Is For

I would buy the DGX Spark again.

It is particularly well suited to people who want to:

• run their own models locally;
• experiment beyond consumer-GPU memory limits;
• explore inference, embeddings and fine-tuning;
• compare serving runtimes such as llama.cpp and vLLM;
• learn more about the NVIDIA AI ecosystem;
• build a compact but capable personal AI lab.

It is less compelling for someone whose only objective is maximum token throughput for models that already fit on a high-end consumer GPU.

The Spark’s strongest feature is not that it beats every alternative on speed. It is that it makes a much wider range of experiments practical on one compact and resilient system.

Footnotes

1. Missing ARM64 artefacts in nomad-device-nvidia 1.1.0 ↩