For months the vision model ran under Ollama. In September I moved it to plain llama-server from llama.cpp. Here is why, with the numbers.

The card sets the rules

The P40 is cheap VRAM with old silicon:

• compute capability 6.1, no tensor cores, no bf16;
• fp16 runs at 1/64 of fp32 (I learned that the expensive way);
• my Debian 13 host has NVIDIA driver 550. The R580 branch is the last one that supports Pascal, and CUDA 13 dropped it.

That last point matters more than it looks: it freezes which versions of everything you can run.

Where Ollama got stuck

Ollama is a convenient layer on top of llama.cpp. On this card the layer started to cost more than it gave:

1. Version freeze. Ollama 0.13+ needs driver 570 for Pascal. I was stuck on 0.12.x, so newer models (and newer vision support) were out of reach.
2. A regression on P40s. Versions 0.30 to 0.32 loaded everything on the CPU with P40s (ollama#17290). Upgrading was not an option either.
3. Forced context. With qwen3-vl, Ollama 0.12 set an 8192-token context whatever num_ctx said. That is about 12 GB of VRAM for an 8B model, on a card that also has to host a virtual try-on model.

I didn't use Ollama's model catalogue or its model swapping: one model, always loaded. So I was paying for features I didn't need.

What else runs on Pascal? Not much

Before switching I checked the alternatives:

Everything that works on Pascal is llama.cpp underneath. So I went to the source.

The setup

One container, one model, loaded at start and never unloaded:

• Qwen3-VL-8B-Instruct, GGUF Q4_K_M, with the mmproj in Q8_0 for vision;
• llama-server -c 4096: the context I actually need, set for real;
• the OpenAI-compatible API (/v1/chat/completions), so the app code became a standard client.

Porting the app meant changing two call sites from Ollama's /api/generate to the OpenAI format, behind an environment variable so I could switch back instantly. The old Ollama container is still there, stopped, as the rollback path. Never run both at once: they fight for the same VRAM.

The numbers

Benchmark on 26 real clothing photos, same prompts:

Same accuracy, more than twice as fast, and 1.2 GB back for the try-on model.

Four traps I hit

1. Use the -Instruct model. The qwen3-vl:8b tag in Ollama is the "thinking" variant, which returned empty answers for my JSON prompts.
2. llama-server doesn't read WebP. My app stores WebP; I convert to JPEG (with sharp) before sending the image.
3. Phone photos blow the context. A full-resolution phone photo turns into more image tokens than a 4096 context can hold. Resizing to 1024 px on the long side fixed it.
4. Count VRAM at peak, not at rest. The card also hosts a try-on model and a background remover. Weights fit easily; activations during a run are what pushed it to 22.5/23 GB once. Budget for the worst moment.

Takeaway

If you have an old Pascal card and only serve one model, Ollama's convenience may be costing you context, VRAM and access to new models. llama-server is a single binary with an OpenAI API, and on this card it was faster at the same accuracy.

The app this runs is Fring, free, with no ads. Questions about the P40 setup are welcome in the comments.

This article was drafted with AI assistance.

It wasn't. The model was loaded in fp16, because "use half precision to save VRAM" is the default advice everywhere. On the P40 (GP102, compute capability 6.1), fp16 runs at roughly 1/64 of fp32 throughput. Same inputs, same seed:

The images were practically identical (mean difference 0.77/255). The card never needed the VRAM savings.

Two more traps from the same month

• A CPU-only torch wheel in the Dockerfile, inherited from an older 4 GB card. nvidia-smi saw the GPU while torch.cuda.is_available() returned False. Check inside the container, not on the host.
• An unpinned mediapipe upgrade removed mediapipe.solutions. The health check only tested the import, so it kept reporting pose estimation as healthy while the service silently fell back to a centred paste.

What I changed

The service now picks fp16 only when the GPU actually accelerates it (Volta and newer, or P100), or when fp32 would not fit in VRAM. An environment variable can force either.

If you're buying a P40 today: the R580 driver branch is the last one to support Pascal, CUDA 13 dropped it, and PyTorch cu128/cu129 wheels no longer ship sm_61 kernels. Treat it as a frozen target. For LLMs, llama.cpp's llama-server built against CUDA 12 runs well on it: Qwen3-VL-8B at Q4_K_M answers in 1.3 s using 6.4 GB.

The general lesson for old datacenter cards: measure before applying the usual advice.

Written with AI assistance, reviewed and published by the author.