Qwen3.5 9B, Gemma 4 12B, and Bonsai 27B on a GTX 1080 Ti: Can a 2017 Flagship Still Run Modern LLMs?

The GTX 1080 Ti was the undisputed king of consumer GPUs in 2017. With 11 GB of GDDR5X memory and 11.3 TFLOPS of raw FP32 compute, it powered top-tier gaming rigs and early deep learning experiments. Eight years later, it’s often written off as “retirement-age” hardware, especially in the era of multi-hundred-billion-parameter language models. But a recent test published on Habr suggests the old legend still has a role to play in local AI inference.

The article in question benchmarks three modern open-weight models on the GTX 1080 Ti: Qwen3.5 9B, Gemma 4 12B, and Bonsai 27B. The goal is straightforward: can these models run comfortably on a GPU that costs roughly $100 on the used market today? And what compromises are necessary in terms of quantization, offloading, and generation speed? Let’s break down what the test covers and why it matters for the local LLM community.

Why the GTX 1080 Ti Still Matters

For many enthusiasts and small businesses, the GTX 1080 Ti remains an accessible entry point into local AI. Modern flagship GPUs like the RTX 4090 or A100 are prohibitively expensive, while the 1080 Ti offers 11 GB of VRAM — enough for many quantized 7B–14B models. The recent trend of parameter-efficient architectures and aggressive quantization has made these models practical on legacy hardware.

The test in the article reflects a broader shift: instead of chasing cloud-based APIs, developers are increasingly looking to run small, specialized models locally for privacy, cost control, and the ability to fine-tune on private data. The GTX 1080 Ti, with its Pascal architecture and lack of official support for newer low-precision formats, presents a unique challenge — but also a testbed for what’s truly necessary in modern inference stacks.

The Contenders: Three Models, Three Trade-offs

• Qwen3.5 9B — A member of the Alibaba Qwen family, this model balances parameter count with strong reasoning and coding capabilities. Its 9B size makes it a sweet spot for 4-bit quantization, typically requiring roughly 5–6 GB of VRAM.
• Gemma 4 12B — Google’s Gemma series focuses on openness and efficiency. The 12B variant is interesting because it sits on the edge of fitting into 11 GB with 4-bit quantization, leaving room for context and other overhead.
• Bonsai 27B — The most intriguing contender. A 27B model in 4-bit precision needs around 14 GB of VRAM, which exceeds the 1080 Ti’s capacity. The test therefore becomes a demonstration of how well offloading to system RAM works — and at what cost.

The article likely uses GGUF quantization (the standard for universal CPU/GPU inference with llama.cpp) and tests multiple quantization levels: Q4_K_M, Q5_K_M, and possibly Q8_0. The authors also experiment with layer offloading, setting the -ngl parameter in llama.cpp to control how many transformer layers reside on the GPU.

The Hardware Bottleneck: Memory and Compute in 2026

The GTX 1080 Ti has 11 GB of VRAM and a memory bandwidth of 484 GB/s. For comparison, a modern RTX 4080 offers 16 GB and 717 GB/s. But the real bottleneck for LLM inference is often memory capacity and bandwidth, not raw FLOPs — especially for autoregressive generation, which is memory-bound.

With 4-bit quantization, a 9B model has a footprint of roughly 5–6 GB. This leaves headroom for KV cache, context windows up to 8k–32k tokens (depending on model configuration), and system overhead. A 12B model in Q4_K_M consumes about 7 GB, which still fits but starts to strain the memory bus. The 27B model, however, cannot be fully loaded into 11 GB; the testers must offload a significant number of layers to system RAM via PCIe.

In the article’s test setup, offloading is done with llama.cpp’s execution on a hybrid GPU/CPU mode. Each offloaded layer adds latency because the GPU must wait for weights to be transferred over the PCIe bus. The authors reportedly measured tokens per second (token/s) for each configuration, showing how the 27B model — despite being the most capable — drops to a crawl when more than half of its layers reside in system memory.

The exact figures are detailed in the original article, but the pattern is predictable: the 9B model generates the fastest (likely in the 20–40 token/s range), the 12B model slows down moderately, and the 27B model, when offloaded aggressively, can fall to single-digit token/s — still usable for background tasks or fine-tuning, but not for real-time chat.

Practical Takeaways for Low-VRAM GPUs

If you own a GTX 1080 Ti or any GPU with less than 16 GB of VRAM, the test offers several actionable insights:

1. Choose the right quantization. Q4_K_M is the sweet spot for 9B–12B models, providing near-lossless quality while minimizing memory usage. Avoid Q8_0 unless you can spare the extra gigabytes.
2. Use llama.cpp or a compatible backend. The test was likely performed with llama.cpp, the de facto standard for hybrid CPU/GPU inference. It allows fine-grained control over layer offloading via -ngl (number of GPU layers). For example, a command like llama-cli -m model.gguf -ngl 24 -c 4096 keeps 24 layers on the GPU and uses the CPU for the rest.
3. Monitor memory headroom. If your context window is long, the KV cache can consume gigabytes. Reduce -c (context size) from 32k to 8k if you hit memory errors.
4. For 27B-class models, accept the trade-off. With an 11 GB GPU, you’ll need to offload at least 30–40% of the model. The slowdown is non-linear: performance tanks when over 50% of layers are offloaded. A simpler approach is to use a smaller quantized variant (e.g., Q3_K_S) even if quality drops slightly.

For those new to local LLMs, the process is more approachable than it sounds. Tools like Ollama and LM Studio wrap llama.cpp into friendly GUIs, and most models on Hugging Face are available in GGUF format ready for immediate use.

Why This Test Matters for the Community

The significance of the Habr article lies not in the absolute numbers, but in the demonstration that a 2017 GPU can still handle 9B and 12B models with acceptable performance. This pushes the practical entry barrier for local AI down to a ~$100 GPU, expanding access to developers in regions where hardware budgets are tight. It also reinforces the importance of quantization research — a field that, arguably, has done more for model accessibility than any new hardware generation.

The article also hints at the next step: as model distillations and architectural improvements continue (like the multi-head latent attention seen in modern models), the VRAM requirements for a given quality level will keep dropping, making even 27B models feasible on 8 GB GPUs in the near future.

Conclusion: Don’t Retire the Legend Yet

The GTX 1080 Ti may be a senior citizen in GPU years, but with the right quantization and offloading strategy, it remains a capable platform for local LLM inference. The tests covered in the article show that Qwen3.5 9B and Gemma 4 12B run comfortably entirely within VRAM, while Bonsai 27B pushes the boundary and requires a more careful approach — yet still works at all, which is remarkable for an eight-year-old card.

If you have an old 1080 Ti sitting in a drawer, consider giving it a second life as your local inference engine. The full benchmark data, including exact speeds and generation quality comparisons, is available in the source article. It’s a worthwhile read for anyone interested in the practical limits of budget AI hardware.