How We Ran Ornith-35B on a Laptop with 8GB VRAM and Hit 99 Tokens/s on AMD Strix Halo

Introduction: The Impossible Becomes Possible

What if I told you that a massive 35-billion-parameter language model could run on a laptop with just 8GB of VRAM—and generate text at 99 tokens per second? That’s not a typo. It’s the reality demonstrated by a team of developers who squeezed every ounce of performance from AMD’s Strix Halo APU.

In July 2026, a detailed case study surfaced on Habr, showcasing a remarkable feat: running Ornith-35B, a model that would traditionally require a multi-GPU server, on a portable machine. The key? Clever quantization, strategic use of on-chip memory, and the unique architecture of AMD’s latest integrated graphics. Let’s dive into how they did it, what it means for the future of on-device AI, and why this matters for developers and enthusiasts alike.

The Problem: Why 35B Models Are Usually Server-Only

Large language models (LLMs) with 35 billion parameters are notorious resource hogs. At full precision (FP16), Ornith-35B requires roughly 70GB of memory just to load the weights—far beyond the capacity of any consumer GPU. Even with 4-bit quantization, you’re looking at around 17.5GB, which still exceeds the 8GB VRAM found on many mid-range laptops.

Conventional wisdom says you need a cloud GPU instance or a workstation with multiple RTX 4090s to run such models. The authors of the original article challenged this assumption by targeting AMD’s Strix Halo—a chip codenamed Hawk Point—which integrates a powerful RDNA 3-based GPU with shared memory architecture.

The Solution: Quantization, Memory Optimization, and AMD’s Secret Sauce

The team behind the experiment leveraged three key techniques:

1. Aggressive Quantization with AWQ

They used Activation-Aware Weight Quantization (AWQ) to compress the model to 3-bit precision. This reduced memory requirements by over 60% compared to FP16, bringing the model footprint to roughly 10GB. With additional optimizations like weight sharing and dynamic range reduction, they squeezed it into 8GB.

2. Unified Memory Magic

AMD’s Strix Halo APU features a unified memory architecture where CPU and GPU share the same pool of DDR5 RAM (up to 128GB). The developers configured the system to allocate 8GB as dedicated VRAM for the GPU, while the rest served as system RAM. This allowed the model to spill over into system memory when needed, with minimal performance penalty—thanks to the high bandwidth of LPDDR5X memory.

3. Custom Inference Engine

Rather than using standard frameworks like llama.cpp or vLLM, the team wrote a custom inference engine optimized for the RDNA 3 instruction set. They exploited the GPU’s Wave32 and Wave64 compute modes to parallelize matrix operations, achieving near-ideal utilization.

The result? A sustained 99 tokens per second—fast enough for real-time chat, code generation, and even interactive document analysis.

The Hardware Setup: What You Need

Here’s the exact configuration used in the case study:

Notably, the laptop was a standard consumer model—no exotic cooling or custom power delivery. The developers noted that the system drew about 120W under load, which is well within normal laptop thermal limits.

Performance Breakdown: Tokens per Second vs. Competitors

To put the achievement in perspective, here’s how Ornith-35B performed on different hardware:

The Strix Halo’s result is impressive not because it beats the RTX 4090 (it doesn’t), but because it does so with a fraction of the memory and in a laptop form factor. The M4 Max comparison is also telling—Apple’s unified memory advantage is real, but AMD’s raw compute throughput wins out here.

Challenges and Workarounds: What Went Wrong

The original article didn’t sugarcoat the difficulties. The team encountered several roadblocks:

• Driver Issues: AMD’s ROCm stack for Windows is still immature for inference workloads. The authors switched to Linux (Ubuntu 24.04) to gain access to stable ROCm 6.2 libraries.
• Memory Fragmentation: The 3-bit quantization led to irregular tensor shapes, causing VRAM fragmentation. A custom memory allocator was required to resolve this.
• Thermal Throttling: After 15 minutes of continuous inference, the laptop’s temperature hit 95°C, and performance dropped by 15%. A simple laptop stand with active cooling (like the IETS GT500) resolved the issue.

These practical insights are gold for anyone attempting similar experiments. The authors shared their code and configuration on GitHub, which you can find linked in the original article.

Real-World Use Cases: What Can You Do with 99 Tokens/s?

At 99 tokens per second, Ornith-35B becomes usable for real-time interactions. Here are some practical applications the team tested:

• Code Assistant: Generating and explaining code snippets in real time. The model responded to prompts in under 200ms.
• Document Summarization: Summarizing a 10-page PDF in under 3 seconds.
• Interactive Chat: Running a conversational agent with near-instant responses, suitable for customer support or tutoring.

For developers building local AI tools, this opens up possibilities previously locked behind cloud subscriptions. Imagine a private coding assistant that never sends your code to a server—running on your laptop at speeds comparable to GPT-3.5.

The Big Picture: What This Means for On-Device AI

This case study is more than a technical curiosity. It signals a shift in the hardware landscape. AMD’s Strix Halo APU, with its powerful integrated GPU and unified memory, is uniquely suited for local LLM inference. Unlike NVIDIA’s discrete GPUs, which require high-bandwidth VRAM, AMD’s architecture allows large models to use system memory with minimal latency penalty.

For consumers, this means that by 2027, a $1,200 laptop could run models like Ornith-35B at usable speeds. For enterprises, it means reducing reliance on cloud GPU rentals—saving costs and improving data privacy.

However, the article also points out that software ecosystems need to catch up. While Linux + ROCm works well, Windows support remains spotty. Tools like LM Studio and Ollama are working on native AMD support, but the authors recommend sticking with Linux for now.

How to Replicate This: A Step-by-Step Guide

If you’re inspired to try this yourself, the original article provides detailed instructions. Here’s a condensed version:

1. Get the right hardware: Any laptop with AMD Strix Halo (Ryzen 9 8940HS or similar) and at least 32GB RAM. Allocate 8GB as VRAM in BIOS.
2. Install Ubuntu 24.04 LTS and ROCm 6.2. Follow the official AMD installation guide.
3. Download Ornith-35B from Hugging Face in AWQ 3-bit format. The authors used a custom quantization script.
4. Build the custom inference engine from the provided GitHub repo. It’s written in C++ with HIP for AMD GPU acceleration.
5. Run with optimal settings: Batch size 1, context length 2048, temperature 0.7. Expect 99 tokens/s.

For those who prefer a simpler path, the authors mention that llama.cpp with GGUF quantization now supports RDNA 3, but performance is about 30% lower (around 70 tokens/s).

Limitations and Future Work

No achievement is without caveats. The authors acknowledge:

• Quantization loss: 3-bit AWQ degrades output quality slightly compared to 4-bit or FP16. For creative writing, the difference is noticeable.
• Limited context length: The 2048-token context is fine for most tasks, but long document analysis suffers.
• No Windows support: The setup is Linux-only for now. Windows users must wait for better ROCm support.

Future improvements include using 4-bit quantization with a larger VRAM allocation (16GB) and experimenting with speculative decoding to boost throughput beyond 100 tokens/s.

Conclusion: A Milestone for Local AI

The ability to run a 35B-parameter model at 99 tokens per second on a laptop with 8GB VRAM is a watershed moment. It proves that the hardware and software are ready for on-device AI at scale—no cloud required. AMD’s Strix Halo, combined with clever quantization and custom inference code, has pushed the boundaries of what’s possible.

For developers, this is a call to explore local AI solutions. For consumers, it’s a glimpse of a future where powerful language models fit in your backpack. The original article on Habr is a must-read for anyone serious about edge inference.

Source

This article is based on a case study published on Habr. All facts and figures are derived from that source. The authors recommend checking the original post for up-to-date code and configurations.