32B models on a single RTX 4090. 65k context.

v1.1.0 Headline: NF4 + Streaming KV on a Single RTX 4090

NF4 4-bit weight quantization (via bitsandbytes) shrinks Qwen2.5-32B's weights from ~64 GB BF16 to ~18 GB. Streaming KV keeps the compressed KV cache on CPU and streams only the slice needed for each attention computation. The combination fits a 32B model on a 24 GB consumer GPU — first time TurboAgent has done this.

How to reproduce: install turboagent-ai[torch,bnb,native]==1.1.0, then call TorchEngine(model_id="Qwen/Qwen2.5-32B-Instruct", quantize_weights="nf4", kv_storage="cpu_streaming"). The hardware detector picks NF4 automatically when it sees a single 24 GB GPU and a 30B+ model.

Streaming KV — Validated Across 3 Models, 5 Context Lengths

BF16 streaming-KV results on workstation hardware — the same engine code path that powers the single-4090 NF4 numbers above. 50/50 needle-in-haystack depths PASS across Qwen2.5-7B, Qwen2.5-32B, and Gemma 4 31B at lengths from 4k to 96k tokens.

Gemma 4 31B (BF16, 2× RTX PRO 6000 Blackwell)

Qwen2.5-32B (BF16, 2× RTX PRO 6000 Blackwell)

Qwen2.5-7B (BF16, Single RTX 4090)

Streaming KV runs zero compression intra-turn. Each turn's KV is buffered raw (BF16) on CPU during prefill and decode; quantization to TurboQuant happens once at turn boundary, so cosine similarity inside the active turn is > 0.999. Pinned-memory async prefetch overlaps PCIe transfer with attention compute. Block-allocated buffers (16-token blocks) eliminate per-step heap allocations.

NIAH tests retrieval fidelity; downstream task validation (LongBench, RULER full suite, agent-loop coherence) is ongoing — see the limitations section below.

What's Next: v1.2 — Llama-3.1-70B on a 4090

v1.1.0 ships the streaming KV + NF4 layers. v1.2 adds two more: BitTorch fine-tuning (QLoRA / LoRA / DPO) and FlexTensor automatic weight offloading. Together with the existing TurboQuant KV compression, this is the four-layer memory stack that unlocks Llama-3.1-70B with 200k+ context on a single RTX 4090 at ~3–8 tok/s decode.

How TurboAgent Compares

Architectural positioning vs. the closest open-source alternatives for running large models with long context on consumer hardware. We don't have head-to-head benchmark numbers across these stacks; the table reflects what each approach does and doesn't do.

llama.cpp is the closest functional comparison — it serves 4-bit weights on consumer GPUs via GGUF quantization. The key gap: llama.cpp doesn't compress the KV cache, so long contexts blow out VRAM regardless of weight quantization. vLLM offers FP8 KV (2x) but no CPU offload of the cache, so it needs multi-GPU for 32B+. TurboAgent's combination — NF4 weights + 5.28x compressed KV streamed from CPU — is the only Python stack we're aware of that fits 32B at 65k on a single 24 GB GPU.

Methodology, Reproducibility & Limitations

What we measured, how we measured it, and what we explicitly haven't measured yet. Honesty about benchmark scope is more useful than a "zero accuracy loss" claim that doesn't survive contact with downstream tasks.

Evaluation methodology

• RULER-style needle-in-haystack: 5 needle depths (0%, 25%, 50%, 75%, 100% of context) × N context lengths (16k, 32k, 65k, 96k).
• Needle: randomly generated alphanumeric authorization code (e.g. XK-9234) embedded in a [CRITICAL INFORMATION] bracketed phrase.
• Haystack: repeating natural-language filler text. Final prompt asks the model to repeat the code; grading is exact-string match against the model's first 64 generated tokens.
• Compute settings: temperature=0.0 (greedy), SDPA attention, num_logits_to_keep=1 for prefill, chunked prefill at 512 tokens.
• Hardware: NF4 single-4090 results from a Vast.ai RTX 4090 24 GB instance. BF16 results from 2× RTX PRO 6000 Blackwell (192 GB).

Hardware detector NF4 fallback

When you call TurboAgent(...) without specifying quantize_weights, the hardware detector picks based on model size and total VRAM:

All recommendations can be overridden by passing the keyword argument explicitly. If bitsandbytes isn't installed, NF4 falls back to BF16 with a clear log message instead of failing silently.

What we haven't measured (yet)

NIAH is the standard for retrieval but it's narrow — it tests "can the model find the needle?", not "does compression preserve multi-hop reasoning, JSON-schema adherence, or agent-loop coherence?" Items planned for v1.2 evaluation:

• LongBench / RULER full suite: multi-document QA, code completion, summarization at 16k–96k context with NF4 + Streaming KV.
• Agent-loop coherence: 10-turn TurboSwarm session with shared compressed KV; does the critic agent still attend correctly to the researcher's earlier output after 5 compress/decompress cycles?
• JSON-schema adherence under compression: structured-output benchmark before vs. after KV compress/decompress, measuring schema validity rate.
• Head-to-head throughput: Qwen2.5-32B at 65k on single 4090 across TurboAgent / llama.cpp Q4_K_M / vLLM AWQ — same prompt, same hardware.

Features

Everything you need for long-context agentic AI on consumer hardware.

Compression Modes

Validated against the paper's theoretical bounds on real hardware.

Validated at Scale

Cross-model, cross-hardware validation with production models.

Total: 65/65 NIAH depths PASS across 4 model/hardware configurations. Combined Streaming KV + NF4 path enables the "32B on 24 GB" headline result.