FreeToken From Berkley Makes Your PC an AI Data Centre

University of California, Berkeley researchers have open-sourced FreeToken, a local inference engine that runs the Qwen3.6-35B language model on an 8GB GPU, despite a 35B model typically requiring 70GB for its weights. The engine achieves 39.3 tokens per second with Qwen3.6-35B, running two to four times faster than the ollama framework. The team explains that dynamically splitting memory misses overcomes compute limitations and dramatically accelerates performance, particularly for coding agents where first-token latency dropped from 232 seconds to under 44 seconds.

FreeToken Enables Qwen3.6-35B on Limited 8GB GPU Memory

Qwen3.6-35B is now operational on an 8GB GPU, a feat previously requiring 70GB of memory for its weights. University of California, Berkeley researchers achieved this reduction in hardware demands with FreeToken, a newly open-sourced local inference engine designed to exploit Mixture-of-Experts architectures. Initial testing demonstrates FreeToken processing Qwen3.6-35B at 39.3 tokens per second. The engine bypasses traditional compute limitations by focusing on the MoE router and dynamically splitting memory misses between the CPU and PCIe, based on a machine’s bandwidth.

This architecture significantly benefits coding agents, which often rewrite history and process thousands of tokens during prefill; FreeToken reduces first-token latency for these agents from 232 seconds, using llama.cpp, to under 44 seconds by saving checkpoints at agent framework boundaries. Beyond Qwen3.6-35B, FreeToken also supports DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens per second and GLM-5.2 753B on a 96GB GPU at 14.9 tokens per second, all while remaining compatible with OpenAI and Anthropic APIs and licensed under Apache 2.0.