FreeToken: Making Large MoE Models Practical on a Single Machine

2026-08-25 14 min read

Paper: FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution
arXiv: 2608.16157
Link: https://arxiv.org/abs/2608.16157

Large Mixture-of-Experts models are interesting for local inference because they are huge in total parameter count while activating only a small fraction of those parameters for each token. On paper, that sounds ideal: a 200B+ parameter MoE model may use only around 10–20B parameters per token, so the actual compute cost can be much smaller than the total model size suggests. But there is an obvious problem:

Even if only a few experts are active at a time, all of the expert weights still need to live somewhere.

For consumer hardware, that usually means keeping most experts in CPU memory while only part of the model fits in GPU VRAM. FreeToken is about making this setup much more efficient, and its main idea is surprisingly simple:

Instead of always moving a missing expert to the GPU, or always computing it on the CPU, FreeToken uses both paths at the same time and decides the split based on the machine’s actual memory bandwidth.

That idea is combined with different strategies for prefill, decode, expert caching, agent workloads, and GPU memory management.

The Core Problem

A typical local MoE setup looks roughly like this:

                GPU
        ┌─────────────────┐
        │ Attention       │
        │ Shared layers   │
        │ Expert cache    │
        └────────┬────────┘
                 │ PCIe
                 │
        ┌────────▼────────┐
        │ CPU DRAM        │
        │ Full expert pool│
        └─────────────────┘

The non-expert parts of the model can stay on the GPU, while the full set of experts remains in host memory and a subset is cached in VRAM. At that point, serving performance depends heavily on how well the runtime manages three resources:

GPU compute
PCIe bandwidth
CPU memory bandwidth

The important part is that prefill and decode behave very differently, and that distinction drives much of FreeToken’s design.

1. Prefill and Decode Are Different Problems

For a single decode token, the router may only select a few experts. For example:

token t
   │
   ▼
 router
   │
   ├── Expert 3
   ├── Expert 7
   ├── Expert 12
   └── Expert 18

This is the normal advantage of MoE: only a small subset of the model is active. But prefill is different. Suppose a prompt contains thousands of tokens. Each token may select different experts:

token 1  → E1, E3
token 2  → E4, E8
token 3  → E2, E9
token 4  → E5, E11
...
token N  → E6, E10

Once we take the union over thousands of tokens, the model may end up touching almost every expert in the layer. So during prefill:

MoE sparsity becomes much less useful.

This leads FreeToken to use two completely different execution strategies.

Phase	Main problem	FreeToken’s strategy
Prefill	Almost all experts are touched	Stream full expert layers
Decode	Only a few experts are active	Cache experts and handle misses dynamically

This separation is one of the cleanest ideas in the paper.

2. Prefill: Stream the Entire Layer

During prefill, trying to predict which experts will be needed is not very useful. Instead, FreeToken assumes that most experts will eventually be used and simply streams an entire MoE layer from CPU memory to the GPU. The key optimization is double buffering. Imagine the GPU is computing layer L.

At the same time, FreeToken transfers the experts for layer L+1.

time ─────────────────────────────────────>

GPU:
[ compute layer L ]
                  [ compute layer L+1 ]
                                      [ compute layer L+2 ]

PCIe:
[ load layer L+1 ]
                  [ load layer L+2 ]
                                      [ load layer L+3 ]

Two buffers alternate:

Buffer A → current layer

Buffer B → next layer being transferred

After layer L finishes, the buffers swap roles. In the ideal case, the execution time becomes roughly:

\[T \approx \max(T_{\text{PCIe}}, T_{\text{compute}})\]

instead of:

\[T = T_{\text{PCIe}} + T_{\text{compute}}\]

The difference is important because, instead of waiting for transfer and then computing, the runtime tries to hide one behind the other.

Why This Matters

The paper reports an experiment where an expert pool of roughly 64 GB is streamed over a PCIe connection delivering around 52.7 GB/s. The measured prefill time is very close to the time required to simply stream those weights once, which suggests that most of the GPU computation is effectively hidden behind the transfer. That is exactly what a good overlap-based runtime should achieve.

3. Decode: Exploit Expert Locality

Decode has a very different access pattern. A single token activates only a handful of experts, so streaming every expert would obviously be wasteful; FreeToken instead keeps an LRU expert cache inside GPU memory. The intuition is straightforward:

Consecutive tokens often choose similar experts.

For example:

token t-1:
E3 E7 E9 E12 E17 E24

token t:
E3 E7 E9 E12 E17 E24

The selection will not always be identical, but there is enough temporal locality that recently used experts are often useful again, which makes keeping these hot experts in VRAM worthwhile.

Cache Hit

If the router selects an expert that already exists in the GPU cache:

Router
  │
  ▼
Expert cache
  │
  └── HIT → run on GPU

This is the easy case. No transfer is needed.

Cache Miss

The interesting part happens when an expert is not in GPU memory. A normal runtime has two obvious options.

Option A: Move the expert to the GPU

CPU DRAM
   │
   │ PCIe
   ▼
GPU VRAM
   │
   ▼
GPU compute

Option B: Keep it on the CPU

CPU DRAM
   │
   ▼
CPU compute

Many systems strongly favor one of these paths. FreeToken asks a different question:

Why not do both?

4. The Most Important Idea: Bandwidth-Adaptive Execution

Suppose there are m missing experts for the current layer. FreeToken sends some of them to the GPU and computes the rest directly on the CPU.

             Cache misses
                  │
         ┌────────┴─────────┐
         │                  │
         ▼                  ▼
   PCIe → GPU          CPU execution

The two paths run in parallel. The only remaining question is:

How many experts should go to each side?

This is where the paper introduces its most important equation.

5. Deriving ($q^*$)

Let:

$m$: number of cache misses
$q$: number of missed experts transferred to the GPU
$S$: size of one expert
$B_P$: measured PCIe transfer bandwidth
$B_H$: measured host-side expert processing bandwidth

If q experts are transferred to the GPU, their transfer time is approximately:

\[T_{\text{fill}} \approx \frac{qS}{B_P}\]

The remaining:

\[m-q\]

experts stay on the CPU. The important observation is that PCIe DMA is also reading from host DRAM. So CPU execution and PCIe transfer compete for the same memory subsystem. FreeToken approximates the bandwidth left for CPU execution as:

\[B_H-B_P\]

Therefore:

\[T_{\text{cpu}} \approx \frac{(m-q)S}{B_H-B_P}\]

Because both paths execute simultaneously, the best split is approximately the point where both finish at the same time:

\[T_{\text{fill}} \approx T_{\text{cpu}}\]

So:

\[\frac{qS}{B_P} = \frac{(m-q)S}{B_H-B_P}\]

After simplifying:

\[\boxed{ q^* \approx m\frac{B_P}{B_H} }\]

This is probably the single equation worth remembering from the paper.

6. A Simple Example

Suppose:

\[B_H = 4\]

and:

\[B_P = 1\]

with four cache misses:

\[m=4\]

Then:

\[q^* = 4\times\frac14 = 1\]

So FreeToken chooses roughly:

1 expert  → PCIe → GPU
3 experts → CPU execution

Instead of forcing all four experts onto one side, it uses both available execution paths.

7. Why the Policy Depends on the Machine

One subtle but important point is that FreeToken does not use theoretical bandwidth from hardware specifications. It measures the actual system. The paper reports machines with very different ratios between PCIe bandwidth and CPU-side expert bandwidth. For example:

System	PCIe bandwidth (B_P)	Host expert bandwidth (B_H)
RTX 5090 server	52.7 GB/s	77.3 GB/s
RTX 4090	25.1 GB/s	63.2 GB/s
RTX 5090 desktop	49.0 GB/s	53.8 GB/s
RTX 4060 laptop	11.8 GB/s	47.5 GB/s
RTX PRO 6000	51.5 GB/s	178 GB/s

These systems should not use the same policy. Take the RTX 4060 laptop:

\[\frac{B_P}{B_H} \approx 0.25\]

If there are four cache misses:

\[q^*\approx1\]

So only around one expert should be transferred to the GPU. The remaining experts are better handled on the CPU. Now look at the RTX 5090 desktop:

\[\frac{49}{53.8}\approx0.91\]

In that case, transferring almost every missed expert to the GPU makes much more sense. This is the reason for the name:

Bandwidth-Adaptive Execution

The runtime adapts the execution policy to the actual machine.

8. Better Expert Caching with LRU

Bandwidth-adaptive execution handles cache misses. But FreeToken first tries to reduce the number of misses in the first place. For this, it uses a global LRU expert cache. The paper reports significantly lower miss rates compared to KTransformers and llama.cpp.

For one Qwen model, the reported miss rates are roughly:

FreeToken       16%
KTransformers   41%
llama.cpp       62%

For DeepSeek-V4-Flash:

FreeToken       39%
KTransformers   59%
llama.cpp       89%

The exact numbers depend on the hardware and cache configuration, but the overall trend is clear. Dynamic locality-aware caching works better than more static placement strategies for this workload.

9. Why Static Expert Placement Can Be Limiting

A simple MoE runtime may classify experts into two groups:

GPU experts
CPU experts

Experts assigned to the GPU run on the GPU. Experts assigned to the CPU stay on the CPU. That is easy to implement, but it assumes expert popularity is relatively stable. FreeToken instead makes the system much more dynamic.

router
   │
   ▼
GPU LRU cache
   │
   ├── hit → GPU
   │
   └── miss
          │
          ▼
      q* policy
       /     \
      /       \
 PCIe → GPU   CPU

The execution path can change from token to token. That makes FreeToken less like a simple CPU offloading system and more like a heterogeneous scheduler.

10. Dynamic Scheduling Without Giving Up CUDA Graphs

There is another implementation challenge. Expert routing changes every token. One token may produce:

misses = {3, 8}

The next:

misses = {2, 9, 10}

And the next:

misses = {7}

A naive runtime could return to the CPU after each routing step:

GPU
 ↓
CPU scheduling
 ↓
GPU
 ↓
CPU scheduling
 ↓
GPU

But frequent synchronization and host-side scheduling hurt decode latency. FreeToken instead performs many routing-dependent decisions directly on the GPU. The runtime handles operations such as:

expert deduplication
hit/miss classification
determining (q)
choosing eviction victims
rewriting logical expert IDs to physical cache slots

without repeatedly bouncing control back to Python. The CPU-side expert work is managed by persistent native worker threads, and the system is designed to preserve CUDA Graph execution as much as possible. That implementation detail is important because the scheduling policy would not be nearly as useful if the scheduler itself introduced large per-token overhead.

11. Dynamic GPU Memory Management

Another practical problem is VRAM allocation. GPU memory is not only used by expert weights. It also needs space for things like:

non-expert model weights
KV cache
temporary buffers
expert cache
other applications using the GPU

And KV cache grows as the conversation gets longer. A static allocation might start like:

Expert cache : 12 GB
KV cache     : 4 GB

But later the same session may need:

Expert cache : 7 GB
KV cache     : 9 GB

FreeToken can resize the expert cache and give more VRAM to the KV cache when necessary. The full expert pool in CPU memory acts as the source of truth, so evicting or rebuilding the GPU expert cache does not affect correctness. This is a small detail compared to the main scheduling idea, but it matters a lot in a real local inference runtime.

12. Agent Workloads Introduce Another Problem

The paper spends quite a bit of attention on agent workloads. That makes sense because modern agents repeatedly alternate between generation and tools. A simplified agent loop looks like:

User request
    │
    ▼
LLM reasoning
    │
    ▼
Tool call
    │
    ▼
Tool result
    │
    ▼
More reasoning
    │
    ▼
Another tool call

The context keeps growing. At the same time, agent frameworks may rewrite or remove parts of older context. For example:

system
reasoning
tool call
tool output
reasoning
tool call
...

Some intermediate reasoning blocks or tool outputs may be removed later. That creates a problem for prefix reuse.

13. Semantic-Aware State Caching

FreeToken does not place reusable checkpoints at arbitrary token positions. Instead, it creates them at semantic boundaries such as:

</think>

</tool_call>

</tool_output>

The reasoning is simple. Agent frameworks usually modify context at meaningful block boundaries. So if the runtime stores state at the same kind of boundaries, there is a better chance that an old checkpoint remains reusable. For example:

system
reasoning
tool call
tool output
--------------------- checkpoint
new suffix

If the prefix above the checkpoint is still unchanged, FreeToken only needs to recompute the new suffix. Without this mechanism:

re-prefill entire context

With semantic state caching:

reuse old state
+
prefill changed suffix

This becomes especially useful for long-running agent sessions.

14. FreeToken vs. Traditional MoE Offloading

A simplified comparison looks like this.

Static-style offloading

Expert placement decided beforehand

GPU experts → GPU

CPU experts → CPU

The placement is relatively fixed.

FreeToken

                     router
                       │
                       ▼
                 GPU LRU cache
                   /       \
                hit         miss
                 │            │
                 ▼            ▼
                GPU        q* policy
                           /        \
                          /          \
                  PCIe → GPU        CPU
                          \          /
                           \        /
                              merge

The important difference is that placement and execution are continuously adapted. An expert that ran on the CPU for one token may later be cached in the GPU. A cache miss may be transferred to the GPU on one machine but executed directly on the CPU on another machine. The hardware determines the policy.

15. Performance Results

The paper reports fairly strong decode performance on a single RTX 5090. For Qwen3.6-35B-A3B, FreeToken reaches roughly:

77–83 tokens/s

with around a:

1.8×–2.3×

speedup over the strongest baseline depending on the workload. For DeepSeek-V4-Flash 284B, it reaches roughly:

22–25 tokens/s

with around:

1.5×–1.9×

speedup over the strongest baseline. These results are interesting because the model is much larger than GPU memory. The system is not trying to make the full model fit in VRAM. It is trying to make not fitting efficient.

16. Tail TTFT Is Especially Important for Agents

One metric I liked in this paper is the emphasis on tail Time to First Token. For a normal chatbot, a slow request is annoying. For an agent, it can be worse. External tool clients may have timeouts.

A very long prefill can therefore turn into an actual system failure rather than just bad latency. The paper reports worst-case TTFT below roughly:

44 seconds

for FreeToken across its evaluated workloads. Some baselines reached much larger values, including cases in the hundreds of seconds. This leads to a useful framing:

For agent systems, tail TTFT can become an availability problem.

I think this is more meaningful than reporting only average tokens per second.

17. Running Very Large Models on Consumer Hardware

One of the more eye-catching results is the RTX 4060 Laptop experiment. The GPU has only 8 GB of VRAM, but FreeToken runs a 35B-class MoE model at around:

39.3 tokens/s

The system obviously relies heavily on host memory, but it shows how far a good heterogeneous runtime can stretch relatively limited GPU hardware. At the other extreme, the paper runs a 753B parameter model on a single RTX PRO 6000. The configuration includes:

GPU VRAM : 96 GB
Host DRAM: 512 GiB
Checkpoint: ~433 GB

FreeToken reports around:

14.9 tokens/s

while llama.cpp reports roughly:

7.3 tokens/s

So “single GPU” here does not mean the entire model fits in GPU memory. It means one GPU is being combined with a very large host-memory pool. That distinction matters.

18. What I Think Are the Strongest Parts

There are a few things I particularly like about the paper.

1. Prefill and Decode Are Treated Separately

This sounds obvious after reading the paper, but it is an important design decision. The access pattern is fundamentally different.

Prefill
→ many tokens
→ almost every expert gets touched
→ streaming makes sense

Decode
→ one or a few tokens
→ sparse expert activation
→ caching makes sense

Trying to force the same optimization strategy onto both phases would leave performance on the table.

2. The $q^*$ Policy Is Very Simple

The policy boils down to:

\[q^*=m\frac{B_P}{B_H}\]

There is no large predictive model or complicated scheduler. It takes two measured bandwidths and uses them to balance two execution paths. Simple models are attractive in runtime systems because they are easier to understand, profile, and adapt.

3. It Uses Real Hardware Measurements

The paper does not assume that “PCIe 5.0 means X GB/s” or that DRAM bandwidth equals a number from a spec sheet. It profiles the actual expert workload. That matters because effective bandwidth depends on much more than the interface’s theoretical maximum.

4. The Paper Goes Beyond One Isolated Optimization

FreeToken includes:

prefill streaming
expert caching
CPU/GPU scheduling
CUDA Graph integration
persistent CPU workers
agent state caching
elastic VRAM management

So it feels like a runtime paper rather than a single micro-optimization.

19. Limitations and Things to Keep in Mind

There are also a few places where I would be careful not to over-generalize the results.

The $q^*$ Model Is Still an Approximation

The model is largely bandwidth based:

\[T \approx \frac{\text{bytes}}{\text{bandwidth}}\]

That makes sense when expert execution is strongly memory bound. But CPU execution can also depend on:

SIMD efficiency
quantization and dequantization
NUMA behavior
cache effects
thread scheduling
expert dimensions
batch size

FreeToken partly handles this by measuring effective expert bandwidth rather than using theoretical DRAM bandwidth, which is a good choice. Still, the policy may need to be re-profiled when the workload changes significantly.

This Is Mainly a Small-Batch Local Serving System

The target workload is mostly:

single user
interactive inference
small batches
personal machine
agent workload

That is very different from a datacenter inference server running continuous batching across many users. With large batches, expert reuse and scheduling behavior can change significantly. So I would not assume the exact policy transfers directly to something like a large vLLM deployment.

“753B on One GPU” Needs Context

The paper does run a 753B model with one GPU. But that machine also has:

512 GiB of host memory

The full checkpoint is hundreds of gigabytes. So the result demonstrates efficient GPU + CPU heterogeneous execution, not magical compression of a 753B model into 96 GB of VRAM.

20. The Deeper Idea: Do We Need to Move the Data?

I think this is the most interesting systems insight in the paper. A common optimization question is:

How can I move the weights to the GPU faster?

FreeToken asks something slightly different:

Do I even need to move them?

If an expert already lives in CPU memory, there are two possibilities:

CPU memory
   │
   ├── move weights → GPU → compute
   │
   └── compute directly where the weights already are

Moving data is not free. Sometimes the fastest way to execute something on a GPU system is simply not to move it to the GPU. This way of thinking applies far beyond MoE models. It is a general heterogeneous-systems principle:

Compute placement should depend on both compute capability and data movement cost.

21. A Better Way to Think About the Paper

I would not describe FreeToken simply as an “expert offloading system.” A better description is:

FreeToken treats local MoE inference as a heterogeneous resource scheduling problem.

The runtime has to manage:

GPU compute
GPU VRAM
GPU memory bandwidth
PCIe bandwidth
CPU compute
CPU DRAM bandwidth
KV cache
expert locality
agent state

all at the same time. The goal is not:

How do we fit the model on the GPU?

The model often cannot fit. The better question is:

Where should every piece of computation happen, and when should its data move?

That is a much more useful systems perspective.