Measures real speculative decoding speedup using the official HuggingFace
assistant_model API across 4 model pairs and multiple output lengths.
Validates speculative-decoding-sim (project 6) and resolves the KV cache bug from speculative-decoding-validation (project 10).
For design details see DESIGN.md.
distilgpt2 (82M) -> gpt2-medium (355M)
K=1, n=512 output tokens
Baseline: 8.19s Speculative: 4.69s Speedup: 1.747x
ratio=0.23 (distilgpt2->gpt2-medium): 1.747x at n=512
ratio=0.35 (gpt2->gpt2-medium): 1.490x at n=512
ratio=0.46 (gpt2-medium->gpt2-large): 1.170x at n=256
ratio=1.00 (gpt2->gpt2, self): 0.821x (never wins)
Validates speculative-decoding-sim cost_ratio breakeven prediction.
distilgpt2->gpt2-medium (K=1):
n=16: 0.914x
n=64: 1.125x
n=128: 1.439x
n=256: 1.590x
n=512: 1.747x <- speedup keeps growing
gpt2->gpt2-medium (K=5):
n=64: 1.101x n=256: 1.467x n=512: 1.490x
ratio=0.23: K=1 optimal (low alignment)
ratio=0.35: K=5 optimal (moderate alignment)
ratio=0.46: K=7 optimal (better alignment)
At K=1 with ratio=0.23, speculation acts like assisted decoding:
one cheap candidate per verifier step. Trying K>1 adds overhead
that outweighs the benefit at this alignment level.
python3 -m venv venv
source venv/bin/activate
pip install torch transformers
python3 profile_speculative.py # basic pairs
python3 profile_speculative_v2.py # all 4 pairs + long outputs
| Aspect | Project 10 | This project |
|---|---|---|
| API | Manual KV cache | HF assistant_model |
| KV issue | DynamicCache bug | Handled by HF |
| Best speedup | 1.14x | 1.747x |
| Output lengths | up to 64 tok | up to 512 tok |
results/sweep_k.csv K sweep for original pairs
results/output_length.csv Output length for original pairs
results/all_results_v2.csv All v2 results (37 rows)
Project 20 in a series on LLM inference infrastructure. Full series: https://github.com/JohnScheuer