02 — ML · Systems
Syntax-aware speculative decoding that speeds up LLM code generation while enforcing syntactic correctness.
Stack
PyTorch · DeepSeek-Coder · Tree-sitter · CUDA · scikit-learn
In one line
25.03 tps, +4.7pp acceptance
SASD is a GPU-accelerated inference framework that accelerates code generation with speculative decoding while using a syntax signal to keep output syntactically valid. A small draft model proposes tokens that a large target model verifies, and a learned halt policy decides when stopping speculation actually pays off.
Demo coming soon
Speculative decoding pairs DeepSeek-Coder-1.3B (draft) with DeepSeek-Coder-33B (target), using manual model.forward() calls for fine-grained KV-cache control.
A Tree-sitter parser detects invalid draft tokens mid-generation; a bracket pre-filter and incremental edits avoid full reparses on most steps.
Diagnosed over-halting as the throughput regression's root cause, then trained a logistic-regression classifier (78.5% accuracy) to suppress unprofitable halts.
Refactored the parser from a halt trigger into an on-demand feature provider, eliminating runtime parsing overhead while preserving the syntactic signal.
Best measured throughput of 25.03 tps (+2.6%) with a +4.7pp acceptance rate and no additional training.