Rules · protocol 28
Scoring
Every submission is measured beside BF16 of the same model, in the same pod on one B200, and earns one number: the LowBScore. BF16 scores 1, higher is better, and each task's board ranks by it.
The score
Three axes, each against BF16, in a geometric mean; the gates set g.
LowBScore = g · T⅓ · P⅓ · A⅓
The axes are weighed equally: doubling one multiplies the score by 2⅓ ≈ 1.26, and halving one divides it by as much, so a loss on one axis costs what the same factor gained on another earns. BF16 has every axis at 1 and scores 1. A missed gate keeps a tenth of the score: the solution stays below BF16 whatever it gains, and still ranks among the solutions that miss one.
The three axes
Each is 1 for BF16; above 1 is better.
TPS
T = (sserve · sinteractive · slong)⅓
÷ max(1, 1 + ΔL)
- s: throughput over BF16's in the same pod, on each workload: output tokens per second for the batch workloads, time per output token for one request at a time.
- ΔL: the change in generated length against BF16 on the same items and seeds. Longer answers cost time, so they divide T; shorter ones earn nothing.
Profit
P = (cserve / c′serve · clong / c′long)½
- c: how many of the workload's requests one GPU holds at once: the memory BF16's run reserved, less the weights and other memory, over the KV cache and state one request holds; c′ is BF16's.
- The KV cache is charged for every request in flight: an engine that runs fewer at once and queues the rest earns no capacity by it.
- On serve and long context: one request at a time is bound by latency, not memory.
Accuracy kept
D = 0.79 × drop + 0.21 × 42.8 × KL A = exp(−max(D, 0) / 10)
- drop: the mean paired drop from BF16 over the gated domains, in points. KL: KL(BF16 ‖ solution) in nats per token on hidden prompts, at 42.8 points per nat. The weights were fitted on the references.
- No drift keeps A at 1, and A never exceeds 1: one point of drift keeps 0.905, five points keep 0.607.
The gates
Checked on every evaluation, each against BF16 in the same pod.
| Check | Passes when | If not |
|---|---|---|
| Accuracy | Every gated domain's one-sided 95% upper bound on the paired drop from BF16 is at most 5 points (tier B; tier A is 3). A scored evaluation pools two runs with independent seeds. | g = 0.1 |
| Speed | The throughput gains' geometric mean, less 1.645 × 0.05 / √3, is at least 0.95: shown to serve no more than 5% slower than BF16. | g = 0.1 |
| Length | Generated tokens within ±10% of BF16's on the same items and seeds. | g = 0.1 |
| Contract | A quantization of the task's model: its weights in any format, other tensors at most 1% of its bytes, and KL from BF16 at most 0.15 nats per token. | score 0 |
| Engine rules | The checkpoint is sealed before any engine code runs; the engine runs
unprivileged with loopback networking only, keeps no weights or KV blocks in host memory, answers
prompt_logprobs for the KL probe, and holds its requests under a hard cap of its timed peaks
(all the rules). | score 0 |
Qwen3-8B: tier B, speed floor 0.95×; Qwen3.5-27B: tier B, speed floor 0.95×. The first gate missed is the reason a board shows.
A worked example
The best participant on Qwen3.5-27B, QAD NVFP4 + rot NVFP4 KV + fused int8 GDN state + GEMV, with its own numbers.
| Axis | Measured against BF16 | Value |
|---|---|---|
| T · TPS | serve 2.04× · interactive 1.83× · long context 2.13×; geometric mean 1.998; generated length -1.4% | 1.998 |
| P · Profit | serve 4.83× · long context 4.70×; geometric mean | 4.762 |
| A · Accuracy kept | drop 0.56 points, KL 0.0267; D = 0.68 | 0.934 |
| g | every gate passed | 1 |
| LowBScore | 1.998⅓ × 4.762⅓ × 0.934⅓ | 2.071 |
What is measured
Speed, memory, and accuracy, each beside BF16 on the same GPU.
Three workloads
| Workload | Requests | Prompt | Generated | At once | Used in |
|---|---|---|---|---|---|
| serve | 64 | 8,192 | 512 | 32 | T, P |
| interactive | 32 | 1,024 | 256 | 1 | T |
| long context | 128 | 16,000 | 256 | 64 | T, P |
Each runs beside BF16 in the same pod with the same requests. Peak memory counts the weights, reservations, and the KV cache and state the requests held, polled every 0.5 s.
Ten benchmarks
| Domain | Benchmarks | Gated |
|---|---|---|
| code | HumanEval, MBPP, LiveCodeBench v6 | yes |
| math | GSM8K, MATH-500 | yes |
| knowledge | MMLU-Pro | yes |
| commonsense | ARC-Challenge, HellaSwag | yes |
| expert | GPQA Diamond | reported only |
| long context | RULER 16K | yes |
Development evaluations use public items (20 per task, never ranked); scored ones use hidden items (3 per task; the last one ranks). The KL probe reads 256 hidden prompts (64 public for development). GPQA is reported, not gated: its 89 hidden items are too few to bound a 5-point drop.
Ranking
One board per task, participants and references ranked apart.
- Participants rank by the LowBScore of their last scored submission. Solutions that pass every gate rank above those that miss one; equal scores share a rank.
- References, standard recipes and published checkpoints served by the stock vLLM, are ranked apart under the same gates.
- GPU-hours are shown, not charged: the time your own server served after it started, over all your evaluations of the task, development and scored. Start-ups, BF16's runs, and evaluations that failed because of the arena are not counted.
- Shown, not scored: perplexity against BF16, and W·A·KV as declared or read by the service.
- Each submission's page shows how its score was computed, with its own numbers.
Every rule and constant on this page is quoted from the code that scores, and is machine-readable at /api/protocol. A rule change gets a new protocol version and re-scores every stored submission from its stored measurements.