vLLM 0.32.0 adds structured decisions: how to compare them with constrained JSON

A new route from documents to decisions

vLLM 0.32.0 enabled a structured-decision endpoint by default when it was released on October 18, 2026. The release lists choice, noul, and score questions over a structured state. It names support for Cloudflare Clef-Omni, LiquidAI d1-3B, and DiffusionGemma. For document intake, the practical question is whether this route can assign a queue and urgency with less work than constrained JSON. Sources: vLLM on GitHub

The distinction is how the model supplies answer values; both approaches can return JSON. vLLM’s structured-output documentation describes generating schema-constrained fields and parsing them into typed objects, including schemas derived from Pydantic models. The generic structured-decision strategy instead reads single-token answer labels and normalizes their probabilities over the permitted options. Reduced decoding work is a hypothesis to test alongside decision quality and uncertainty handling. Sources: vLLM

Three probability signals with different meanings

vLLM’s documentation distinguishes normalized answer probabilities from confidence, the winning label’s probability across the full vocabulary. It also exposes label_mass, the total probability assigned to valid answer labels. An illustrative calculation shows the difference: suppose valid labels receive 0.10 total probability and the winner takes 90% of that mass. Its normalized probability is 0.90, but its full-vocabulary confidence is 0.09. This example describes the documented relationship, not a measured response. Sources: vLLM

Neither signal automatically represents calibrated correctness; the original implementation described its probabilities as uncalibrated. System One Bench’s October 7, 2026, results give d1 an accuracy index of 82.7 and calibration error of 2.9%. Mercury Decide recorded a similar index of 82.4 but calibration error of 13.1%. The index weights dataset accuracies equally; it does not measure document-triage accuracy. These results do not establish whether vLLM 0.32.0 outperforms constrained JSON with the same model and hardware. The comparison below is a proposed experiment, not a performance verdict. Sources: vLLM on GitHub, System One Bench on GitHub, System One Bench

Keep the documents and model constant

Use three questions for each document: which queue owns it, whether it needs urgent attention, and whether it contains enough information to route. Suggested queue labels are billing, technical support, security, and unrelated. The remaining questions can use yes or no and sufficient or insufficient. Include clear cases, mixed topics, conflicting statements, irrelevant material, and incomplete documents. Annotate missing information separately from application abstention, which determines whether to trust an answer.

A proposed starting design uses 2,000 adjudicated documents: 500 for development, 500 for calibration, and 1,000 untouched test examples. These sizes are a planning suggestion, not a power calculation. Group related documents and templates before splitting, and have two annotators label difficult cases and record disagreements. Use the same supported generative checkpoint, pinning its revision, tokenizer, quantization, vLLM image digest, hardware, and server configuration. Preserve document content, question wording, label descriptions, and available information. Capture rendered prompts where possible to check what each arm actually receives.

The table below separates the proposed comparison arms by output requirements. Minimal JSON measures decision delivery without generated probability vectors. The probability arm adds uncertainty information; an optional explanation arm adds a rationale and should be reported separately. Compare specialized decision models separately, because changing the model cannot isolate the endpoint’s effect.

Measure correctness before trusting confidence

Report per-question accuracy, macro-averaged F1 score, confusion matrices, urgent-case recall, and the share of documents with every answer correct. Break results out by ambiguity and document length, counting failures separately and in overall success. Assess probability quality with multiclass Brier score, negative log-likelihood, reliability plots, and expected calibration error. Define Brier score as the mean summed squared difference between predicted probabilities and target indicators, without dividing by the number of classes. Evaluate full-vocabulary confidence separately as a possible abstention signal.

Publish raw and calibrated results, fitting a temperature transformation only on calibration data and freezing it before testing. Treat generated probability vectors separately where needed, and assess the full distribution rather than only the top confidence. Validate that JSON probabilities include every label, contain finite values between zero and one, and sum to one within a declared tolerance. Check that the chosen label matches the maximum probability. Publish first-pass outcomes before repairs, then report the frequency, accuracy impact, and latency cost of normalization or retries. Use paired accuracy tests and document-level bootstrap intervals, keeping each document’s answers together during resampling.

Time the complete request and fallback path

System One Bench describes an October 15, 2026, rerun through OpenRouter from one cloud server with eight requests in flight. It says earlier caller placement made the same calls 1.5 to 3.5 times slower. Its repository reports 188 ms for d1 in a later rerun and identifies 659 ms as the earlier measurement. Those hosted measurements describe different runs, not local endpoint overhead. Record the run date and deployment path beside each latency result. Sources: System One Bench on GitHub, System One Bench

Measure median, 95th-percentile, and 99th-percentile latency, throughput, and timeout rate under matched concurrency, document lengths, cache conditions, and client location. Test single-question and multiple-question requests separately, along with cold-schema and warm-schema JSON requests. vLLM documents prefix-cache reuse across question reads, but warns that batching and cache hits can change probabilities. Its batch-invariance setting trades throughput for repeatability, so evaluate ordinary serving and reproducibility settings separately. Include serialization, queueing, validation, retries, and escalation in workflow timing. Measure tail latency directly rather than inferring it from average first-stage and escalation timings. Sources: vLLM

Choose by safe automation and integration cost

When human review is available, compare automatic coverage at the same accepted-case error rate. Otherwise, define a conservative fallback, such as retaining a document, before selecting thresholds. Track urgent documents incorrectly deprioritized, and include the second model’s latency and cost whenever escalation is permitted. Repeat evaluation on later documents or a different source without retuning thresholds. Count implementation time, validation code, calibration tooling, monitoring, retries, and fallback handling under the same production requirements.

Verify request limits against the pinned build: the original pull request described up to 128 options, while current documentation lists 26 choice options. Test whether label ordering changes answers and whether low label_mass predicts mistakes or mainly formatting problems. Check whether calibration and abstention thresholds transfer across document sources and lengths. Before rollout, inspect confidently wrong urgent-case decisions at the intended automation coverage. Set acceptance criteria for those errors before using latency or integration savings to choose an approach. Sources: vLLM, vLLM on GitHub