Selected upstream review, 10 October 2026¶
This review covers the 39 upstream commits from
1adc59f7e371deb601fcfa18a14e25db238addcc
to 68804629e8ccd9d616d48a40e87de9fabbeae069,
the 0.4.2 release. The source comparison
records the review boundary, rather than a claim that every upstream change was adopted.
Laya-Pro remains version 0.3.29. Its English fallback, criteria insertion order and typed cache signatures are preserved. This change does not replace checkpoints, fit temperatures or recompute pretrained accuracy and calibration tables. The earlier review records the preceding merge and cache work.
Adopted changes¶
| Area | Result and upstream source |
|---|---|
| Python shortlist and tournament | Remove the original option_order after narrowing, with one warning per question per call. The previous permutation failed Agent validation. Caller data and passthrough questions stay intact. Source |
| Router diagnostics | Warn when a custom name is one edit from a built-in, including an adjacent letter swap. The custom name remains separate. Warnings promoted to errors leave registration and attachment unchanged. Source |
| HTTP responses | Add x_jev_confidence to choice and score answers for consumers using Jev's confidence definition. Existing confidence fields retain their meaning; strict Jev projection removes extensions. Source |
| Compose | Forward extra checkpoint registrations, idle unloading, batch token limits and strict Jev controls to the server. Registration controls, batch and wire controls; configuration guide |
| Local evaluation | Add saved-run agreement comparison, local student evaluation, paired case bootstrap intervals and a CLI. Validate aligned inputs and preserve unknown teacher revisions. Source, usage and limits |
| Java | Port single-option graph padding, grouped Router batches, tournaments and hook/error-ordering fixes. Further review fixed hook-rewritten batch cardinality and propagation of outer async deadlines. Single-option fix, batching, tournament, hook ordering |
Measured costs¶
The Windows/Python 3.12 setup benchmark alternates baseline aba90f3 and this implementation,
nine repeats of 10,000 calls. Warnings are ignored during timing; no checkpoint or model
forward runs.
| Operation | Before, median µs | After, median µs |
|---|---|---|
Router registration of built-in english |
4.664 | 4.593 |
Router registration of custom papers |
4.909 | 12.640 |
Router registration of typo englsh |
4.905 | 14.336 |
Shortlist without option_order |
24.596 | 24.398 |
The diagnostic adds configuration cost, depending on the name; it runs outside prediction. Stale-order timings were noisy and the baseline input failed real Agent validation, so no speed improvement is claimed. Harness, raw samples.
HTTP response enrichment took medians of 6.478 µs for three answers and 154.068 µs for 64 answers, seven repeats of 2,000 calls. These measure response enrichment alone, excluding inference, serialization and networking. Workstation load was uncontrolled in both benchmarks. Harness, raw samples.
The Java English smoke timed the same two requests through the Router: 1,501.15 ms sequentially versus 1,298.06 ms in one batch, median whole-call time with two ONNX Runtime threads. Each mode had two warmups and three timed repeats, alternating execution order; answers and usage stayed identical. These two sample inputs on an active workstation do not establish a general throughput improvement. Samples and settings.
Checks and limits¶
All 92 suites in the shared local Python runner passed; its pytest component reported 497 passed and one skipped. Targeted checks passed: 227 shortlist assertions, 802 routing assertions, 36 Router batch tests, 219 evaluation/agreement/shortlist-evaluation tests, 331 HTTP/server tests and 32 TypeScript HTTP tests. Contract checks passed: 50 evaluation API assertions and 674 hook/API assertions. A live TypeScript HTTP client also passed single/batch transport checks against local Python inference in offline mode, including the added confidence fields.
Java JDK 17 and 21 each passed 1,177 tests, with 23 skipped and no failures; the JDK 17 build also passed compilation with warnings as errors, Javadoc and packaging. Three added regressions cover extra or empty hook-rewritten batch states and an outer async-hook deadline.
Reference regeneration pins tokenizers==0.23.2: that version reproduced the committed Java
pre-tokenizer table exactly with gen_pretokenizer_tables.py --check. An unpinned tokenizer
can change Unicode classification and version-stamped output, producing fixture drift even
when the runtime code is unchanged. Pinned toolchain,
table generator.
The retained English checkpoint 55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 matched Python at
returned precision in a small Java check: two states with choice, score and yes/no questions,
singles and batches, single-option choice/score, Router calls and a 20-label tournament.
Maximum reported difference was zero at tolerance 1e-4. Multilingual and typed-checkpoint
graph checks remain skipped. This small workload does not establish general accuracy or calibration.
Raw parity results,
Python comparison harness,
Java harness,
regression suites.
The agreement example reproduced the saved report through the CLI on 32 fixed cases from a pinned public dataset. Only 1 of 32 student decisions matched the teacher reference, while the teacher repeat matched 26 of 32. This is poor agreement on a small academic panel, not production accuracy. The teacher's immutable model revision is unknown, and its prompt differs from Laya's typed inputs. No teacher API is required: observations are saved and student inference runs locally. Pinned metadata and measured summary.
TypeScript timeout changes remain deferred: the late compilation fixes do not complete timeout validation or stop an abandoned hook from modifying its context. Native TypeScript retains awaited hooks. The .NET multilingual default is not adopted; Laya-Pro keeps its English fallback. Docker was unavailable on the local Windows review host, so Compose environment controls were checked statically here. GitHub CI on PR 9 and PR 10 passed 28 checks per PR, including Docker HTTP and Linux amd64/arm64 smoke tests. Those runs precede the final Java follow-up.