Where a request's time goes, and what was cut¶

The profile¶

Milliseconds per request, median of 10 runs:

The ticket is 97 characters and the email 1,125. The encoder is 89-95% of every request measured. Every question row carries the whole state, so time follows the number of questions times the length of the state. The full table, with batches and Spanish, is in Where a request's time goes.

Speed-ups made, with identical output¶

Language detection output was compared against the previous code: 267,449 comparisons over 38,247 inputs, none different. These cut milliseconds from a request that takes hundreds on CPU. They matter more on an accelerator, where the encoder is short and the milliseconds around it are a visible share.

Runtime options tried¶

Use every physical core, and use bf16 only on hardware with native support.

The CPU fast path¶

Over 13 English requests (1 to 10 questions, tickets to 3,000-token emails), interleaved over 5 repeats:

• ONNX fp32 loses on long inputs because the export unfuses attention into a full [heads, length, length] tensor per layer. It also added a NaN check after each softmax that cannot fire. scripts/export_onnx.py now strips those checks, which took the email from 0.79x to 0.90x. Fusing the attention is the next step.
• int8 changes answers. Every changed answer had an fp32 margin of at most 0.0006, which is a coin flip under random weights. Trained checkpoints will change far fewer; measure yours. The int8 ONNX file is 579 MB, against 1.68 GB for fp32.
• The cache pairs with int8. Fp32, int8 and ONNX of one checkpoint share one set of decisions (see the cache report), so a repeated request gets its first answer whichever backend computed it.

Trimming the state¶

Since time follows state length, trimming is the largest lever on a new request. laya.email.clean_email_body drops quoted history, signatures and disclaimers, and caps the rest at 3,000 characters:

The quoted history is gone too, so keep it when the questions depend on it.

Encoding the state once (research spike)¶

k questions put the state through the encoder k times. Encoding the state once and each question separately, then letting the two head layers cross-attend, would cost this much of today's forward pass (speedup, median of 3):

It pays from about three questions on a hundred-token state, and loses on one question. It is compute only: the answers mean nothing until a model is trained this way, most likely by distilling from today's.

Reproduce¶

python research/scripts/bench_stages.py --device cpu     # or mps, cuda
python research/scripts/bench_question_heads.py
python scripts/export_onnx.py --model ID --output english.onnx --int8
python research/scripts/bench_cpu_fast_path.py --onnx fp32=english.onnx --onnx int8=english.int8.onnx
python research/scripts/spike_encode_once.py