Helpers¶

Full detection result for a state.

Returns script, script_profile, language (best effort, may be None), is_english, non_latin_fraction and mixed_segment (the line or field that made a mostly English state non-English, else None). Only the segment scan reports mixed_segment; a per-value language verdict can return is_english=False with no segment.

String values are what get read. When a state has several of them, one non-English value is enough: joining every value into one window let a long English note fill the 4000 characters, or outvote a short German message, and that message was then sent to the English checkpoint (#384). The segment scan still stops at 4000 characters, which is what keeps a huge field cheap; a value it did not reach is read on its own afterwards.

Dominant script of text: 'latin', 'han', 'devanagari', ... or 'unknown' if there are no letters.

True when the English checkpoint can be expected to read this state.

Remove quoted email history, signatures and disclaimers to keep input focused.

max_chars is the length the result is cut to, 3000 characters unless raised -- see email_state, which takes the same budget and passes it through.

Construct a clean state dictionary for email classification.

max_chars is the budget clean_email_body cuts the body to, and it is worth raising for a long message: at the default the body stops after 3000 characters, so a request that arrives in the last paragraphs never reaches the model -- including through predict_long, which scans a state in windows precisely so it can read past one window's worth. Ignored when clean=False, which passes the body through whole.

Any other keyword becomes a field of the state, so it is read by the model; a typo here is an input mutation, not an error. The exception is a keyword whose value is exactly None: it is dropped rather than added, so email_state(..., thread_id=None) gives the same state as leaving the keyword out. The key is absent, not null -- and the distinction reaches the model, because laya.common.serialize_state turns the state into JSON before it is tokenized, so a retained None would put "thread_id": null in the prompt. sender is dropped on the same call by a different rule, falsiness rather than None, so sender="" leaves no from field either.

Preset questions for customer support ticket triage.

Preset questions for inbound email triage and threat filtering.

Preset questions for real-time LLM input guardrails.

Preset questions for content safety and moderation.

Preset questions for intelligent model routing.

How far each answer is from flipping, from 0 (on the boundary) to 1 (as far as it gets).

For choice and score answers the margin is the probability gap between the two most likely options. For noul answers it is twice the distance from P(yes) to the threshold the caller acts on, capped at 1; at the default threshold of 0.5 that is the gap between P(yes) and P(no), so every type is on the same scale.

A small margin means a small change can flip the answer: another run of a model whose output varies between runs, other hardware or precision, a reworded request or a recalibration. The margin is read from the payload alone, so it works on Laya's answers, on answers a DecisionCache replays, and on the same schema from a Jev-compatible API:

margins = decision_margins(router.predict(state, questions))
borderline = [name for name, margin in margins.items() if margin < 0.1]

Parameters:

• result (Mapping[str, Any]) –

  A prediction result, or its answers mapping.

• noul_threshold (Union[float, Mapping[str, float]], default: 0.5 ) –

  The P(yes) a caller acts on: one value for every noul answer, or a mapping from question name to threshold in which a missing name means 0.5.

Returns:

• Dict[str, float] –

  {question name: margin}, rounded to 4 decimals, for every answer that carries probabilities or a noul value. Other entries are left out.

Return the top-k choice labels for state.

embed_fn maps a list of strings to an array of shape (len(texts), dim). It is called once, with the query text first and then one string per option in criteria order. Option strings match render_options for a choice question.

When k is at least the number of labels, every label is returned in its original order and embed_fn is not called.

Ties keep the earlier label. Ranking is a signed cosine, not a similarity floor: a label that scores 0 -- no signal at all, or a non-finite vector treated as one -- does outrank an earlier label that scored negative, and k drops the negative labels first.

With return_scores=True the return is the (labels, scores) pair, where scores holds the signed cosine per kept label in rank order -- the same values predict_shortlist reports in its shortlist metadata. scores is None when nothing was dropped, exactly as in that metadata.

Shortlist each choice question, then call predict or system_one once.

Non-choice questions are forwarded unchanged. A choice whose label count is <= k is forwarded unchanged and does not call embed_fn. The caller's questions dict is not mutated.

A narrowed choice that carried an option_order has it dropped, with one RuntimeWarning naming the question: the order has one slot per option, so an order that fitted the caller's label set no longer fits the kept labels and the agent refuses the question. Remapping the order onto the kept labels would silently move options between slots, so it is dropped instead. A passthrough choice and non-choice questions keep theirs.

The returned dict is the model result plus a shortlist entry. Probabilities on a shortlisted choice are over the kept labels only. shortlist[qid] holds labels, scores, k, n, and passthrough. labels is the rank order a shortlist produced, or the criteria order itself when passthrough is set and no ranking ran; scores is the signed cosine of each kept label in that order -- negative included, never clamped to 0 -- or None when nothing was dropped.

Extra keyword arguments are forwarded to predict / system_one (for example model= on a Router).

Narrow each large choice question by elimination, then call predict once more.

A choice with more than group_size labels is cut, in criteria order, into groups of near-equal size and at most group_size. One predict call answers every group of every such question -- the groups go in as separate questions, so they share one forward pass -- and each group's answer goes through to the next round. Rounds repeat until no choice has more than group_size labels left; at the default 16, up to 256 labels take one round. Unlike predict_shortlist this needs no embedder, and each label is read with the option budget of a group_size-label question, not of the whole label set.

The final call answers every question of the request, with each choice that went through a round cut to its finalists. Non-choice questions and choices of at most group_size labels go to it unchanged, so when nothing needs a round it is the only call. The caller's questions dict is not mutated.

A choice that went through a round loses an option_order it carried, with one RuntimeWarning naming the question: the group and finalist questions hold fewer labels than the caller's, and an order no longer fits the kept labels. Remapping it onto them would silently move options between slots, so it is dropped instead. A choice that needed no round keeps its order.

The returned dict is the final call's result plus a tournament entry. tournament[qid] holds labels (the finalists, in criteria order), n (the label count) and rounds for each choice question. Probabilities, confidences and usage come from the final call, so a tournament choice's probabilities are over its finalists only.

Extra keyword arguments are forwarded to every call (for example model= on a Router). A Router's original question schema is routed once before narrowing, then that checkpoint and detected language are retained across rounds. Route hooks also run on each prediction.

Mean-pool the checkpoint encoder already loaded on agent.

The callable embeds a list of strings with agent.tok and agent.model.encoder. It does not run the decision head and does not download weights. A dedicated bi-encoder passed as embed_fn will usually shortlist better; this helper is for callers who only have the Laya checkpoint in memory.

Padding positions are excluded from the mean. The encoder's train/eval flag is left as the caller set it (a loaded Agent is already in eval). Each call uses the current agent.device, including after CPU fallback.

Cache embed_fn output per input string, under an LRU bound.

predict_shortlist embeds the query plus every option text on each call. When the same option set is shortlisted on every request -- a fixed intent or label list, as in the README's BANKING77 example -- the option rows do not change between calls, yet they are re-embedded every time. Wrapping the embedder once::

embed_fn = cached_embed_fn(embed_fn_from_agent(agent))

leaves the first call unchanged and reduces each repeat call to embedding the new query alone.

Lookups are exact string matches. Texts missing from the cache are deduplicated and embedded in a single embed_fn call, so a cold cache costs the same number of batched calls as the unwrapped function. Rows are stored as float32; the cache holds at most maxsize strings and then evicts the least recently used entry, bounding memory at about maxsize * dim * 4 bytes. Nothing is cached when embed_fn raises or returns a bad shape.

The wrapper is safe to share between threads: the lock covers only cache reads and writes, never the embedding call. The returned callable carries cache_info() -- a dict with size, maxsize, hits and misses -- and cache_clear(). Clear the cache if the model or weights behind embed_fn change. In-flight calls keep their captured rows but cannot refill a cache cleared after they started.

Validate opt-in abstention threshold min_confidence (#361, #394).

Either a real number in [0.0, 1.0] (one threshold for every answer; booleans rejected even though isinstance(True, int)), or a per-bucket mapping (see :func:check_min_confidence_map) so the threshold can differ by option count. Returns the value in its validated form -- a float for the scalar case, a dict[str, float] for the mapping case -- which the gate functions below both accept.

Validate a per-bucket abstention-threshold map (#394).

Keys are the option-count buckets in common.temp_bucket's spelling -- every :func:bucket_keys member ("choice:2", "choice:3-5", "score:6-10", "noul:2" and so on) -- plus the optional "default" used for any bucket the map does not name. Anything else raises: a key no answer can produce is not an unused entry, it is a bucket the caller believes is gated and is not, because :func:resolve_min_confidence falls past it to "default" and then to 0.0 -- gate nothing. Values are floats in [0.0, 1.0]. One confidence threshold does not transfer across option counts (#394); this lets a caller gate each bucket at the level its calibration actually earns. Fit one with :func:laya.calibrate.fit_abstention_thresholds, whose keys are this vocabulary by construction.

The threshold this answer's option-count bucket is gated at, under a per-bucket map.

Falls back to the map's "default" entry, then to default (0.0 -- gate nothing), for a bucket the map does not name, so an unconfigured bucket never abstains by surprise. A map that went through :func:check_min_confidence_map can only miss here by omitting a bucket: a key mis-spelled well enough that no answer produces it is refused at validation, because it would otherwise reach this fall-through and leave that bucket gated at nothing while its caller believes it is gated.

Opt-in abstention marker (#361): flag answers whose confidence falls below min_confidence.

Reads answer_confidence (max(p), the quantity the calibration figures describe and the one that does not drift with the number of options), falling back to confidence if answer_confidence is absent. The raw answer and confidence stay intact; low_confidence: True is added when the answer falls below the threshold, and removed if a previously-flagged answer now clears it (e.g. when a result dict is reused or re-evaluated with a different threshold).

min_confidence is either a float (one threshold for every answer) or a per-bucket mapping (#394), in which case each answer is gated at the threshold of its own option-count bucket via :func:resolve_min_confidence.

Exactly 0.0 runs this same loop rather than short-circuiting out of it. Nothing falls below it, so it marks no answer -- and the pass is what clears a marker an earlier, stricter threshold left on a reused payload, which is the state #910 was filed about.

Report the confidence gate's state, on the answers a gate was actually applied to.

A gate is a policy, and a policy whose application cannot be observed is not one. With min_confidence set, this writes abstention -- one of :data:GATE_STATES -- onto every answer, plus abstention_threshold, so a caller can answer three questions it otherwise cannot:

• what fraction of decisions abstained, rather than inferring it from whether low_confidence happened to be set;
• how many answers the gate could not decide on, which a boolean cannot express at all;
• what threshold produced these results -- flag_low_confidence consumes the threshold and drops it, so without this a batch run with per-class thresholds cannot be re-split.

GATE_UNEVALUATED is the case a boolean cannot express: the gate ran and the answer carried no usable confidence, so the gate could not decide. Reporting that as a pass is the same lie as reporting it as a flag.

With min_confidence unset, this writes nothing. No abstention, no abstention_threshold, no flag. That is the whole contract: an ungated call returns exactly the payload it returned before, and the presence of the field -- not a fourth value read out of it -- is what tells a caller the gate ran. Call it unconditionally, once per call, in place of an if min_confidence is not None: guard: that guard is what leaves a path reporting nothing at all, which is the state this function exists to distinguish.

The flag itself stays :func:flag_low_confidence's -- this delegates rather than re-implementing the rule, so the boolean and the reported state cannot drift apart.

A min_confidence of exactly 0.0 was set, so states are reported. Nothing can fall below it, so :func:flag_low_confidence marks no answer at it and clears whatever a stricter threshold left on a reused payload. Every answer carrying a usable confidence therefore reads passed, and the threshold echo is what distinguishes that from a real pass at a real threshold.

Probability mass on the answer being reported: max(p).

This is the quantity temperature scaling fits, and the quantity every calibration figure in this repository is computed on -- both benchmark harnesses take conf = max(probs) before calling ece_score. The README's gating section relies on the property that goes with it: of the answers returned at confidence c, about c of them are right. That property is conditional, and the condition is not met by default -- it holds only after the temperatures have been fitted and validated on held-out data for this checkpoint and this option count. The shipped checkpoints are over-confident: choice:11+ is fitted at 0.1006, a ~10x sharpener. clamp_temperature refuses it and applies TEMP_MIN (0.5), which still sharpens 2x. The clamp limits the distortion; it does not establish calibration. Thresholds still require held-out validation for the checkpoint and option count (issue #394).

confidence_from_probs below reports a different quantity on a different scale and carries no such guarantee, so the two must not be compared against the same threshold.

Normalized Shannon entropy confidence: 1 - H(p) / log(k).

How concentrated the whole distribution is. Useful, but not calibrated: it is not what temperature scaling fits and not what the reported ECE measures. See answer_confidence. For fewer than two options it returns 1.0, since there is no entropy scale. Abstention gates use answer_confidence first; this is only their fallback.

Expected Calibration Error across confidence bins.

Fit one scalar T by NLL + LBFGS on log T.

The result is clamp_temperature of the optimised scale, so it lies in [TEMP_MIN, TEMP_MAX] (or is the neutral 1.0 when the value is not a number). The search is bounded to LOG_T_MARGIN e-folds outside that band, which covers any minimum the answer can use; past it the float32 scale drifts toward exp() underflow, where the objective divides by zero and every gradient becomes NaN, and clamp_temperature reads that NaN back as the neutral 1.0. Returns 1.0 when fewer than min_n pairs are given. min_n defaults to MIN_BUCKET_N (the per-bucket floor). Type-level fits pass MIN_TYPE_N, which is lower, so a dataset that fills no bucket still gets a scalar instead of staying at 1.0.

Fit type-level scalars and per-bucket temperatures.

MIN_BUCKET_N is the per-bucket floor: smaller buckets are omitted from temperature_by_options and the type-level scalar covers them. MIN_TYPE_N is the separate, lower floor for that scalar only.

compute_ece=False (the default, and the path Agent.fit_temperatures stores) fits on every record and returns no report key. seed is ignored on this path.

compute_ece=True holds out ECE_HOLDOUT_FRAC of each bucket, stratified by temp_bucket, using seed so the same records always split the same way. Temperatures are fit on the remainder only and ECE is scored only on the held-out records. report["n"] is the number of records passed in; report["n_eval"] is the held-out count the ECE rests on. A bucket that would drop below MIN_BUCKET_N after the holdout is fit on all of its records, left out of the eval set, and named in report["buckets_excluded_from_eval"] instead of being dropped. n_by_bucket always counts the full input, including when the fit itself used a subset.

Fit a per-temp_bucket abstention threshold so a gate keeps a target error in every bucket.

A single min_confidence does not transfer across option counts (#394): the calibrated confidence of a 2-option and a 12-option answer live on different scales, so one cut over- or under-abstains depending on the question. This fits one cut per bucket instead, keyed exactly like temperature_by_options (common.temp_bucket, e.g. "choice:3-5"), and the result is a min_confidence map that :func:laya.confidence.check_min_confidence / :func:laya.confidence.apply_confidence_gate accept directly.

records are the same (qtype, logits, target[, k]) tuples fit_temperature_map consumes (records_from_labeled builds them). Confidence is the calibrated max(p) -- the logits are scaled by the fitted temperature / temperature_by_options first, so thresholds and the numbers the runtime reports are on the same scale. target_error is the tolerated error among accepted answers; min_bucket_n omits buckets too small to fit, and conservative adds a one-sample margin. The thresholds are empirical cuts on the calibration set, not a formal coverage guarantee -- validate on held-out data (fit_temperature_map(..., compute_ece=True) gives a held-out split) for a production gate.

Pass binning_map when the agent that will serve these thresholds has one installed -- by Agent.fit_binning, or by a calibration payload that carries binning_map -- because the runtime recalibrates answer_confidence through that map before anything reads it, so a cut fitted without it is a cut on a scale the gate never sees. The thresholds are then on the binned scale, and the order the two were fitted in stops mattering. Measured on 1,200 synthetic 12-option records at target_error=0.10: the cut fitted without a map holds 9.8% error over 50% coverage on un-binned confidences, and admits 94.5% of answers at 25.6% error once the same number is compared against binned ones.

Fit a per-temp_bucket histogram-binning recalibration map for answer_confidence.

Temperature scaling applies one scalar per bucket; it cannot fix a bucket whose reliability curve is not a simple sharpening/softening (the pathological choice:11+ the shipped English checkpoint carries is one). Histogram binning is the non-parametric alternative: split the calibrated confidences of a bucket into bins equal-width bins over [0, 1], and map every confidence that lands in a bin to that bin's empirical accuracy. It needs no monotonicity assumption and no extra dependency (NumPy only; isotonic regression would pull in scikit-learn).

records are the same (qtype, logits, target[, k]) tuples fit_temperature_map consumes; confidence is the calibrated max(p) (logits scaled by the fitted temperature / temperature_by_options first), so a binning map composes on top of a temperature map rather than replacing it. Returns {bucket: {"bins": N, "values": [recalibrated confidence per bin]}}; buckets below min_bucket_n are omitted. Apply it with :func:apply_binning_map. An empty bin (a confidence range the calibration set never produced) maps to its own midpoint, i.e. leaves that region unchanged, so an unseen value is never recalibrated to a fabricated 0.

Recalibrate one answer_confidence for its option-count bucket (common.temp_bucket).

Returns the confidence unchanged when the map has no entry for the bucket, so a bucket the map was not fit for passes through rather than being forced to a wrong value.

Fit a histogram-binning map on top of this agent's fitted temperatures and store it.

records are the same (qtype, logits, target[, k]) tuples as fit_temperatures consumed. The map is keyed exactly like temperature_by_options, composes on top of the current temperatures, and is written out by save_calibration as binning_map.

Render option texts in label-index order. Noul semantic order is always [false, true].

Strictly proper scoring rule reward: log score + spherical score + ranked probability score.

q: [..., N, K] reported distributions target: [N, K] (one-hot or soft target distributions)

TD(lambda) targets for multi-turn conversation trajectories.