SamplerScope | Ansh Dawda

SamplerScope was developed during the Digital Minds Research Sprint with Apart Research to isolate the effect of decoding on language agent behavior. The evaluation caches a model's action logits, applies multiple decoding rules to the same trace, and propagates each resulting policy through a known finite environment. It now has a public project page on Apart Research, while review and feedback are still pending.

Because the transition model is known, dynamic programming computes each outcome exactly. The analysis uses neither repeated rollouts nor an LLM judge. Each result is a paired comparison of decoder-induced behavior under fixed model outputs.

Evaluation summary

504
Local forward passes: 42
Decision states: 6
Action-label mappings: 11
Decoder configurations

Controlled evaluation design

A sampled action is often interpreted as evidence about the model. That interpretation is incomplete when greedy decoding, temperature scaling, or probability truncation changes the action distribution after the logits have been produced. Within each stratum, SamplerScope holds the prompt, grammar, environment, reward, and cached logits constant while varying only the decoder.

01 · fixedState and promptThe grammar, semantic actions, transition rules, and reward remain unchanged.
02 · fixedCached action logitsEach decoder reuses the same provenance-checked trace.
03 · variedDecoderGreedy, temperature, top-k, top-p, min-p, or an ordered composition.
04 · exactPolicy outcomesState occupancy, terminal probabilities, stakeholder cost, and return.

Within each model, environment, and label mapping, every decoder comparison reuses the same cached logits. Only the decoding rule changes before the induced policy is evaluated.

Benchmark design

The benchmark evaluates pinned Qwen2.5 0.5B and 1.5B Instruct checkpoints in two synthetic four-step environments. Semantic action order remains fixed, while the one-token labels A, B, and C are assigned to those actions in all six possible permutations. Each mapping is an exhaustive stratum, not a repeated sample.

Both environments use the following fixed scalar return:

R = \mathrm{success} - 0.5\,\mathrm{failure} - 0.25\,\mathrm{stakeholder\ cost}

On this page, benchmark-optimal means optimal under these weights. A different scalarization can alter which action satisfies that definition.

18 decision states

Service recovery

Choose whether to prepare a backup, repair a service, or restart it. The reward combines task success, failure, and stakeholder cost.

24 decision states

Queue control

Choose whether to add capacity, serve normally, or use burst service as load changes over four rounds.

Decoder configurations

Each trace is evaluated with raw softmax, greedy decoding, two temperatures, top-k, two top-p thresholds, and two min-p thresholds. The sweep also includes both orderings of temperature 0.5 and top-p 0.8, which tests whether the two operations commute.

Exact policy evaluation

For each state, the model produces full-vocabulary logits. A grammar mask restricts the logits to the three valid action labels, and decoder D transforms the masked logits into an action policy. The known transition model then propagates that policy through the four-step horizon.

\pi_D(a \mid s) = \operatorname{normalize}\!\left(D\!\left(\operatorname{mask}(z(s))\right)\right)

V_t^D(s) = \sum_a \pi_D(a \mid s)\!\left[r(s,a) + \sum_{s\prime} P(s\prime \mid s,a)V_{t+1}^D(s\prime)\right]

Dynamic programming computes exact return, terminal outcome probabilities, and state occupancy for every decoder condition.

Decoder effects on return

Decoder effects varied in direction across benchmark cells and label mappings. Greedy decoding increased return in 12 of the 24 fixed strata and decreased return in the remaining 12. Across all decoder conditions, the exact return shift relative to raw softmax ranged from −1.224 to +0.313.

Observed decoder effects

24
Fixed strata: −1.224 to +0.313
Return shift: 12 / 12
Positive / negative greedy shifts: 0.275
Largest order effect

Benchmark summary

The finite MDP and fixed scalarization define the optimum for each task. Raw softmax and greedy entries are equal-weight means across the six exhaustive label assignments. Service recovery and queue control differ in their dynamics, attainable returns, and optimal values, so comparisons should be made within columns rather than across tasks.

	0.5B · ServiceQwen2.5	1.5B · ServiceQwen2.5	0.5B · QueueQwen2.5	1.5B · QueueQwen2.5
Exact return
Benchmark optimumknown finite MDP	0.980	0.980	0.121	0.121
Raw softmaxdecoder baseline	0.491	0.566	−0.128	−0.288
Greedy	0.067	0.441	0.025	−0.298
Top-p 0.6
Optimal-action censoringraw-policy occupancy	49.5%	73.3%	18.5%	34.2%

Exact returns and top-p 0.6 censoring for all four benchmark cells. The six A/B/C mappings are exhaustive strata, not repeated samples.

Mean effect of greedy decoding

Greedy decoding increased the 0.5B queue result, reduced both service recovery means, and changed the 1.5B queue mean only slightly. The direction and magnitude of the effect therefore depended on both model size and environment.

Bars report the paired mean effect of replacing raw softmax with greedy decoding within each model and environment cell. Positive values indicate a higher exact return under greedy decoding. Each value is the equal-weight mean across six exhaustive label assignments; no error bars are shown because the assignments are fixed strata rather than repeated samples.

Variation across action-label mappings

Replacing semantic action names with A, B, and C holds the semantic action set fixed while changing which prompt token refers to each action. Greedy decoding increased return in 12 paired strata and decreased it in 12, but those shifts were distributed unevenly across the four benchmark cells.

Each circle is one exhaustive A/B/C assignment; color indicates whether greedy increased or reduced exact return. The rail is the observed assignment range, and the labelled diamond is the equal-weight mean. These ranges are descriptive and are not confidence intervals.

Decoder effects by condition

Extending the paired comparison to all decoder settings shows no common direction of effect. More concentrated distributions often reduced return in service recovery, while several of the same settings improved the 0.5B queue result. Applying temperature 0.5 and top-p 0.8 in opposite orders also produced different returns.

Each cell reports exact decoded return minus raw-softmax exact return, averaged equally across the six label mappings. Negative cells fall below raw softmax and positive cells exceed it. These are descriptive means over exhaustive strata, not population estimates.

Optimal-action censoring under top-p 0.6

Return measures the downstream effect of decoding. Censoring identifies one mechanism: top-p 0.6 can assign zero probability to every action that is optimal under the fixed scalar return before the environment receives an action.

Depending on the benchmark cell, top-p 0.6 assigned zero probability to every benchmark-optimal action for 18.5% to 73.3% of raw-policy decision occupancy. Each bar is the equal-weight mean across the six action-label mappings.

Sensitivity to surface labels

Permuting A, B, and C changed the prompt tokens and resulting logits while preserving the order of the semantic actions. Across the four model-task cells, the range of raw-policy returns among mappings was 0.227 to 0.445. Under greedy decoding, the corresponding range widths were 0.349 to 1.670. Averaged across the six mappings, the dominant surface label accounted for 85.4% to 95.8% of actions selected by greedy decoding of the raw logits in each model-environment cell.

Behavioral outcomes in this benchmark are properties of the model, prompt, grammar, and decoder together. Studies that attribute an action to a checkpoint should report the decoding configuration and test whether their conclusions persist under other plausible settings.

Scope and reproducibility

The repository contains four cached logit traces, their exact analyses, synthetic controls, trace hashes, and the source data for the figures. Reproducing the decoder analysis uses the committed traces and therefore requires neither model checkpoint nor an additional forward pass.

This controlled sprint benchmark supports only bounded conclusions. It covers two small checkpoints from one model family, two handwritten MDPs, one prompt family, one-token actions, and a four-step horizon. It does not establish a population-level effect and should not be generalized directly to free-form generation.

View exact traces and analyses