The most useful output of an AI evaluation may be a decision not to ship the model.

That happened in a JodyInk research and content workflow. A local model candidate completed the technical run. It produced files, returned valid-looking structures in some cases, and could summarize portions of the supplied material. It still failed the role the product needed it to perform.

The no-go was not a general verdict on local models. It was a bounded answer to a specific product question: could this candidate produce grounded, structured cross-source analysis from the frozen source sets in the test?

Write the acceptance test before the output looks convincing

The evaluation used three source sets with different kinds of pressure: dense scientific material, contested policy research, and technical documentation. Each output had to satisfy four dimensions.

It needed the complete required section structure. Its claims had to remain grounded in the provided sources. Its machine-readable summary had to follow the expected schema. The result also had to pass the existing validator used by the rest of the workflow.

The gate required a clean pass across all four dimensions for at least two of the three topics. That threshold mattered because a fluent sample could not redefine success after the fact.

Without a written gate, a team can keep the most impressive paragraph and explain away the rest. The evaluation then measures enthusiasm rather than fitness for the job.

Structural validity did not guarantee grounding

At the smaller context setting, the candidate often produced all of the required headings. A structural validator could see the correct sections and report success on that dimension.

Inside those sections, however, the model named sources that were not in the frozen fixture set. Some labels appeared invented; other references seemed to come from general training knowledge rather than the files supplied to the run. A valid outline therefore contained unsupported attribution.

This is an important distinction for product teams. Schema compliance answers whether output has the expected shape. Grounding answers whether the claims belong to the permitted evidence. One can pass while the other fails.

A strong evaluation keeps those checks separate. Combining them into a single “quality” score makes it harder to see what kind of control the product actually needs.

More context made parts of the result worse

The candidate was also tested with a larger context window. More source material entered two of the runs, but both lost the required section structure. The added input did not simply produce a longer or more complete answer. It appears to have crowded out instructions that had held at the smaller setting.

That observation does not establish a universal relationship between context length and quality. The sample was small, used one candidate, and tested one workflow. It does establish a failure mode worth designing around: increasing the amount of available context can change instruction-following behavior, not just source coverage.

The practical response is not “use less context” as a rule. It is to test the actual source volume and output contract together. A model that works on one document may fail when the product asks it to compare seven.

A no-go preserves more than quality

Once the candidate failed the native test, the workflow stopped before evaluating application wrappers around it. That saved effort, but efficiency was not the main value.

The no-go protected the architecture from being organized around a component that had not met the grounding requirement. It also preserved the frozen fixtures, baselines, and validator, along with the failed outputs from the final run, so another candidate could be tested against the same task later.

Rejection becomes useful when it leaves behind a better decision surface:

  • the product role is named precisely;
  • the source sets and expected outputs are retained;
  • failure modes are recorded by dimension;
  • the result does not expand into a claim about every model; and
  • a future candidate can be compared without rebuilding the test around it.

The failed candidate is then evidence about the system, not wasted experimentation.

When another tuning pass changes the question

Prompt revision is a reasonable response when a candidate is close to the bar. A few examples might improve section compliance. Chunking or retrieval might increase source coverage. A larger instruction-tuned model could perform differently.

Those were legitimate next experiments. They were not reasons to turn the current result into a conditional yes.

The boundary between iteration and rationalization is the acceptance test. If a new retrieval layer materially changes the task, it belongs in a new evaluation. If a different model is introduced, it should run against the preserved fixtures. The team can learn from the failure without pretending the failed configuration passed.

A reusable evaluation record

For a model feature that reads sources and produces structured work, record at least six things:

  1. The exact job the model is being considered for.
  2. The source material it actually received, not merely the files available nearby.
  3. The required human-readable and machine-readable output shapes.
  4. The conditions that constitute a grounding or attribution failure.
  5. The threshold for acceptance across more than one representative case.
  6. The result that would stop the next integration step.

This record makes an AI decision easier to revisit. It also prevents a successful API call from being mistaken for a successful product capability.

The evaluation described here was limited. It did not compare every model, prove that local inference is unsuitable, or measure a production workflow. It did something narrower and more useful: it showed that one candidate did not meet one defined role, and it made “not yet” an executable product decision.

The evaluation sits behind turning research into reusable content.