How to evaluate document extraction beyond valid JSON

A complete invoice can still name the wrong supplier

A JSON schema can establish that a supplier field exists and that a total is numeric. An arithmetic check can establish that the line items add up. Neither establishes that the supplier and line items came from the document.

Authored example: balanced totals with an incorrect supplier

Field	Source-reviewed reference	Candidate output
Supplier	Supplier A	Supplier B
Line-item amount	100	100
Tax	20	20
Total	120	120

This is a demonstration of a failure mode, not a measured model result. If supplier identity matters to the receiving system, the output fails even though it is complete and balanced. Likewise, replacing a line item with another item of the same price can preserve the total while corrupting the record.

Define correctness before comparing models

Write a short rubric with the person responsible for the downstream result. Specify which facts must match the source, how repeated fields are matched and when an uncertain document must be reviewed. Severity should follow the consequence of the error, not how easy the field is to score.

• Critical facts: supplier identity, invoice identifier, currency, quantities, amounts and the correct line items.
• Structure: required fields, types and the output contract expected by the consumer.
• Arithmetic: agreed relationships between subtotals, tax and totals, with explicit rounding tolerance.
• Minor differences: formatting differences that the customer confirms do not change meaning.
• Uncertainty: unreadable content, ambiguous references or information absent from the source.

Do not assume lowercasing is harmless for every identifier. Do not compare repeated line items only by array position unless ordering is a requirement. Preserve an explicit record of the matching rule, including how duplicate items and missing rows are handled.

Build reviewable examples and a separate test set

Each example needs the permitted source, its reviewed reference and enough context to resolve ambiguity. Count distinct source documents as well as outputs. Several outputs from one invoice do not provide the same diversity as several invoices from different layouts or suppliers.

Use development cases to inspect errors and improve prompts or checks. Keep a separate held-out set for assessing the frozen change. Where possible, split related document families together to reduce accidental overlap. The general reason is data leakage: letting test information influence development makes the eventual score too optimistic. scikit-learn's data-leakage guidance.

Include realistic difficult cases, such as missing optional fields, repeated items, noisy text and incorrect outputs that still reconcile. Keep ordinary production-like cases too. A collection containing only difficult failures cannot estimate the overall review rate of your normal workload.

Measure fields, documents and the evaluator

Field-level precision and recall explain different extraction failures: incorrect predicted fields and missing reference fields. Google Document AI calculates these metrics by matching predicted entities with test annotations, and reports them by label as well as in aggregate. Google Document AI evaluation metrics.

Also report whole-document acceptance under your rubric. A useful aggregate field score can hide one critical error on many documents. Show the source, reference and rejected output for representative failures instead of relying on a single headline percentage.

The evaluator itself needs evaluation. Have source review identify which outputs are acceptable, then compare the checker's decisions against those judgments.

Critical false-accept rate =
  critically wrong outputs accepted / critically wrong reviewed outputs

False-rejection rate =
  correct outputs rejected / correct reviewed outputs

As a hypothetical arithmetic example, take ten correct outputs and ten outputs with a critical error. A checker that rejects one correct output and accepts two critically wrong outputs has a 10% false-rejection rate and a 20% critical false-accept rate. It accepts eleven of twenty outputs, but that acceptance rate alone does not describe its reliability.

When a denominator class is absent, report the rate as unknown rather than zero. Always show counts. A small set with no observed critical false accepts does not establish a production failure bound.

Improve the checks without moving the goal

A legitimate evaluation improvement may stop rejecting harmless whitespace while continuing to reject the wrong supplier. Removing a critical supplier check would change the requirement, not improve the evaluator.

1. Identify a concrete disagreement with source review on development cases.
2. Propose a narrow rule change and retain its reason and version.
3. Freeze that proposal before evaluating fresh held-out cases.
4. Inspect false accepts, false rejections and the exact affected outputs.
5. Obtain explicit approval, then compare workflow candidates using the frozen rubric.

If a held-out failure leads to another rule change, those examples have become development evidence. Collect fresh cases for the next independent assessment. Avoid tuning a workflow and its grading rules together and treating the resulting pass rate as a verified improvement.

Separate benchmark evaluation from new-input verification

A reference checker can compare an output with a known answer. It cannot obtain the answer to a new document by reusing another document's reference. New inputs require an agreed source-verification method or human review, with its own accuracy and cost evidence.

Keemlo's local SDK preserves that distinction. Its reference checker refuses a different source. A structured verification callback records its method, decision, time and attributed charges. Structural-only approval, unresolved correctness or a verifier timeout requires review rather than automatic acceptance. This is an integration contract, not a universal source verifier.

The developer kit also contains calibration tools for supplied source-reference checks. Try the authored demo to inspect an error, or download the kit to evaluate your own permitted examples. Start with the mistakes that would make your downstream system wrong, then make their detection measurable.

Sources and further reading

• Google Document AI evaluation metrics
• scikit-learn guidance on data leakage

Keemlo team content, drafted and reviewed with AI assistance. Methods and product claims are checked against the linked sources and shipped software. About these guides.

PUT IT INTO PRACTICE

Inspect the trade-offs yourself.

Compare three approaches in the demo, inspect a rejected output and download a runnable configuration. The sample uses authored outputs, not measured model savings.

Try the demo ↗Test one workflow with us