Answering questions about a PDF does not prove document understanding

DocVQA includes over 12,000 document images with 50,000 questions, comparing existing models to humans at the time; it reported that while human accuracy was 94.36%, a significant gap remained for models, particularly requiring improvement in questions needing document structure understanding. [1]

In invoices or applications, even if strings are readable, answers change if the correspondence between item names and values, marginal notes, repeated tables, or page headers are misinterpreted; evaluations separate OCR string matching, correct region identification, and extraction of values corresponding to the question, sampling scan quality, rotation, handwriting, and multiple languages as distinct layers.

As a Catalysphere diagnostic proposal, compare answers to the same question using the original image, correct OCR text, and correct region information in sequence. If the image fails but correct text succeeds, investigate the reading stage; if field associations still fail with correct text, investigate structural understanding. Record failures by stage and form type, and set an acceptance condition that withholds automatic confirmation while a required stage performs below the current procedure. DocVQA does not validate the effectiveness of this diagnostic sequence. [1]

ChartQA collected 9,600 human questions and 23,100 summary-derived questions to evaluate visual and logical reasoning on real-world graphs; questions included not only references to series or colors but also operations such as maximums, differences, aggregations, and comparisons, leaving complex reasoning challenges for proposed models. [2]

For graph answers, legend series, axes, units, periods, value reading, and calculations are recorded one by one; judging final values alone as correct causes missed answers that coincidentally produced the same value from an incorrect series; questions relying only on color, cut axes, dual axes, stacked charts, and annotated figures are included as representative examples.

For graphs, we fix series, axes, units, periods, read values, and operations as scoring fields in addition to accuracy; even if the final value is correct, if any part of this path does not match the original figure, it is treated as a coincidental correct answer; the preservation of conclusions even with altered legend or axis representations becomes an adoption condition before transferring to different formats. [2]

TAT-QA combined tables and text from actual financial reports to build question answering requiring addition, subtraction, multiplication, division, counting, comparison, and sorting; the proposed model TAGOP achieved 58.0% F1, exceeding the previous best baseline by 11.1 points, but fell significantly short of the expert 90.8% F1. [3]

In financial and management materials, definitions, target periods, exceptions, and units are found in the body or footnotes, not just table values; instead of having AI generate only answers, we have it return structured data including used cells, text ranges, operators, and intermediate values; we hold cases where the original total does not match the recalculation result or where required values are not found.

Processing spanning tables and body text is tested by separating closed problems with all necessary cells and notes, problems lacking necessary information, and mutually contradictory problems; if values are filled in even when missing, the business fails regardless of arithmetic accuracy; when transferring evaluations obtained from financial reports to contracts or Japanese forms, both evidence extraction and calculations are re-verified for each document type. [3]

DocVQA, ChartQA, and TAT-QA address that character, placement, figure reference, and arithmetic are not the same capabilities. [1][2][3] However, benchmark scores alone cannot distinguish whether the company's answer read the original evidence or inferred the answer from form templates or question text.

The following are inspection recommendations; create documents with only font, color, column order, and page position changed while preserving original meaning, defining groups where answers should remain unchanged; conversely for groups where only units, target periods, or legend correspondences are changed, pre-write expectations that answers and evidence move according to the changed parts; attach reproducible rules to each deformation and regenerate the same contrasts after updates.

As a hypothetical example, if the answer changes when the same table is flipped horizontally, it may depend on position rather than values; if values and explanations do not move when only units are changed, it may indicate units were not read; this is treated not as determining failure causes but as a falsification test to narrow down suspicious processes, saved for each original document type.

For cases where answers diverged via two paths, do not majority-vote only on the final value; use the same document version, combine the path read directly from the image, the path assembled from extracted strings, and the result of definitive recalculation into one evidence bundle, formatted so a confirmer unaware of the answer can determine which path used which region and conversion; also create an adjudication display in the evidence bundle with the model answer hidden.

Even if values from both paths match, do not pass if the reference page or target period differs; conversely, if values differ but can be explained by rounding rules or display unit differences, classify them separately from reading failures; dividing adjudication reasons into 'Original Match', 'Rule Match', 'Insufficient Information', and 'Conflicting Evidence' avoids mixing coincidental matches with tolerable differences.

The recommended adoption condition is that the conclusion and reason category of adoption, hold, or rejection are reproducible when the same evidence bundle is passed to a different confirmer; cases that are not reproducible are not immediately returned to model accuracy aggregation but become targets for resolving correct definitions or document ambiguities; for out-of-scope formats, existing adjudication rules are not automatically inherited.

• TEST ID: Link original and deformation versions, specifying elements and formats that may be changed
• EXPECTED RELATION: Define in advance whether the answer remains unchanged or changes according to which part
• EVIDENCE BUNDLE: Bundle region, conversion, intermediate values, and difference reasons for each path into one item
• ADJUDICATION: Record conclusions by distinguishing original match, rule match, insufficient information, and conflicting evidence

The evaluation set is created as a table of document types (contracts, regulations, invoices, reports, slides, etc.) and question types (simple extraction, positional relationships, multi-location integration, arithmetic, figure/table interpretation); weight combinations with high impact if erroneous, including unanswerable questions, not just by frequency.

In the operation screen, show the evidence region of the page image simultaneously with the answer, allowing direct return to cited locations; high-impact values undergo dual confirmation with the original or definitive recalculation; when updating models or OCR, measure differences with the same evaluation set, checking if specific forms have worsened beneath the improved average; loop user corrections back to document types and failure processes to add to the next evaluation sample; for different formats with the same form name, track them as separate groups, requiring operations that prevent minority formats from being buried in the overall average.

Operational responsibility is divided among business owners managing form formats, quality personnel confirming answer paths, and technical personnel managing OCR or model changes; even if the average improves after updates, revert to the previous version if evidence region match or recalculation results worsen for high-impact document types; use the same failure samples for recovery judgment, not restarting based solely on claims of correction.

• Acceptance Condition: Confirm evidence match at least equal to current procedures for each target group of document and question types
• Hold Condition: Do not determine if any of page, region, unit, original value, or operation cannot be reproduced
• Change Management: Record changes to OCR, model, preprocessing, and form formats separately and compare using the same samples
• Responsibility Allocation: Business owners own correct answers and impact, quality personnel own re-verification, and technical personnel own recovery