Short answer: RAG chatbots get table numbers wrong because every stage of the pipeline was built for prose, not grids. Parsers flatten tables, chunking cuts headers off values, embeddings barely register numbers, retrieval can’t add up rows, and language models are shaky calculators. Fix them in that order: parse tables intact, chunk them whole, search with keywords plus vectors, and send arithmetic to code.
Here’s a tiny annual-report table, flattened the way many PDF pipelines flatten it:
(USD millions) FY2025 FY2024 / Revenue 4,210 3,980 / Cost of sales (2,530) (2,410) / Gross profit 1,680 1,570
Now imagine the chunker slices right after “Cost of sales (2,530).” The second chunk starts at “(2,410) Gross profit 1,680 1,570,” with no years in sight. A user asks, “What was FY2024 gross profit?” The retriever finds the chunk, and the model sees two numbers with no column labels. Some reports put the newest year first and others put it last, so the model guesses. It answers “1,680” in a calm, confident tone, and it’s wrong.
Nobody lied and nothing crashed. The chatbot just never saw the headers. (The example is illustrative, but it’s the pattern behind most “the AI got my table wrong” complaints.)
The research says this is no small problem. On FinanceBench, a test built from real SEC filings, GPT-4-Turbo with a shared vector store answered only 19% of questions correctly in the 2023 paper. Handed the right document directly, the same model reached 85% (FinanceBench). On a newer benchmark of 23,088 text-and-table questions, published at EACL 2026, the best retrieval setup in the original study scored 41% on numeric answers, versus 72–79% when the correct document was simply supplied (arXiv, April 2026). The models could do the math. They just weren’t given the right grid.
Where numbers die: the six stages
Think of the pipeline as six hand-offs. A number can be lost at every one.
- Parsing: the table is turned into text, or isn’t found at all.
- Chunking: the text is cut into pieces.
- Embedding and retrieval: the pieces are indexed and fetched.
- Context assembly: the fetched pieces are handed to the model.
- Generation: the model reads the grid and computes.
- Evaluation: someone checks whether any of it worked.
The rest of this guide takes them one at a time, with the best evidence available for each.
Stage 1: The parser flattens the table (or misses it)
A table in a PDF isn’t an object. It’s ink: lines, and text placed at coordinates. A parser has to rebuild rows, columns, merged cells and header hierarchies from geometry, and the tools differ enormously. In two vendor-run benchmarks from 2026, table-structure scores (TEDS, where 1.0 is perfect) ranged from about 0.93 for Docling to 0.73 for pymupdf4llm and 0.27 for markitdown (Nutrient), with some basic extractors scoring near zero (OpenDataLoader). Vendors benchmark their own corpora, so treat the exact numbers as directional. A separate 2026 benchmark found no frontier vision-language model dominated expert-level parsing, and that the specialist parser MinerU beat all the general models on table scores (Dr. DocBench).
Complex layouts make it worse. RealHiTBench tested 25 LLMs on tables with multi-level headers, found the benchmark hard, and showed that organising the headers into a tree helped models reason (ACL Findings 2025).
Spreadsheets add their own traps, starting with the fact that a sheet isn’t a table. One sheet can hold several tables, notes and blank rows. Microsoft’s SpreadsheetLLM team found that encoding every cell with its address, value and format was too token-hungry to be practical. They compressed sheets about 25 times and still reported only a 78.9% F1 score for finding where the tables are (arXiv). Everyday gotchas include:
- Merged cells. Only the top-left cell holds the value, so exports leave blanks under it.
- Two-row headers. A “Q3” banner over “Units” and “Revenue” vanishes when the sheet becomes a CSV.
- Formulas with no cached value. Files written by scripts can read back empty until something recalculates them.
- Displayed versus stored values. A cell showing 7.5% may store 0.075.
- Units in the title. “$000s” sits in a heading nobody indexed.
- Parentheses, dashes and locale. (2,530) means negative, a dash can mean zero or n/a, and 1.680,5 is a perfectly good number in many countries.
Stage 2: Chunking cuts the header off the numbers
Fixed-size chunkers don’t know a table from a paragraph. They slice where the token count says to, which is why a header row so often lands in a different chunk from the numbers it explains.
Researchers at Unstructured tested this on 80 SEC filings and found that chunking by document element, with each table kept as its own chunk, beat fixed-size chunking and removed the need to tune a chunk size (Yepes et al.). The authors work for a parsing vendor, but the logic holds up: a table is one unit of meaning. If a table is too big for one chunk, split it by row groups and repeat the title, the column headers and the units at the top of every slice.
Stage 3: Embeddings barely register numbers
Dense retrieval turns text into vectors that capture meaning. Meaning isn’t the problem with “FY2024 gross profit 1,570.” Digits are. A 2026 EACL Findings study tested 13 text embedding models on whether they could tell numerically different statements apart. Average accuracy was 0.54, against 0.5 for random guessing. Models struggled most with decimals, large integers and comma-separated numbers, and LLM-based embedders did better than older encoders (Numeracy Gap).
That explains a result that surprises people. On 23,088 financial questions over 7,318 documents, plain BM25 keyword search beat a top commercial embedding model on every retrieval metric except one near-tie at depth 20. Recall@5 was 0.644 for BM25 and 0.587 for dense search. Fusing the two lifted it to 0.695, and adding a strong cross-encoder reranker over 50 candidates reached 0.816 (arXiv, April 2026). Two cautions from the same study. Reranking only paid off with enough candidates (Recall@5 was 0.458 with 20 and 0.826 with 50). And HyDE, the trick of embedding a model-written fake answer, scored worse than plain dense search (0.544), because the fake answers contain plausible but wrong figures.
Stage 4: The right table never arrives (or only part of it)
Even with better search, retrieval is where most of the damage happens. In the original T²-RAGBench paper, ordinary RAG scored around 40% on numeric answers, while the same models with the right document supplied scored 72–79%. The authors concluded that retrieval, not reasoning, was the main bottleneck (Strich et al.). The April 2026 follow-up found the correct document missing from the top five results for 31.1% of queries. In a 100-case sample of those misses, 73% were classed as “table structure mismatch”: the answer lived in a table whose text form didn’t line up with how the question was phrased. An LLM assigned those categories, so read the percentage as indicative.
FinanceBench shows what that looks like from the user’s side. With one vector store per filing, GPT-4-Turbo answered 50% correctly. With a single store across all filings, only 19% were right, and 68% ended in a refusal or failure to answer. Depending on the model, a retrieval miss becomes “I can’t find that” or a confident number from the wrong company, year or table.
Then there’s aggregation. RAG fetches a handful of chunks, so “total revenue across all regions” can’t work if the regions sit in forty rows. Berkeley and Stanford researchers point out that RAG handles point look-ups and Text2SQL handles relational queries, while real questions mix both. On their benchmark of match, comparison, ranking and aggregation questions, standard RAG and Text2SQL baselines never passed 20% exact match, while hand-built pipelines combining database queries with a language model reported 20–65% higher accuracy (TAG paper). Those questions also need language knowledge, so the figure isn’t a pure table score, but the lesson stands.
Stage 5: The model misreads the grid, then does shaky math
Once the right table reaches the model, the format it arrives in matters. In Microsoft’s “Table Meets LLM” study, results shifted with table format, content order and prompting, and the best setup, HTML plus a format explanation and a role prompt, reached 65.43% overall on seven structure-understanding tasks (Sui et al.). A benchmark of multi-table questions found Markdown generally beat other formats, while HTML needed nearly three times the tokens of CSV (TQA-Bench). There’s no universal winner, so test on your own tables.
Then comes arithmetic. A 2024 study showed that how a number is cut into tokens changes whether a model can add it: forcing right-to-left grouping with commas lifted GPT-4’s addition accuracy from 84.4% to 98.9% (a March 2023 model) (Singh and Strouse). In February 2026, Google researchers found that frontier models degrade as numbers get longer, with many below 20% on 50-digit addition. Their errors came from misaligned digits and dropped carries, and the one setup that held up was the one running code (AI-rithmetic). Table cells are far shorter than 50 digits, but year-over-year changes, margins and totals chain several operations together, and each step is a chance to slip.
The nasty part is how it looks. FinanceBench’s authors noted that models often produced coherent, well-justified answers with extensive calculations, and those are the hardest errors to catch.
Stage 6: Your evaluation doesn’t notice
Many RAG test suites score relevance and faithfulness. An answer that cites the right table but swaps two columns can pass both. Number errors need number tests: compare against known answers with a tolerance, and track retrieval separately from reasoning. T²-RAGBench is built this way, and separating the two is what made it possible to say where the failures came from.
What actually works: a build order
- Route by question type. Look-ups (“What was FY2024 gross profit?”) suit RAG. Totals, comparisons, rankings and trends belong to SQL, a dataframe, or code the model writes and you execute. SpreadsheetBench found that letting models see execution feedback over several rounds improved results on 912 real forum questions (SpreadsheetBench).
- Parse tables as tables. Pick the parser by testing candidates on your own documents, keep the page image and table coordinates, and sample-check outputs against the source.
- Chunk by element. One table per chunk, with title, headers and units repeated on every slice.
- Enrich every chunk with metadata such as company, period, statement and units. In the 2026 study, contextual enrichment lifted Recall@5 by 2.2 points on hybrid search. A self-reported May 2026 leaderboard entry using metadata-aware BM25 with GPT-5.4 claims roughly 74% average numeric accuracy, close to the oracle level (T²-RAGBench leaderboard). Treat that leaderboard number as promising but unreviewed.
- Retrieve with hybrid search and a strong reranker, filter by metadata, and skip HyDE for numeric questions.
- Make the model show its work. Ask for the value, unit, period and source cell as structured output, do the arithmetic in code, and tell it to abstain when headers are missing.
- Verify before you display. Recompute totals, check that parts sum to the whole, confirm units and periods, and show the source table beside the answer. Keep a human in the loop for high-stakes numbers.
- Measure continuously. Keep a numeric test set and re-run it whenever you change the parser, chunker, embedder or model.
Spreadsheets need a different architecture
Don’t treat a spreadsheet as a document. Load it. Detect each table, unmerge cells, flatten multi-row headers into single column names, convert text numbers to numeric types, recalculate formulas so cached values exist, and record units. Store the result in a database or dataframe, give the model the schema plus a few sample rows, and let it write SQL or pandas that you run. Show the generated query next to the answer so a person can audit it. Use RAG only for the text around the numbers: definitions, notes and policies.
That’s the idea the research keeps pointing to. Language models are good at understanding the question and bad at being the calculator, so let each part do what it’s good at.
A 60-minute table audit
- Pick 30 real user questions: 10 look-ups, 10 comparisons or totals, and 10 involving units, periods or footnotes.
- Write the correct answers by hand, with the source cell for each.
- Run them, and log what the retriever returned, what the model answered, and whether the number matches within a tolerance.
- Label each miss using the cheat sheet below.
- Fix the most common label first, then re-run the same 30.
The cheat sheet for labelling misses:
- Wrong year or column: a header was lost in parsing or chunking.
- Off by 1,000 or 1,000,000: units or scale never made it into the chunk.
- Right number, wrong company or period: retrieval or metadata.
- “I couldn’t find it,” but it’s in the file: parsing or retrieval.
- Sums and growth rates that don’t reconcile: arithmetic, or missing rows.
How do I stop a chatbot from confidently giving a wrong number?
Make it return the value, unit, period and source cell. Recompute with code, reconcile against totals, show the source table, and have it abstain when headers or units are missing. Keep a human in the loop for high-stakes figures.
The bottom line
Table errors feel mysterious because the answers sound so sure. They aren’t mysterious. They’re a trail of small, fixable hand-offs: a header left behind, a chunk cut in the wrong place, a keyword that vectors don’t match, a sum the model guessed. Fix the trail in order and the numbers start to hold up.
Start with the 60-minute audit. It will show you which stage is costing you the most, and it’s a better use of an afternoon than swapping models.
