Salesforce found why its RAG app scored 90% in tests but 46% for real customers

Salesforce's retrieval-augmented generation (RAG) pipeline exceeded 90% accuracy on standard text-retrieval benchmarks, but fell to approximately 46% when tested against complex, customer-representative enterprise documents such as insurance policies with multi-page tables and merged cells. The gap surfaced only after two quarters of customer escalations, and the obvious suspect was the LLM itself.

Rather than replacing the model or rewriting prompts, the engineering team worked backward from each wrong answer: was the retrieved context complete? If not, did retrieval pick the wrong chunk, or did chunking split related information, or did parsing lose it first? Testing showed table headers were getting separated from their data across page breaks, and fixed-size chunking was slicing procedures at arbitrary token boundaries, stripping the headings and diagrams that gave them meaning.

The team then changed one pipeline stage at a time and re-ran the same staged benchmark to attribute each gain. Adding page-level 'Intelligent Parsing' that routes complex tables and charts to model-based processing took a staged baseline from 65.1% to 84.4%. Enriching each chunk with metadata and question representations, combined with an embedding model that extended context capacity from 512 to 8,000 tokens, added another 2.4 points to 86.8%. Across the full set of changes, accuracy on the broader enterprise-document benchmark rose from approximately 46% to more than 90%.

Debug the pipeline backward, one isolated stage at a time, instead of blaming the model

Testing the pipeline backward, stage by stage, isolated which of parsing, chunking, enrichment, embedding, or retrieval actually caused each wrong answer, instead of guessing.

Changing exactly one stage before re-running the same benchmark made each gain attributable — 65.1% to 84.4% to 86.8% — rather than one opaque combined score.

Routing pages by structural complexity (simple text vs. tables/charts/diagrams) avoided sending every document through expensive model-based parsing.

The fix targeted where meaning was being lost — headers separated from data, procedures split at token boundaries — instead of asking a smarter model to compensate for missing context.

When an AI system fails in production despite passing benchmarks, isolate and re-test each pipeline stage separately before touching the model — the failure is usually upstream of it.

Salesforce also moved smaller-document indexing from a Spark-based batch architecture to a service-based just-in-time architecture, cutting indexing time three- to fourfold, from 30+ minutes to roughly seven to ten minutes. The team positioned GraphRAG, which traverses a knowledge graph across documents for multi-hop questions, for general availability in November 2026.