Why Vector RAG Fails on
a Bank's Connected Tables
Point plain RAG at a bank's payments, accounts and customer tables and it will answer confidently — and wrongly. Here is exactly why vector search breaks on connected data, and how GraphRAG fixes it, walked through on a real multi-table banking example.
I have spent the last few years building retrieval systems on top of relational data — banking, payments, and fraud. If there is one lesson I keep re-learning, it is this: the moment your questions depend on how records connect to each other, vanilla vector RAG stops working. Not "works a bit worse." Stops working — while still returning a fluent, confident, wrong answer, which is the most dangerous failure mode of all.
This article shows exactly where that wall is, and why GraphRAG walks straight through it. We will use one concrete example the whole way: a bank with five connected tables.
The scenario: a bank with five connected tables
A real payments system is never one table. Ours is a modest five, joined by foreign keys — a column in one table that points to a row in another — the kind of schema (table layout) you would find behind any retail bank:
customers
- customer_id 🔑
- name
- country
- risk_tier
accounts
- account_id 🔑
- customer_id ↗
- type
- opened_at
payments
- payment_id 🔑
- from_account ↗
- merchant_id ↗
- amount, ts
merchants
- merchant_id 🔑
- name
- category
- country
fraud_flags
- flag_id 🔑
- merchant_id ↗
- reason
- flagged_at
Notice the arrows (↗): every foreign key is a relationship. An account belongs to a customer. A payment moves from an account to a merchant. A fraud flag attaches to a merchant. The interesting business questions live in those connections, not inside any single row.
Here is the question we want a natural-language assistant to answer — the sort a risk analyst actually asks:
Answering that requires touching all five tables: filter customers by country, walk to their accounts, follow payments to merchants, keep only merchants that appear in fraud_flags, sum the amounts per customer, and filter by a threshold. That is four hops (four jumps from one table to the next) and an aggregation (adding the amounts up). Hold that thought.
Why generic (vector) RAG fails here
Generic RAG has exactly one move: turn each piece of text into a vector (a long list of numbers that captures its meaning — also called an embedding), find the vectors closest in meaning, and stuff that text into the prompt. To make our tables "retrievable," we first have to flatten them into text chunks — usually one chunk per row. Then we search by semantic similarity (how alike two pieces of text are in meaning).
Break it down into the four things this question needs, and vector search misses every one:
How GraphRAG works
GraphRAG changes what you retrieve over. Instead of a pile of independent text chunks, you retrieve over a knowledge graph: your data stored as nodes (the things — customers, accounts, payments) and edges (the links between them), which the AI can traverse — walk from one thing to a connected thing. The tables you already have map onto it almost perfectly — foreign keys simply become edges.
The five tables as one graph. Each foreign key is now a labelled, walkable edge.
One modelling choice worth calling out: a Payment is a node, not an edge. You could draw it as a single arrow from account to merchant, but because it carries its own properties — amount, timestamp, status — promoting it to a node lets you filter and aggregate on those directly. Rule of thumb: reserve edges for plain "belongs-to" links (OWNS, TO); anything with attributes of its own becomes a node.
Step 1 — Ingestion: tables become a graph, once
This happens offline, on a schedule. An ETL job reads the relational schema and materialises nodes and edges: each row becomes a node, each foreign key becomes a typed edge. Numeric columns (amount, timestamps) ride along as node properties so you can filter and aggregate on them later.
There is nothing exotic about it — a scheduled job streams each table and upserts nodes and relationships. In Neo4j that is a periodic LOAD CSV or a batched driver script built on idempotent MERGE statements, so re-running it never duplicates data. The foreign key is exactly where the edge gets created:
Step 2 — Retrieval: translate, traverse, then answer
At query time GraphRAG does something vector RAG never does: it lets the LLM write a structured graph query — in a language called Cypher, which is to graph databases what SQL is to spreadsheet-style tables — runs it against the graph database, and feeds back the resulting subgraph (just the connected records the query pulled out, already aggregated) to the model to phrase the answer.
Here is the graph query the model generates for our risk question. Because the graph knows the relationships, four hops and an aggregation collapse into a few readable lines:
The database returns a precise, complete result set — every qualifying customer, the exact total, and the specific accounts — and the LLM's only remaining job is to turn that into a sentence:
Vector RAG vs GraphRAG, side by side
| Capability | Vector RAG | GraphRAG |
|---|---|---|
| Follows relationships (joins/hops) | ❌ No — chunks are independent | ✓ Native — edges are the data |
| Aggregation (SUM, COUNT, HAVING) | ❌ Guessed by the LLM | ✓ Computed by the database |
| Exact filters (IDs, country, dates) | ❌ Blurred by similarity | ✓ Exact predicates |
| Completeness | ❌ Top-K truncates | ✓ Full result set |
| Best fit | Unstructured docs, FAQs, prose | Connected / relational / multi-hop data |
| Auditability | Hard — why these chunks? | Easy — the query is the citation |
When to reach for GraphRAG (and when not to)
GraphRAG is not a strict upgrade — it is the right tool for a specific shape of problem. Reach for it when:
- Your answers depend on how records connect — customers to accounts to transactions to counterparties.
- You need multi-hop reasoning, aggregation, or exact numeric/temporal filters.
- Completeness and auditability matter — finance, compliance, fraud, healthcare, supply chain.
Stick with plain vector RAG when your corpus is genuinely unstructured prose — policy PDFs, support articles, wikis — and questions are answered by a passage, not a join. And in practice the strongest production systems are hybrid: a graph for the structured backbone, vectors for the free-text notes hanging off it (a payment's memo field, a merchant's description), with a router deciding which to use per question.
“So why not just Text-to-SQL?” That is the honest first question for data that already lives in tables — and often the right answer. If every query is a fixed, known join, letting the LLM write SQL against your existing schema is simpler and needs no second datastore. GraphRAG earns its place when the traversals are variable-depth or open-ended (“find any chain of transfers linking these two accounts” — a query whose join count you don't know in advance), when relationships span systems no single SQL schema covers, or when one connected model has to answer many different question shapes. Reach for Text-to-SQL first; graduate to GraphRAG when the joins stop being predictable.