RAG cost per query is a lie — the real bill is the re-index you haven't done yet
Every RAG cost estimate I see starts with the wrong number. Someone prices the per-query LLM call, multiplies by expected traffic, and calls it a budget. Then three sprints in, the real bill arrives — and it's not the queries. It's the day you change your chunking strategy or swap your embedding model and have to re-embed the entire corpus. The recurring cost of production RAG is the re-index you haven't done yet.
Why re-embedding is the actual line item
Here's the thing nobody warns you about: you will re-index, probably more than once, early. Teams commonly go through one to three full re-embeds in the first six months — once when they refine chunking, often once when a better embedding model ships, sometimes once for a metadata change. And every one of those re-bills your whole corpus, not the delta.
The numbers are concrete. Embedding ~100 million tokens on a hosted API runs roughly $6–15 depending on the model (OpenAI's text-embedding-3-large is about $13 at $0.13 per million tokens — that one you can check against the list price yourself). That sounds cheap until you scale: 50 million documents at ~512 tokens each is 25 billion tokens, on the order of $500 per re-index event — and you're doing that every time you change your mind about chunking.
The lock-in nobody prices in
This is the insight that reframes embedding-model selection from a benchmark question into an architecture decision. Swapping the chat model behind your app is a string change that takes effect on the next request. Swapping the embedding model is not reversible cheaply, because embeddings from different models occupy geometrically incompatible vector spaces. The vectors you stored are meaningless to the new model. Every single one has to be regenerated and re-indexed before the new model can answer one query.
Choosing an embedding model is a lock-in decision wearing a benchmark score's clothing. You pick it in week three and live with it for two years — because changing it means re-embedding everything you own.
Which is why picking your embedder off the top of the MTEB leaderboard is a mistake. Aggregate benchmark scores hide task-specific trade-offs, and a chart-topping model can fall apart on your noisy real-world text. The right criteria are domain fit on your corpus, the cost to re-embed at your scale, and — underrated — whether you can self-host it, because an open embedder lets you pin a checkpoint indefinitely and never get a surprise API deprecation that forces a re-index on someone else's schedule.
What I'd actually do
- Budget for re-index, not just queries. Model at least one or two full re-embeds a year into the running cost. The query bill is the part you already see coming; the re-index is the one that blows the estimate.
- Track Recall@K monthly against a frozen eval set. You want a fixed set of questions with known-good answers so you can tell whether a candidate embedding model is actually better on your data before you pay to re-embed for it. A common bar: only migrate if the new model beats the incumbent by more than ~5% on your corpus.
- Prefer self-hostable, open embedders (BGE-M3 and friends) when the volume justifies it. The economics flip with scale: a hosted API wins below roughly 15M tokens/month, but a self-hosted GPU's fixed monthly cost (~$1,000+) amortizes once you're re-embedding constantly — and either way you escape the API-deprecation gun.
The fair caveat
If your corpus is small and static — a docs site, a fixed knowledge base — none of this really applies; a nightly or occasional re-index costs pennies and you should just use the best hosted model and move on. And at very high query volume against a rarely-changing corpus, the per-query generation cost genuinely does dominate again, and self-hosting embeddings becomes the rational play. The point isn't "re-indexing is always the biggest cost." It's that it's the cost everyone forgets to put in the spreadsheet, and it's the one that turns a tidy per-query estimate into a surprise. Price the re-embed before you commit to a dimension count.