Generative AI, RAG and Agentic Frameworks
Three overlapping terms that describe three different amounts of risk. What separates a prompt from a retrieval system from an agent, and which one your problem actually needs.
The short version
- Generation, retrieval and agency are three levels of a ladder. Each one adds capability and a disproportionate amount of failure surface.
- Most problems labelled “we need an agent” are solved by retrieval plus a well-specified prompt.
- RAG quality is a retrieval problem far more often than a model problem. If the right chunk never reaches the context, no model saves you.
- Agents are worth it when the sequence of steps genuinely cannot be known in advance. That is rarer than the current discourse suggests.
Three terms get used almost interchangeably, and the conflation is expensive. Generative AI, retrieval-augmented generation and agentic systems are not three names for the same thing. They are three points on a ladder, and each rung up adds capability along with a considerably larger increase in ways the system can fail.
Being clear about which rung you actually need is the highest-value decision in the whole project.
Rung one: generation
A model, a prompt, a response. Everything the model knows comes from training, plus whatever you put in the prompt.
This is enough for a surprising number of real problems: summarising text you supply, classifying or extracting structure from documents, rewriting and translating, drafting from a template, converting between formats. If the task is a transformation of input you already have, you are on rung one and should stay there.
The constraints are well known. The model does not know your data, its training has a cutoff, and it will produce a confident answer to a question it cannot actually answer. That last point is the one that matters: a generation-only system has no mechanism for knowing what it does not know.
Rung two: retrieval
RAG adds a step in front of the model. Before generating, the system searches a corpus you control and puts the relevant passages into the prompt. The model answers from what it was given rather than from memory.
This is the right shape for anything grounded in your own content: documentation assistants, support answers drawn from a knowledge base, search across internal documents, answering questions about a contract or policy set.
The important and frequently missed point is that the quality of a RAG system is dominated by the retrieval half, not the generation half. If the correct passage does not make it into the context window, the model has nothing to work with, and a better model will only give you a more articulate wrong answer.
Where retrieval actually goes wrong
- Chunking that cuts through meaning. Splitting on a fixed character count severs tables, code blocks and multi-paragraph arguments. Chunk on document structure — headings, sections, list boundaries — and overlap slightly.
- Pure vector search. Embeddings are good at meaning and bad at exact tokens. A user searching for an error code, a product SKU or a specific function name is served far better by keyword matching. Hybrid search, combining vector and BM25-style lexical retrieval, outperforms either alone on almost every real corpus.
- No reranking. Retrieve twenty candidates, then use a cross-encoder reranker to pick the best five for the context. This is a small addition that produces one of the largest quality improvements available.
- Missing metadata filters. Users are usually asking about a particular product, version or time period. Filtering before you search is cheaper and more accurate than hoping the embedding captures it.
- Stale indexes. A retrieval system silently answering from documentation deleted six months ago is worse than no system, because it is trusted.
Diagnose retrieval before blaming the model. Take twenty questions your system answered badly and look only at the retrieved chunks. In our experience the correct information was absent from the context in the clear majority of cases, which makes it a search problem with a search fix.
Rung three: agency
An agent adds a loop. The model can call tools, observe the results, and decide what to do next, repeating until it judges the task complete. The sequence of steps is decided at runtime rather than written by you.
That is a genuine capability increase and it is also where the cost profile changes shape:
- Errors compound. A step that is right 95% of the time is right about 60% of the time after ten steps. Long autonomous chains are unreliable for arithmetic reasons rather than model reasons.
- Cost and latency become unpredictable. Every loop iteration is another inference call over a growing context. The same request can cost wildly different amounts on different runs.
- Debugging is genuinely hard. When a non-deterministic ten-step trajectory produces a wrong answer, finding the responsible step requires tracing infrastructure you have to build or buy up front.
- Tools are a security boundary. Any tool the agent can call, a sufficiently well-crafted piece of injected text in retrieved content can persuade it to call. If a tool writes data, sends messages or spends money, it needs authorisation independent of the model’s judgement.
The middle ground most teams should occupy
Between a single prompt and a fully autonomous loop sits the workflow: a fixed sequence of steps, each of which happens to use a model. Classify the request, retrieve the relevant policy, draft a response, validate it against a schema, route to a human if confidence is low.
Every step is testable in isolation, cost is predictable, failures are attributable, and you keep most of the capability. This is what a large share of successful production “AI agents” actually are, and there is no shame in it.
Choosing where to sit
- Does the task only transform input you already have? Generation.
- Does it need knowledge specific to your organisation? Add retrieval.
- Is the sequence of steps known in advance? Build the workflow explicitly.
- Is the sequence genuinely dependent on intermediate results in a way you cannot enumerate? Then an agent — with tracing, step limits, and a human gate before anything irreversible.
What to build regardless of rung
Three things separate systems that survive production from demos that do not, and none of them are model choices.
An evaluation set. A few dozen real inputs with known-good outputs, run on every change. Without it you cannot tell whether a prompt edit improved things or moved the failure somewhere you were not looking. This is the single most skipped step and the most costly to skip.
Tracing. Every prompt, retrieved chunk, tool call and response, stored and searchable. When a user reports a bad answer next week, you need to see exactly what the system saw.
A graceful failure path. “I do not have information about that” is a correct and valuable answer. A system that always answers is a system that sometimes fabricates, and users calibrate their trust on the worst answer they have seen.
Boolean Solutions experience with AI systems
We build these systems and, more often, we get called in after one has stalled between demo and production. The pattern is consistent: the demo works because it was tried on questions the corpus answers well, and the production version fails on the long tail nobody tested.
The fix is almost never a bigger model. It is better retrieval, a smaller and more explicit scope, and an evaluation set that makes the next change measurable. If you are working out which rung your problem needs, we are glad to talk it through, and you can read more about how we approach this on our AI and ML services page.
Further reading
- Building Effective Agents — Anthropic; the clearest published argument for workflows over agents
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — the original RAG paper
- Rerankers and Two-Stage Retrieval — a practical explanation of the highest-leverage RAG improvement
- Your AI Product Needs Evals — Hamel Husain, on why the evaluation set is not optional
Udit Mittal
Founder at Boolean Solutions. Twenty years of building and rescuing web, mobile and AI products for SaaS companies and startups — and writing down what actually worked.
Get in touch
