Field notes / AI
Giving an AI access to your documents is useful. Making sure it retrieves the right evidence before answering is the real engineering problem.
Igniter Studio · 2 Oct 2026 · 4 min read
One of the most compelling AI demos is simple.
Upload some documents.
Ask questions.
Get answers.
It feels like the model suddenly knows everything about the business.
Usually, it does not.
A well-built system is doing something more practical:
finding relevant information first, then asking the model to answer using that information.
This general pattern is commonly called retrieval-augmented generation, or RAG.
The name sounds more complicated than the idea.
Imagine a company has:
You could theoretically try to train a model on all of it.
Often that is unnecessary.
Instead, when someone asks:
What is our warranty process for commercial customers?
the system searches the relevant knowledge first.
Maybe it finds:
Those sections are supplied to the model with the question.
The model then produces an answer grounded in the retrieved information.
The important thing is not that the AI “knows” the policy.
It found the evidence required to answer.
This is one of the biggest things people miss.
You can use an excellent language model and still get poor answers if the retrieval layer gives it the wrong information.
Suppose the question is:
What happens when a customer cancels within 24 hours?
If the system retrieves an old policy from 2023 instead of the current policy, the model can produce a beautifully written wrong answer.
That means the real engineering questions include:
The model is only one part of the system.
A useful knowledge system should understand source quality.
Imagine these documents all mention refunds:
They should not necessarily be treated equally.
The system might prefer:
Current approved policy
over
informal historical discussion
Metadata can help.
For example:
Retrieval becomes much stronger when the system understands more than raw text.
If an AI gives an important answer, users should ideally be able to see where it came from.
For example:
Commercial cancellations made more than 48 hours before the booking may be rescheduled without a fee.
Source: Commercial Booking Policy → Cancellation → Section 4.2
That changes the experience.
The answer becomes inspectable.
Users can verify it.
And if it is wrong, they have somewhere to investigate.
This is especially valuable for:
Confidence should come from evidence, not just fluent writing.
Not every question requires a generated answer.
Suppose a user searches:
invoice template
The best result might simply be:
Invoice Template.docx
No summary needed.
A good knowledge product can combine:
AI should improve retrieval, not make straightforward information harder to access.
Imagine a company knowledge assistant.
A staff member asks:
What is the CEO's current compensation package?
The system finds a confidential board document.
Technically, retrieval worked perfectly.
Security did not.
A knowledge system should respect the same access rules as the underlying information.
If the user cannot access a document normally, the AI should not be able to retrieve it on their behalf.
This is a critical part of production RAG systems.
If the system cannot find reliable evidence, it should be able to say so.
For example:
I could not find a current policy covering this scenario.
That answer is much more useful than inventing something plausible.
Useful AI systems need an evidence threshold.
Not every question deserves an answer.
“Ask anything about our entire company” sounds impressive.
A narrower system is often better.
For example:
Ask about workplace policies
or:
Search technical product documentation
or:
Ask questions about this client's project files
A clear domain makes it easier to:
You can expand later.
A proper evaluation should include questions with known answers.
For each question, check:
Did the system retrieve the correct source?
Did it ignore outdated or irrelevant sources?
Did the answer accurately reflect the source?
Did it cite the evidence?
Did it refuse appropriately when evidence was missing?
This separates a useful knowledge system from a polished demo.
The most important question is not:
How smart is the AI?
It is:
Can the system reliably find the right information before the AI speaks?
Your next chapter starts here
Tell me what you’re working on. I’ll help you find a practical way forward.
Let’s talk about it