AI in practice · Advanced

Connecting your knowledge to the model: retrieval without big words

In one line: The model doesn't know your documents — you need to find the relevant passages for each question and attach them to the prompt.

The four steps

Chunking. Cut by structure — section, heading, row — and keep the document name and heading path in each chunk. Arbitrary cutting is the most common cause of bad answers.

Indexing. Combining meaning-based search with literal search beats either alone, especially in Hebrew and with identifiers.

Retrieval. Few relevant chunks, not many. More chunks cost more, slow down, and add noise.

Answering. Instruct the model to answer only from the material, cite a source, and say explicitly when there's no answer.

The pitfalls

Permissions forgotten in retrieval; conflicting documents with no date-based prioritization; questions that are actually data queries and not text search; and unnormalized Hebrew spelling.

How you know it works

Measure the two steps separately: did the correct chunk appear in the results, and given that it did — is the answer accurate? Measuring only the final result doesn't tell you what to fix.

Going deeper

Build a question set where each has a known correct chunk, and measure how often it appeared in the top five. That's the metric that drives most improvement. Add query rewriting before search, and keep context from a prior turn for follow-up questions.

Data Architecture

Next: Agents: when to let the system act on its own · Previous: How to choose a model — and why it isn't the first decision · Back to library