An AI knowledge base needs a defined path from source change to usable current evidence. Track versions, deletions, failed ingestion and cached answers, not only the last successful sync time. A recently indexed collection can still contain obsolete or incomplete information if its source and update rules are unclear.
Let’s plan your AI knowledge updates.
Bring your workflow and the decisions you need to make. We’ll help you clarify requirements, review the trade-offs and define a practical next step.
Identify the authoritative source
For each content group, record the owner, update mechanism and expected freshness. A published policy and an informal discussion thread may have different authority even when the thread is newer.
Define how superseded documents are marked and which version should answer a question about a past period. Removing every older version can make historical enquiries impossible; keeping every version without context can produce misleading current answers.
Create an update-state register
| Event | Requirement | Check to run |
|---|---|---|
| Source edited | Detect and process the intended version | Ask about the changed passage |
| New source added | Apply ownership and access rules | Retrieve with permitted identities |
| Source deleted | Remove or restrict derived material appropriately | Search and cached-answer checks |
| Ingestion fails | Surface incomplete coverage | Owner receives an actionable exception |
| Policy superseded | Prefer applicable authority and date | Conflicting-version evaluation |
| Connector unavailable | Explain freshness limits where relevant | Stale-source scenario |
Choose update targets from the business use. A slowly changing handbook and a live order-status assistant should not share an unexplained universal sync promise.
Include derived content and caches
For a hypothetical policy assistant, deleting a file from the source may leave chunks in an index and a prior answer in a cache. Map each derived store and define its update or removal behaviour.
Recheck access changes as well as content changes. The RAG permission lifecycle provides the companion access matrix; freshness and authorisation are related but separate responsibilities.
Evaluate with controlled changes
Use a synthetic document, change a specific rule and measure when the assistant consistently uses the new authorised version. Test failure and recovery rather than only a successful refresh.
Retain version identifiers in evaluation evidence. A reviewer should be able to tell whether an incorrect answer came from retrieval, stale source content or unsupported synthesis. Avoid claiming the model knew a policy that was not available in its evidence set.
Freshness questions
Is syncing every hour enough?
Only if that meets the workflow requirement and the complete update path achieves it. Measure actual change-to-answer behaviour.
Should the assistant always choose the newest document?
No. Authority and applicability matter alongside date. Define the precedence rules with the content owner.
Connect freshness tests to the AI monitoring runbook, then review source ownership with Drema's generative AI team.
