{ "schemaVersion": "1.0", "language": "en", "canonicalUrl": "https://catalysphere.co.jp/en/articles/knowledge-base-content-operations/", "machineReadableUrl": "https://catalysphere.co.jp/en/articles/knowledge-base-content-operations/index.json", "publisher": { "name": "Catalysphere", "url": "https://catalysphere.co.jp/", "description": "A research and consulting studio connecting AI and big-data research with practice." }, "article": { "slug": "knowledge-base-content-operations", "title": "Internal AI answers fail on knowledge lifecycle before search accuracy", "summary": "RAG accuracy is determined not only by the model but also by how quickly document version, expiration, source, permissions, and deletion are reflected in the index; we consider operational design from knowledge benchmarks.", "takeaways": [ "Internal knowledge is not data ingested once, but a business product that requires continuous maintenance of source, version, expiration, permissions, and deletion.", "This is because even if the search engine is correct, answer quality cannot be maintained if the knowledge source is outdated, conflicting, unable to trace back to evidence, or has different access permissions.", "Assign document owners and expiration dates, measure the time from change to index reflection, and continuously test freshness and provenance using evidence-based questions." ], "publishedAt": "2026-07-13", "updatedAt": "2026-10-01", "readingMinutes": 8, "topics": [ "Knowledge Management", "Content Operations", "Information Freshness", "Data Governance" ], "domains": [ "data-analytics", "organization-adoption", "governance-risk", "generative-ai" ], "evidenceKind": "research-review", "status": "published", "image": { "prompt": "Abstract scientific editorial illustration for the research question “社内ナレッジを、AIが使える状態に保つには何が必要か”, interpreted symbolically rather than typographically: separate knowledge clusters connected through a shared operating network, with a few gold checkpoints making coordination and handoffs visible; deep Catalysphere navy and indigo field, muted scientific blue, restrained antique-gold structure, translucent geometry, fine particles, subtle paper grain, generous negative space, one clear centered metaphor that survives a narrow mobile cover crop, calm premium research-journal art direction, no readable text, no logo, no literal chart, no photorealistic people.", "negativePrompt": "text, letters, numerals, watermark, logo, dashboard UI, humanoid robot, server rack, binary rain, stock-photo office, literal quantitative chart, red alarm graphics, neon cyberpunk, purple gradient, clutter", "alt": "An abstract diagram showing multiple knowledge clusters connected to a single operational network via golden verification points", "aspectRatio": "16:9", "assetPath": "article-art/knowledge-base-content-operations.png", "status": "approved" }, "lead": "If we improve the search model, can the answers of internal AI be kept correct for a long time? KILT evaluated shared knowledge sources including provenance, StreamingQA evaluated knowledge that increases over time, and FreshQA evaluated answers requiring updates and false premises. [1][2][3] What these three studies reveal is that the knowledge base requires not only search accuracy but also version management and update responsibility.", "sections": [ { "heading": "Manage not only the answer but also which knowledge it is based on", "paragraphs": [ "KILT connected eleven datasets spanning five tasks—fact-checking, entity linking, relation extraction, open-domain question answering, and dialogue—to a single Wikipedia snapshot. It is a common framework that evaluates not only the final output but also the provenance of evidence across approximately 3.2 million cases. [1]", "Within the enterprise, the basis for answers should be traceable not just to the document name, but to the document ID, version, section, owner, and validity period. Even if ranked high in search, answers must not be adopted if old and new versions of the same regulation, company-wide and regional versions, or official procedures and reference materials cannot be distinguished. Provenance is not a decoration for explanation but the primary key for reproducing and correcting answers.", "The acceptance condition for provenance is that one can return from the answer to the corresponding section of the valid original text, and that the same authorized verifier can re-acquire the same evidence. If only the document name matches but the version or scope differs, it is disqualified. By categorizing the reasons for unsearchable evidence into unindexed, permissions, missing identifiers, or conflicting versions, we can return them to the responsible process rather than summarizing them as model errors. [1]" ] }, { "heading": "Index updates alone may leave old reasoning behind", "paragraphs": [ "StreamingQA used approximately 11 million timestamped news articles from 2007 to 2020 over fourteen years, evaluating question answering by feeding new articles quarterly. It reported that while adding new articles to the search space allows for rapid adaptation, if the base language model remains old, performance is lower than that of an updated model. [2]", "In other words, simply adding new documents to the index does not necessarily eliminate conflicts with existing premises or vocabulary. For changed product names, organizations, or rules, run regression questions asking about old knowledge and new questions asking about new knowledge simultaneously. In addition to index reflection time, the time until old answers can be rejected becomes the freshness metric.", "Update tests pair the ability to return new answers with the ability to reject old answers. By comparing the state where new documents are added to search candidates, the state where old documents are excluded, and the state where the base model or summary cache is updated, we can separate index reflection from the persistence of internal and derived knowledge. While old answers remain, display the status as \"in transition\" rather than \"update complete\" and stop automatic confirmation. [2]" ] }, { "heading": "Test updated answers and false premises separately", "paragraphs": [ "FreshLLMs created 600 questions for FreshQA categorized into four types: unchanged, slowly changing, rapidly changing, and false premises, conducting over 50,000 manual judgments. FreshPrompt, which incorporates search results, reported a 32.6% improvement under loose correctness criteria and a 49.0% improvement under strict factual criteria compared to GPT-4 alone at the time. [3]", "The same study also shows that the number and order of acquired evidence affect correctness, and that concise answer instructions reduce hallucinations. [3] In knowledge operations, we must hold not only whether new documents exist but also the priority of conflicting evidence, rules to negate false premises, and the next review date for items where answers change. Update frequency should align with the speed of answer change, not document type.", "For conflicting evidence, do not always choose the one with the newer date, but assign priority rules based on formality, scope, validity period, and exception clauses. Whether to force an answer to a question with false premises or to correct the premise and hold it should be a separate test item. To transfer results improved by external search to internal regulations, it is a condition that these priority rules can be reproduced in the evidence display. [3]" ] }, { "heading": "Include 'as of when' in the question contract", "paragraphs": [ "StreamingQA used timestamped news to compare adaptation to knowledge updates, but it is not a study that verified the operation of future implementation or retroactive corrections of internal regulations. [2] Do not extrapolate direct effects from this; instead, treat the separation of question time and application time as a design issue for knowledge where answers change over time.", "The following are operational recommendations. Before answering, specify whether the question seeks 'current', 'as of specified date', or 'after planned implementation', and on the evidence side, include not only the publication date but also the start and end of applicability. If a question lacks a time point, do not assume the latest; explicitly state the selected time point in the answer, and if there are multiple valid interpretations, return to confirmation.", "As a hypothetical example, if you ask about a procedure scheduled for revision next month for a case this month, returning only the latest document is incorrect. Create separate current and future answers, and schedule the execution of representative questions spanning the implementation date along with the question ID. Record cases where the future version mixes in before implementation and cases where the current version remains after implementation as separate failures." ], "visual": { "kind": "record-table", "eyebrow": "CATALYSPHERE / PROPOSED RECORD", "title": "Keep publication, effective and requested dates separate", "caption": "An unvalidated Catalysphere operating proposal, not a table of observed research results or validated thresholds. Enter local evidence and unknowns.", "provenance": "catalysphere-authored", "columns": [ { "id": "dimension", "label": "Record field" }, { "id": "contrast", "label": "Date to verify" }, { "id": "decision", "label": "Decision protected" } ], "rows": [ { "id": "r0", "label": "Publication date", "cells": [ "When the document became public", "Do not treat a published future rule as already effective" ] }, { "id": "r1", "label": "Effective dates", "cells": [ "Start and end of applicability", "Do not erase facts valid during earlier periods" ] }, { "id": "r2", "label": "Requested date", "cells": [ "Now or a specified historical date", "Select the applicable version under the question contract" ] }, { "id": "r3", "label": "Unconfirmed date", "cells": [ "Missing date and verification owner", "Hold or qualify the answer rather than inventing a date" ] } ] } }, { "heading": "Reverse-index questions to re-verify from change diffs", "paragraphs": [ "Running all representative questions in the same way every time does not reveal whether the test has reached the changed areas. The recommended addition is a reverse-index table linking document sections or claims, the questions relying on them, summaries, and derived rules. From claims with diffs, mechanically candidate the affected answers and user categories, including the reason for the change.", "If only testing affected candidates, omissions are self-reinforcing, so leave a small number of questions deemed irrelevant to the change as negative controls. If even the controls change, suspect alternative paths such as index ranking, shared summaries, or overly broad application rules. Placing both answers that should change and those that should not is the key point of diff testing.", "The condition for closing a change is not only that the target question has switched to the new claim. Verify that unaffected controls are maintained, derived items referencing old claims are removed from the list, and inquiries with changed time points return to the intended version. Claims with unknown dependencies are treated as \"under investigation\" rather than \"updated.\"" ], "bullets": [ "ASSERTION: Identify changed claims by linking them to sections, applicability, and effective time.", "DEPENDENCY: Reverse-index questions, summaries, derived rules, and user categories that use those claims.", "EXPECTED CHANGE: Pre-separate answers that will change, those that should be maintained, and time-specific answers.", "NEGATIVE CONTROL: Detect unexpected ripple effects from changes in questions deemed unaffected." ] }, { "heading": "Make the knowledge base a product with responsibility and SLOs", "paragraphs": [ "Assign owners, approval status, scope, validity start/end, and confidentiality level to each document, and define the flow by which publication, update, expiration, and deletion are transmitted to the index. Upon deletion, track not only the body text but also embeddings, caches, summaries, and evaluation copies. Filter search candidates by viewer permissions; do not rely on designs that hide them after answering.", "Operational metrics include the time from change to searchability, the time until expired documents disappear, answers without evidence, conflicting evidence, permission violations, and absent owners. Link representative questions to document change events and execute them automatically upon each update. By maintaining a separate change log from model improvements, we can isolate whether answer degradation comes from the knowledge source, index, or generation.", "For each change event, the document owner owns the master copy and validity period, the knowledge platform team owns reflection to the index/cache/derived items, and the quality team owns the regression of representative questions. If expired documents remain in candidates, unauthorized evidence is acquired, or the owner cannot be identified, exclude the relevant area from the answer scope. Lift the restriction only after verifying recovery, including the deletion path." ], "bullets": [ "Master Copy Management: Make document ID, version, owner, scope, validity period, and confidentiality level mandatory.", "Freshness Determination: Test the reflection of new answers and the rejection of old answers as pairs for each change event.", "Deletion Tracking: Track body text, embeddings, caches, summaries, and evaluation copies in the same log.", "Stop Condition: Exclude areas where expiration, conflict, permission violation, or absent owner is detected from answers." ] } ], "note": "KILT uses a fixed Wikipedia snapshot, StreamingQA uses English news, and FreshQA is a 600-question English-centric benchmark with search-engine-dependent methods. [1][2][3] Permissions, legal retention, deletion guarantees, Japanese documents, and organization-specific update speeds of internal regulations require separate verification.", "sources": [ { "id": 1, "authors": "Fabio Petroni et al.", "year": 2021, "title": "KILT: a Benchmark for Knowledge Intensive Language Tasks", "venue": "Proceedings of NAACL-HLT 2021, 2523–2544", "url": "https://doi.org/10.18653/v1/2021.naacl-main.200", "finding": "Linked eleven datasets and five knowledge aggregation tasks to a single Wikipedia snapshot, evaluating output and evidence provenance.", "limitation": "In research settings where necessary evidence exists within the knowledge source, it does not reproduce corporate missing documents, permissions, conflicting versions, deletions, or update workflows." }, { "id": 2, "authors": "Adam Liska et al.", "year": 2022, "title": "StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models", "venue": "Proceedings of the 39th International Conference on Machine Learning, PMLR 162, 13604–13622", "url": "https://proceedings.mlr.press/v162/liska22a.html", "finding": "Measured the difference in adaptation to temporal knowledge caused by adding new articles to the search space and updating the base model using fourteen years of timestamped news.", "limitation": "Limited to English news and quarterly evaluation, it does not handle internal document approval, access control, deletion obligations, or fine-grained real-time updates." }, { "id": 3, "authors": "Tu Vu et al.", "year": 2023, "title": "FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation", "venue": "arXiv:2310.03214, preprint", "url": "https://doi.org/10.48550/arXiv.2310.03214", "finding": "Using 600 dynamic FreshQA questions and over 50,000 manual judgments, it analyzed the conditions under which FreshPrompt using search evidence improves factual correctness.", "limitation": "As an unreviewed preprint, it relies on models at the comparison time, Google Search, and relatively short English questions, making direct generalization to internal search impossible." } ], "visual": { "eyebrow": "KNOWLEDGE OPERATIONS / FRESHNESS", "title": "Knowledge is not a stored artifact but an updated state.", "caption": "The numbers are within the scope of the three studies handling provenance, temporal adaptation, and update questions. They indicate dimensions to be maintained in operations, not superiority or inferiority of scale.", "metrics": [ { "value": "11 datasets", "label": "KILT's dataset linked to shared knowledge sources and provenance", "source": "Petroni et al., 2021", "referenceId": 1 }, { "value": "14 years", "label": "The period of timestamped news handled by StreamingQA", "source": "Liska et al., 2022", "referenceId": 2 }, { "value": "600", "label": "Questions in FreshQA including updated knowledge and false premises", "source": "Vu et al., 2023", "referenceId": 3 } ] }, "url": "https://catalysphere.co.jp/en/articles/knowledge-base-content-operations/", "machineReadableUrl": "https://catalysphere.co.jp/en/articles/knowledge-base-content-operations/index.json", "topicFacets": [ { "id": "organization-adoption", "name": "Organization & Adoption", "url": "https://catalysphere.co.jp/en/topics/organization-adoption/" }, { "id": "data-quality-knowledge", "name": "Data Quality & Governance", "url": "https://catalysphere.co.jp/en/topics/data-quality-knowledge/" } ] } }