INSIGHTS / RESEARCH NOTES
We document research on AI, data, behavioral science, and organizations, including methods worth testing in practice. Each note shares not only conclusions, but also hypotheses, observations, and limitations.
69 RESEARCH NOTES
5 / 6 PAGE
Being able to extract text via OCR is distinct from correctly linking layout, graphs, tables, and body text; we organize the unit of business verification based on research in document question answering.
Read article
Designing a system that sends all requests to a high-performance model, or one that fixes them to a low-cost model, is not necessarily optimal; drawing from cascade and router research, we consider routing under quality constraints.
Read article
Even with high accuracy in a single response, errors accumulate in tasks involving continuous planning, search, tool execution, and state updates; we consider the boundaries of reliable automation based on real-world environment benchmarks.
Read article
Rankings from a single benchmark cannot determine production quality, stability, cost, or safety; we must redesign evaluation design by integrating multi-faceted evaluation, human-LLM scoring, and prompt sensitivity research.
Read article
Providing evidence via search does not necessarily eliminate errors; this article outlines methods for verifying RAG in business operations by distinguishing between search omissions, inconsistencies with evidence, and oversights in long contexts.
Read article
Define measurement boundaries for power and water that include usage, processing volume, model, region, time, and equipment, rather than relying on the impression value of a single prompt, and reflect these in procurement and operations.
Read article
We organize vendor evaluations that test data, model dependency, contracts, portability, and stoppage/alternatives before implementation, rather than selecting based solely on accuracy demos.
Read article
This section outlines a method for managing model, data, search infrastructure, and guardrail updates by linking change scope, comparative evidence, approval, monitoring, and rollback.
Read article
We outline an incident design that connects damage assessment, causality, evidence of versions/data/decisions, external reporting, and recurrence prevention, rather than ending with model suspension alone.
Read article
We organize methods for auditing fairness without reducing it to a single score, by examining the usage context, harm, types of errors, intersecting populations, and post-deployment changes.
Read article
We organize methods for separately viewing training, generation, and publication, and for making rights confirmation and provenance recording a single operational audit trail, drawing from primary sources in Japan, the EU, and C2PA.
Read article
We analyze where changes emerge by dividing the impact of generative AI into task time, wages and working hours of existing employees, freelance demand, and youth hiring.
Read article
