RESEARCH TOPIC / EVALUATION-MEASUREMENT
Metrics, benchmarks, quality assurance, and the limits of proxies.
GROUNDING MAP
Explore 29 research notes through a shared decision domain. Each article identifies its conclusion, sources, findings, and limitations.
LLMs can shift answers toward a user's stance or pushback, and ratings miss it. Investment reviews should not count AI agreement as independent verification, should re-verify AI objections, and should probe how models update at rollout and on change.
Read article
After AI-enabled RPA was introduced, interviews suggested monthly research time fell about 90%. Later, only 10–20% of estimated freed time was explained as new work or extra output. This note shows how to measure the time actually freed and where it went.
Read article
Treat faster local tasks as only one signal; redesign bottlenecks, exceptions, expert handoffs, and the allocation of released capacity across the full workflow.
Read article
Compare the existing and AI-assisted workflows under aligned conditions, including human review and revision time, confirmed quality, and unfinished cases. Use a record to decide deployment, hold, retest, or stop.
Read article
Based on meta-analysis corrections, re-analysis of publication bias, and large-scale administrative nudge trials, we shift the decision framework to prioritize baseline rates, absolute differences, heterogeneity, and costs in advance.
Read article
From experiments showing that evaluations change when a work is labeled as AI-generated, we separate blind content evaluation from eligibility reviews of provenance, rights, and production process.
Read article
Evidence that ranking methods yield inconsistent attitudinal effects despite altering exposure and usage necessitates the creation of a ranked evaluation separating information diversity, reliability, time utilization, and well-being.
Read article
Do not trust multi-LLM discussions based on vote counts; instead, determine the added value by assessing the diversity of initial answers, conformity, problem drift, retention of correct answers, and the difference from simple voting.
Read article
We do not judge success solely by the target correct answer for knowledge editing; instead, we measure paraphrase, locality, general capabilities, and forgetting after continuous editing to establish a boundary for switching to RAG or retraining.
Read article
Do not treat the completion of a deletion API as proof of forgetting; instead, comprehensively verify target memory, retention utility, leakage, and continuous deletion by using a retrained model as the baseline.
Read article
Do not deploy based solely on the overall score of translation benchmarks; determine the scope of provision by analyzing culture-dependent items, native-speaker evaluations, worst-case performance per language, and relief pathways.
Read article
Do not equate the productivity of generated code with safety; instead, treat vulnerabilities, user safety perception, static and dynamic testing, and independent review as gates for adopting high-impact code.
Read article
Rather than prompting guesses solely based on accuracy, design the boundaries between answering, confirming, and stopping by accounting for semantic uncertainty, unanswerable questions, and the losses from incorrect answers versus abstention.
Read article
We examine the inclusion of the public test set in training data from multiple perspectives, including data duplication, completion behavior, and example order, to isolate suspicious scores from decision-making.
Read article
We establish boundaries to return to human evaluation by auditing not only the agreement rate with human evaluators but also through reversal of answer order, redundancy, self-preference, and difficult cases.
Read article
For owners approving long-document AI, this article compares full text, chunking, and retrieval under changing positions, task types, and noise, then sets an automation boundary from worst-case quality and total verification cost.
Read article
We present a correction protocol anchored to primary sources, integrating the potential of personalized correction dialogues, the LLM's own epistemic limits, the persistence of effects after warnings, and the decay of effectiveness.
Read article
This article demonstrates an audit design that distinguishes between the fluency of Chain-of-Thought and internal judgment reproduction, focusing on answer accuracy, causal fidelity, hidden cues, and separated execution logs.
Read article
We design an evidentiary procedure that distinguishes AI text detectors from generation-time watermarks and includes paraphrase resistance, unknown models, group-based false positives, the author's version history, and avenues for appeal.
Read article
Whether AI explanations, confidence displays, judgment order, and user control can truly reconcile the adoption of correct advice with the rejection of incorrect advice is examined by organizing conditions from behavioral experiments.
Read article
Conversation continuity and lasting real-world behavior are distinct. Drawing on meta-analyses and field RCTs of conversational coaches, this article outlines evaluation designs that avoid mistaking short-term responses for behavior change.
Read article
We do not end the discussion with the notion that “metrics become targets and the frontline deteriorates”; instead, we distinguish bias toward easily measurable work, gaming, and actual improvement to design robust operations that resist breakdown.
Read article
By tracking data quality not only in terms of accuracy but also meaning, proxy indicators, and on-site correction pathways, we identify where numbers lead to judgment errors.
Read article
Rankings from a single benchmark cannot determine production quality, stability, cost, or safety; we must redesign evaluation design by integrating multi-faceted evaluation, human-LLM scoring, and prompt sensitivity research.
Read article
Providing evidence via search does not necessarily eliminate errors; this article outlines methods for verifying RAG in business operations by distinguishing between search omissions, inconsistencies with evidence, and oversights in long contexts.
Read article
Average effects of AI coding are not enough for investment decisions. Research from laboratories, enterprises, and mature open-source projects reveals the work, personnel, and measurement conditions that determine whether development speeds up.
Read article
Solving a problem with generative AI is not the same as solving it independently. Drawing on meta-analyses and educational experiments, we examine learning designs that separate supported performance, independent reproduction, and long-term retention.
Read article
We synthesize results from randomized trials and clinical studies involving physicians to clarify the conditions under which generative AI advice improves judgment, and why mere access does not yield effects.
Read article
Do not equate productivity gains demonstrated by research directly with business value. We propose designing ROI by linking time, quality, accuracy, and usage costs at the business process level.
Read article
