Does the label "AI-generated work" evaluate the work itself?

Porter and Machery conducted experiments where non-experts attempted to distinguish AI-generated poems from those by renowned human poets, reporting that participants were more likely to judge AI poems as human-made and rated AI poems higher across multiple quality metrics. [1]

This study focused on specific English poems, a generation model, and non-experts, and does not represent quality across all genres. [1] Nevertheless, it demonstrates that intuitions about "AI-ness" do not align with actual generation sources.

The task of identifying the generation source is distinct from the task of evaluating the work as good. One cannot conclude that works are of equal value simply because they cannot be identified, nor that they are of low quality simply because they can be identified.

In selection processes, we separate quality, originality, technique, emotion, eligibility, and provenance into distinct categories. Mixing them into a single overall impression makes it unclear which criteria were influenced by the label.

We do not directly use AI detector outputs for work review. Detector misclassifications combined with human provenance bias may lead to the unjust exclusion of human authors.

• Content Quality: Evaluate based on criteria intrinsic to the work itself
• Source Identification: Measure identifiability as a separate task
• Eligibility: Verify the scope of AI use according to regulations
• Provenance: Review evidence of the production process and rights

Horton et al. included experiments randomly assigning AI or human production labels to the same images, showing a tendency for works labeled as AI to be rated lower in skill and value. [2] By fixing the content, they isolated the label effect.

Comparing actual AI and human works only conflates content differences with label differences. Even in internal reviews, we use simulated works permissible under rights to conduct audits where only the provenance display for identical content is varied.

Even if a label effect is found, we do not simply discard all evaluations as bias, as evaluators may legitimately value human effort, scarcity, intent, and rights. We ask what is being valued.

The problem is the unconscious conflation of quality review and production ethics judgments. We assign content scores first, then separately review the production process, rights, and regulatory compliance.

Audits that hide labels do not propose hiding provenance in the actual event. They are used as research tools to measure bias in the evaluation structure, with necessary disclosures made in the final publication.

Horton et al. also reported conditions under which the creativity evaluation of works labeled as human-made increased after comparison with AI-labeled works. [2] It is not just the penalty for AI works, but the contrast that alters the value of human works.

Evaluations are not absolute and vary based on previously seen works and categories. We randomize the selection order to audit effects caused by the sequence of AI, human, and co-created works.

When treating human effort as a value, we explicitly define criteria for evaluating labor, skill, time, and narrative separate from work quality. We do not hide implicit premiums in the total score.

Co-creation may not be easily represented by simple ratios of human and AI contributions. We record contributions by stage: ideation, selection, editing, generation, finishing, and responsibility.

In calls for entries intended to protect human authors, there may be justification for restricting AI use. In such cases, we design for detection uncertainty, declarations, objections, and evidence, ensuring no conflation with content evaluation.

• Order: Randomize the sequence of work presentation
• Comparison Frame: Explicitly define criteria for human, AI, and co-created works
• Process: Record ideation, generation, selection, editing, and finishing
• Purpose: Declare quality, author protection, and rights separately

Köbis and Mossink compared conditions where AI-generated poems were chosen randomly versus selected by humans, showing that human selection makes it harder for participants to identify the generation source. [3] AI output is not independent of the selection process.

Classifying only the final work as solely AI or solely human loses the contributions of curation, iteration, and editing. We disclose the production process at the necessary granularity.

Comparing an AI work selected by a human from many candidates with a human's initial proposal involves different selection conditions. We either equalize the number of candidates, selection, and editing time, or report the difference.

We avoid denying authorship merely due to AI assistance, just as we avoid labeling everything as human-made for slight editing. We define the contribution information necessary for the evaluation purpose.

Even if the degree of production assistance is continuous, call-for-entry regulations require operable categories. We specify permissible examples, prohibited examples, cases for judgment retention, and objection procedures.

In the first stage, we hide author names and generation sources to the extent possible, evaluating work content against common criteria. In the second stage, we review provenance, rights, regulations, and production process.

There may be features that cannot be blinded. Since expressions perceived as specific to AI may themselves be evaluation targets, we do not guarantee complete concealment but record evaluators' estimates and confidence.

We do not rewrite second-stage ineligibility as low quality scores from the first stage. We accurately preserve the judgment that a work is high quality but non-compliant with regulations.

We train judges to separate content criteria from eligibility criteria, confirming agreement on specific borderline cases rather than general attitudes toward AI.

The final results show the evaluation axes and reasons to the extent publicly permissible. If excluded due to AI use, we clearly state the regulatory clause, evidence, and objection method.

The mere presence or absence of AI use covers a wide range, from spell-checking to full generation. We disclose the roles of processes and tools relevant to the evaluation purpose.

Overly detailed provenance disclosure may impose excessive burdens on authors and disadvantage small-scale creators. We consider necessity, verifiability, and cost.

We trust declarations while establishing random audits and evidence submission. We avoid designs that suspect everyone or punish based solely on weak detectors.

If false declarations are discovered, we categorize by intent, impact, and possibility of correction. We do not apply the same sanctions for minor omissions as for rights infringement or regulatory evasion.

Public labels help users make choices but also alter work evaluation. Instead of hiding this effect, we specify the meaning of the label and conduct misunderstanding investigations.

• Tool Role: Distinguish between assistance, generation, selection, and editing
• Evidence: Request production logs or drafts proportionally
• Audit: Confirm by risk level rather than indiscriminate detection
• Objection: Provide pathways for misjudgment and declaration correction

Non-expert poem evaluations and professional judge evaluations are not necessarily the same. [1] [3] We verify with our own judges whether professional knowledge reduces or increases label effects.

Work purchasers, readers, clients, and authors hold different values. We do not reduce evaluation criteria to a single audience survey.

We record evaluators' AI experience, profession, and attitudes, but do not exclude them based on specific attitudes. We understand result heterogeneity and judge the need for criterion training.

Repeated review may change effects as evaluators become accustomed to AI works. We separate initial and ongoing reviews, controlling evaluation order and fatigue.

We identify axes with large label effects. The necessary criterion explanations vary depending on whether skill, value, emotion, or originality is influenced.

Since models, creative practices, and social norms change, we do not permanently use past label experiments. We update work types and judges to re-audit.

We compare acceptance/rejection rates by human-made, AI-made, co-created, and undeclared categories, examining where differences arise in content scores and eligibility.

We also monitor the problem of specific groups of authors being mistakenly identified as AI-generated. We do not use style, language, or non-native speaker expressions as proxies for AI-ness.

We revert cases overturned by objection to review examples, correcting where the cause lay in the detector, evaluator, or regulations.

Creative works' provenance has legitimate value. Simultaneously, without separating content and provenance evaluation, one cannot explain what is praised and what is regulated. The two-stage review protects both.

The audit record identifies the selection version, label presentation, and reviewer group and separately retains blind and disclosed rankings, group pass rates, and reasons for changes. An unexplained difference outside a predeclared tolerance pauses the internal selection decision for a review of criteria and evidence. Preserve content scores and eligibility judgments separately and maintain required public provenance disclosure. Differences alone do not identify a causal label effect. Resume the decision after the selection owner and auditor jointly review a re-audit by different reviewers.