Order effects in human or LLM judgements

AI evaluation: order effects in a language model’s or a human rater’s yes/no judgements.

Workflow for any judge (people, crowd workers, LLMs): ask two questions in both orders across many items or prompts, count the answer pairs, run the QQ test, and compare models. The counts below are synthetic placeholders; replace them with your own.

import numpy as np
from quantum_mind.core import compare
from quantum_mind.families.order_effects import QuantumOrderModel, QuantumOrderModel4D, AnchoringOrderModel, BayesOrderModel, RANK_STRUCTURES, qq_test, rates

Answer counts

Two yes/no judgements asked in both orders (synthetic counts; replace them with the counts from a human study or from a language model evaluated in both orders).

counts = {'AB': np.array([212, 48, 61, 179]),     # A asked first: yes-yes, yes-no, no-yes, no-no  (synthetic)
          'BA': np.array([240, 33, 52, 175])}     # B asked first  (synthetic)

Test and compare

The QQ test is parameter-free; the BIC comparison shows which model describes the order effect.

q, z, p = qq_test(counts)
print('QQ test: q = %.3f, z = %.2f, p = %.3f (p > 0.05: no evidence against the quantum-like regularity)' % (q, z, p))
for fr in compare([QuantumOrderModel4D, (QuantumOrderModel, {"structures": RANK_STRUCTURES}), AnchoringOrderModel, BayesOrderModel], counts, restarts=12):
    print('  %-22s BIC %.1f  rates %s' % (type(fr.model).__name__, fr.bic, rates(fr.model.predict()).round(3)))

QQ test: q = 0.048, z = 1.92, p = 0.054 (p > 0.05: no evidence against the quantum-like regularity)
  BayesOrderModel        BIC 2378.8  rates [0.552 0.546 0.552 0.546]
  QuantumOrderModel4D    BIC 2382.0  rates [0.522 0.563 0.582 0.526]
  AnchoringOrderModel    BIC 2382.3  rates [0.543 0.553 0.55  0.548]
  QuantumOrderModel      BIC 2465.4  rates [0.42  0.647 0.679 0.375]