Measuring the Human Contribution When AI Is Allowed: Psychometric Foundations for Work-Sample Tests of AI Use
Seng Beng Ho, CH Peh, Roger Simões, Shawn Chia, Elijah Ng, Jennifer Dodgson
Employers want to know how well a person works with AI. Testing that means letting people use a live AI during the test. But the AI can often do the whole task alone, and then the score measures the AI. Our paper sets out how to measure the person instead. It builds on 120 published sources.
Seng Beng Ho, CH Peh, Roger Simões, Shawn Chia, Elijah Ng, Jennifer Dodgson
Books, calculators, computer simulations and scripted role-players each changed how tests were built. Each time, tests were redesigned so the tool could not do the work by itself. AI needs the same, and more.
Can do aloneSupply facts
Tests had toAsk people to apply facts, not look them up
Can do aloneCompute
Tests had toBe written so the calculator helps but does not answer
Can do aloneShow a realistic setting
Tests had toMark what people do in it, step by step
Can do aloneBehave the same for everyone
Tests had toScript the partner, so every candidate meets the same one
Can do aloneOften, the whole task
Tests have toProve, before anyone sits the test, that the AI alone cannot pass it. Every candidate gets the same version of the AI.
A test of AI use needs a live AI during the test, real work, and proof before use that the AI alone cannot pass. The paper reviews the earlier approaches. Most do not test what people can do at all. A 2024 review found 16 AI literacy scales, and only 3 asked people to carry out tasks.
| Approach | Live AI during the test | Real work | Checked before use that the AI alone cannot pass |
|---|---|---|---|
| Skill frameworksDescribe what people should be able to do | No | Does not apply | Does not apply |
| Self-rating scalesPeople rate their own skill | No | No | Does not apply |
| Knowledge testsQuestions about what AI is | No | No | Does not apply |
| Research studiesPeople working with AI, for a study | Yes | Sometimes | No |
| Checks of existing examsCan AI pass them? | AI as a threat | Sometimes | Yes, to keep AI out |
| AI benchmarksHard questions for AI models | The AI is tested | No | Yes, to measure the AI |
| The SIAA Standard | Yes | Yes | Yes, to measure the person |
From the paper's review of earlier approaches (its Table 2).
This is the third thing, and the paper's main rule. We give the question to the test's own AI with no person involved, and see whether its answer passes.
3,906
In October and November 2026, we gave the questions to the test's own AI 3,906 times. We pasted them whole, in pieces or reworded, and sent its answers straight back.
What it meansA question the AI can pass alone is rewritten until it cannot.
4 → 8
In October, the AI left a mistake in 4 of 10 answers to one question. In November, with the same question and the same AI, it left it in 162 of 200, about 8 in 10.
What it meansThe paper requires every question to be checked again after any change of wording, and whenever the AI model changes.
Test writers often make a task hard by hiding facts, adding a calculation or planting errors. We gave tasks like these to the AI with no person helping. It solved every one. So a question built only on these would let someone pass by copying the AI.
| What made the task hard | What the AI did alone | How often |
|---|---|---|
| Facts in a side panel or behind a button | Used them, once they were pasted in | At least 98% of the time (200 runs on each of three tasks) |
| A rule written in the materials | Kept to it | At least 98% of the time (200 runs) |
| A calculation in several steps | Got it right | At least 98% of the time (200 runs) |
| Two figures in the materials that disagree | Noticed the conflict | At least 98% of the time (200 runs) |
| Mistakes in a draft, with the source given | Corrected all of them | At least 98% of the time (200 runs on each of two versions) |
| Pressure or disagreement in the materials | Took the careful course | Every time, on two tasks (October runs) |
| Hints pointing at what to notice | Raised the same points without the hints | At least 98% of the time (200 runs with the hints, 200 without) |
| An AI agent given weak instructions | Acted safely anyway: no refunds, no wrong cancellations, no fake instruction followed | At least 99% of the time (600 runs, three sets of instructions) |
| An instruction to the AI to state something false | Refused | Complied 0 times in 5 (October runs) |
“At least” figures are the lowest rate the runs support, at 95% confidence, with the test's own AI in November 2026.
With questions built this way, we graded answers that were simply copied from the AI.
1 in 155
In grading tests in November 2026, 155 attempts sent back the AI's own answer unchanged. The one that passed was on a question since rewritten. 85 reached Basic: correct and safe, but copied. 69 fell below Basic.
What it meansA pass shows the person's own work with the AI.
The paper says plainly which claims have evidence and which still need it. So far the evidence comes from AI playing the candidates.
These are commitments we make for every test, so buyers can check us against them.
Contact
Tell us about your organisation and what you want to measure. We will reply to discuss a pilot.