The SIAA Standard — byStraits Institute for Applied AI
Talk to us

AI is the first exam-room tool that can do the whole task alone.

Books, calculators, computer simulations and scripted role-players each changed how tests were built. Each time, tests were redesigned so the tool could not do the work by itself. AI needs the same, and more.

  1. Book

    Can do aloneSupply facts

    Tests had toAsk people to apply facts, not look them up

  2. Calculator

    Can do aloneCompute

    Tests had toBe written so the calculator helps but does not answer

  3. Computer simulation

    Can do aloneShow a realistic setting

    Tests had toMark what people do in it, step by step

  4. Scripted role-player

    Can do aloneBehave the same for everyone

    Tests had toScript the partner, so every candidate meets the same one

  5. AI assistant

    Can do aloneOften, the whole task

    Tests have toProve, before anyone sits the test, that the AI alone cannot pass it. Every candidate gets the same version of the AI.

None of the earlier approaches has all three things a test of AI use needs.

A test of AI use needs a live AI during the test, real work, and proof before use that the AI alone cannot pass. The paper reviews the earlier approaches. Most do not test what people can do at all. A 2024 review found 16 AI literacy scales, and only 3 asked people to carry out tasks.

ApproachLive AI during the testReal workChecked before use that the AI alone cannot pass
Skill frameworksDescribe what people should be able to doNoDoes not applyDoes not apply
Self-rating scalesPeople rate their own skillNoNoDoes not apply
Knowledge testsQuestions about what AI isNoNoDoes not apply
Research studiesPeople working with AI, for a studyYesSometimesNo
Checks of existing examsCan AI pass them?AI as a threatSometimesYes, to keep AI out
AI benchmarksHard questions for AI modelsThe AI is testedNoYes, to measure the AI
The SIAA StandardYesYesYes, to measure the person

From the paper's review of earlier approaches (its Table 2).

So we try every question against the AI before anyone sits it.

This is the third thing, and the paper's main rule. We give the question to the test's own AI with no person involved, and see whether its answer passes.

3,906

Times the AI tried our first questions on its own.

In October and November 2026, we gave the questions to the test's own AI 3,906 times. We pasted them whole, in pieces or reworded, and sent its answers straight back.

What it meansA question the AI can pass alone is rewritten until it cannot.

4 → 8

The same question can get a different answer from the AI two weeks later.

In October, the AI left a mistake in 4 of 10 answers to one question. In November, with the same question and the same AI, it left it in 162 of 200, about 8 in 10.

What it meansThe paper requires every question to be checked again after any change of wording, and whenever the AI model changes.

The AI alone solved every usual kind of hard task.

Test writers often make a task hard by hiding facts, adding a calculation or planting errors. We gave tasks like these to the AI with no person helping. It solved every one. So a question built only on these would let someone pass by copying the AI.

What made the task hardWhat the AI did aloneHow often
Facts in a side panel or behind a buttonUsed them, once they were pasted inAt least 98% of the time (200 runs on each of three tasks)
A rule written in the materialsKept to itAt least 98% of the time (200 runs)
A calculation in several stepsGot it rightAt least 98% of the time (200 runs)
Two figures in the materials that disagreeNoticed the conflictAt least 98% of the time (200 runs)
Mistakes in a draft, with the source givenCorrected all of themAt least 98% of the time (200 runs on each of two versions)
Pressure or disagreement in the materialsTook the careful courseEvery time, on two tasks (October runs)
Hints pointing at what to noticeRaised the same points without the hintsAt least 98% of the time (200 runs with the hints, 200 without)
An AI agent given weak instructionsActed safely anyway: no refunds, no wrong cancellations, no fake instruction followedAt least 99% of the time (600 runs, three sets of instructions)
An instruction to the AI to state something falseRefusedComplied 0 times in 5 (October runs)

“At least” figures are the lowest rate the runs support, at 95% confidence, with the test's own AI in November 2026.

So our questions test what only the person can do:

  • Taking the action the task needs
  • Noticing new information that arrives during the task
  • Knowing whose decision it is
  • Handing over work the next person can use as it is
  • Explaining their own work, without the AI

The result: copying the AI almost never passes.

With questions built this way, we graded answers that were simply copied from the AI.

1 in 155

One copied answer in 155 reached a pass.

In grading tests in November 2026, 155 attempts sent back the AI's own answer unchanged. The one that passed was on a question since rewritten. 85 reached Basic: correct and safe, but copied. 69 fell below Basic.

  • Pass: 1
  • Basic: 85
  • Below Basic: 69

What it meansA pass shows the person's own work with the AI.

What is proven so far, and the studies that come next.

The paper says plainly which claims have evidence and which still need it. So far the evidence comes from AI playing the candidates.

Shown so far

  • Copying the AI reached a pass once in 155 attempts.
  • When an AI played a careful candidate, its work passed five of the six questions.
  • With two shorter questions per area, one careless mistake no longer fails the whole test. Leaking private information still fails its area.

Next studies

  1. Real candidates told to copy, to edit lightly, or to work carefully.
  2. Checks that results hold across different questions and different AI answers.
  3. Trained human markers compared with the automated marker.
  4. Every check done again with a newer AI model.
  5. A survey of how people really use AI at work.
  6. Managers judging the same work, labelled as written by an employee or drafted by AI.

Ask us about the research.

Talk to us