AI Fluency Rubric: Instrument and Evidence

The author set every level assignment in section 3. None of the 15 items has been independently keyed. See section 5.

3 items per dimension, 15 in total. This is a fixed design constraint, not an outcome of how much content happened to be written: every dimension carries exactly the same number of items, so each contributes equally to the unweighted composite. Equal item count does not establish equal measurement precision.

Why three, and not one or two

Resolution. A dimension score is the mean of its items. With one item per dimension a score can only take 4 values — 1, 2, 3, 4 — so every person sits exactly on a band and a 3.0 cannot be told from a 3.3. With 3 items it takes 10 distinct values, which is the resolution the product's own copy depends on when it distinguishes a dimension that is nearly Applied from one that has just arrived there.

A majority reading. Three is the smallest odd number greater than one, so two answers agreeing constitutes a majority. Two items can only agree or split, and a split leaves the dimension on a midpoint with nothing to break it.

The boundary is never landed on exactly. With 3 items a dimension score is a multiple of one third, and the composite is a multiple of one fifteenth. A band boundary sits at 1.5, 2.5 and 3.5, and none of them is reachable by either a dimension score or a composite. The rounding rule in section 4 therefore never has to resolve a tie. This is a property of the arithmetic, not a claim about behaviour.

What three costs. Fifteen items rather than five is roughly three times the demand on the respondent. That is the trade being made, and it is the reason the count is not higher.

Withheld from this copy. The level assigned to each option of each item is not printed here. Publishing it openly would make the instrument gameable by anyone who read the page — a score chosen rather than measured — and the whole claim of the product is that the number came from a measurement.

Available to reviewers on request. If you are evaluating the method — a buyer, an advisor, a practitioner asked to key the items, or anyone who needs to check the mapping before trusting a result — write to cody@aifluency.ai and the full key will be sent to you. It is also visible in this document when signed in to an organization account with a leadership role.

What is not withheld: how many items there are and how they are distributed (section 2), how an answer becomes a score and a band (section 4), who has checked the mapping and when (section 5), what the score may be used for (section 6), and every psychometric property that has not been established (section 8). The key is the one part whose publication would change the measurement, and it is the one part that is held back.

Two properties of the key are worth stating without printing it. Every item offers exactly one option at each of the four levels, so no item is weighted differently from another. And the presentation order is deliberately scrambled with respect to level: authoring an item best-answer-first and never shuffling is the most common way a multiple-choice instrument becomes guessable, and it happened here once already. An automated check runs on every build and fails it if any content-free strategy — always pick the first option, always pick the longest — scores above 3.0 on either form.

A dimension score is the arithmetic mean of the levels of the 3 answers given for that dimension. The composite index is the unweighted mean of the five dimension scores, equivalently the mean of all 15 answers. Neither is weighted and neither is transformed.

A band is the word attached to a score. It is assigned by rounding the score to the nearest whole level:

The 2.5 boundary

A score of exactly 2.5 rounds up, to level 3, Applied. The same applies at 1.5 and 3.5. This is stated because a boundary rule that is left implicit is a rule nobody can check, and because it decides which side of a band a person falls on.

For an individual, the rule is never invoked. No reachable dimension score or composite index is exactly 1.5, 2.5 or 3.5: a dimension score is a multiple of one third and a composite is a multiple of one fifteenth, and no half-integer is either.

This does not extend to aggregates. A department or organization figure is a mean across people, not a mean across items, so it can land anywhere — including exactly on a boundary. Where an aggregate is shown, the band word is derived from the figure as printed rather than from the underlying value, so the number on the screen and the word beside it always agree. Two departments showing the same figure will always carry the same word.

The Unaware floor rule

If any single dimension rounds to level 1, Unaware, the band printed beside the composite is held at level 2, Exploring, however high the composite mean is. The numeric score is never altered. Only the word is.

The reason is that a mean hides its worst component. Someone at level 4 on four dimensions and level 1 on Verification averages 3.4 and would otherwise be labelled Applied — reading as broadly competent while being unable to tell whether AI output is true. The rule is deliberately narrow: it triggers only at Unaware, and it holds at Exploring rather than dropping to the floor, because the strength elsewhere is real and the number still shows it.

Where the rule fires, the result page and the report both state that it has fired and name the dimension responsible. A held band is never shown without its reason.

Suppression rules

Three further rules withhold a figure rather than print a weak one. No percentile is shown at all: there is no defined reference population (the norm base in the table of what is not yet established), and a percentile against whoever happened to take the assessment would describe early adopters, not a population. Leadership sees no individual’s result at all, and a group — a department, or any future cut by team, role or location — shows its results only once 5 of its people have been measured, on screen and in the evidence pack alike. A larger group is also withheld when showing it would let a smaller one be worked out from the totals. Two measurement rounds are compared only when each has at least 5 responses on the same version of the assessment; a round holding more than one version is marked and never compared. A department shows round-over-round movement only with at least 8 people measured in both rounds, and an organization figure needs 20. No movement smaller than 0.2 is drawn at all.

The floor for reporting a mean is deliberately lower than the floor for reporting movement, because they are different claims. The movement floor is sized for the error on a difference of two means, which is roughly √2 times the error on either one; using it for a single mean withholds a figure that is better estimated than the movement the same rule would draw. The evidence pack made exactly that mistake and withheld three departments the dashboard reported confidently off the same rows.

Neither floor is derived. The standard deviation of this instrument is not established — see section 8 — so nobody can compute the point at which a mean becomes stable. They are conventions, set where a claim stops being obviously unsupportable, and they should be revisited the moment reliability exists.

0 of 15 items have been reviewed independently. Every level assignment in section 3 is the author’s own, and no item has been keyed by anyone else.

An earlier version of this document reported five items as reviewed, on the strength of a note in the source describing the five original questions that way. That note carried no reviewer, no date and no record of a review taking place, and it was incorrect. It has been removed at the source rather than annotated here.

Items withdrawn

26 items were withdrawn across four dates — four on 2026-08-19, two on 2026-08-25, 15 on 2026-09-22 and five on 2026-09-26 — after an outside reading of the bank. They are not administered and are not counted in the 15 above. They are listed because a withdrawal is a break in comparability rather than a tidy-up: a mean for Awareness, Instruction, Verification, Governance or Integration from before those dates is an average over different items than one from after, and any movement claim spanning either date has to say so.

“Answers held” counts stored answers to that item from counted results only — the same population section 8 reports. Answers given by demonstration and pilot rows are excluded here for the same reason they are excluded there. Withdrawing an item does not delete what was already answered; the rows stay attached to the retired id and still resolve.

What this means for a result. The scores this instrument produces rest entirely on level assignments that one person made and nobody has checked. Until that changes, no individual result should be used to make a claim about a person, and no figure here — individual or aggregated — should be described as a validated measure of AI fluency.

What a score reports is how someone answered these fifteen scenarios, against a scoring key that is withheld from the public (section 3 says who can see it). Aggregating those answers across a department is arithmetic on the same evidence: it inherits every limitation above and adds none of its own. The limitation is in what the number means, not in whether it may be summed. Section 8 sets out what an independent keying would take.

What is stored

Retention review

There is no retention period. What the account owner sets is a retention REVIEW period — 12, 24 or 36 months, or unset. The distinction is not pedantic. A retention period is a promise that data is deleted when it expires, and nothing in this system deletes anything. Setting 24 months retains nothing for 24 months: it flags what is older than that and waits.

Records past the period are flagged for deletion and nothing is deleted without approval. There is no scheduled job. The dashboard counts the assessments older than the period and offers the account owner one deliberate, confirmed action that deletes those assessments and every answer in them permanently. Accounts are not deleted — this removes measurements, not people.

Unset is the honest default and is shown as unset rather than silently treated as a number nobody chose. Every organization created before this existed has no period until someone picks one.

An organization with a real retention obligation is not covered by this. Enforcement — automatic deletion or anonymisation at expiry, with an audit log of what was removed and when — does not exist, and a review trigger is not a substitute for it. Do not represent this control as meeting a statutory or contractual retention requirement.

Contacts from the public path are outside this. A public result has no organization and no retention owner. The name and email are asked for on the entry screen, before the first question, and are kept until the person asks for them to be deleted.

Consent

A workforce respondent is told what leadership can see, and the acknowledgment is recorded before their first assessment. The notice appears twice: on the join screen, before they enter an organization code, and again as a gate that replaces the assessment itself the first time they open it. It names what is visible — their name, department and whether they have completed it, and their results only inside group figures for groups of 5 or more — and what is not, which is their own scores and their answer to any individual question. The gate cannot be scrolled past; the timestamp is written only on an explicit, unticked acknowledgment.

The wording of all three surfaces comes from one place, and each acknowledgment stores the version of the notice that was on screen when it was given. So what a person read and what the record says they acknowledged cannot drift apart, and a change to the wording re-asks everyone holding the older version rather than inheriting their agreement.

Members who predate this have a null timestamp and see the gate on their next assessment. Nothing was backfilled: a consent that was never given is not evidenced by writing a date into a column.

The public path does not show this gate. A public taker has no organization and no leader, so a notice about what leadership can see does not apply to them. Their name and email are held on the result and its contact record, and anyone with their result link sees their name and overall score.

What this still does not do. It does not obtain consent on an employer’s behalf for any purpose beyond this measurement, does not establish a lawful basis under any particular regime, and does not delete anything on a schedule. An organization with its own retention or consent obligation still has to satisfy it; what changed is that this system now records what it was told and when, rather than leaving the whole question to the buyer.

What the evidence pack appendix is

The appendix lists each registered person by name, with their department and whether they have completed the assessment. It carries no one’s results. It is a record of participation: who took part and who has not yet. Every figure elsewhere in the pack is over a group of 5 or more. That is what makes it useful to someone demonstrating that a workforce activity took place — under the EU AI Act, the AI literacy obligation of Article 4.

Coverage is against registered people, and says so in the figure. The pack headlines coverage as “65 of 66 registered” rather than as a bare percentage, because the denominator is the whole meaning of the number and it used to sit in the small print while “98.5%” sat in the table. A share of a self-selected registered subset, headlined without its denominator in a document filed as evidence, is the claim that comes apart the first time somebody checks it against a headcount.

Coverage of the workforce is not established. No organization here records a total headcount that could serve as a denominator — department headcount labels are free text — so the pack claims no share of the workforce at all. Registration is the only population it can count, and a department that registered eight of thirty-seven people will read as fully covered on a basis that says so.

What it is not: it is not a performance record, not an evaluation and not a ranking. It carries the same scope-of-use limits as section 6, and the fact that it is formatted for a compliance file does not extend what its numbers may be used for.

The participation list is personal data: names, departments and whether each person took part. Handle the pack as such — it is generated on request rather than stored, and it should not be circulated more widely than the obligation it answers.

Everything in this section is absent. None of it is estimated, modelled or inferred below, because each has a conventional estimator that would happily return a number from the data on hand and that number would not mean anything.

The evidence that does exist

Counted at the time this page was generated. Only results from people who chose to take the public assessment are included; demonstration and pilot data are excluded by construction.

The third row is the one that governs everything above it. Earlier results were recorded against a five-item bank or an earlier version of this one, and are not evidence about it. Every property in this section stays absent until this number supports the samples described above.

Result-page feedback discloses the keyed answer for at least one dimension. Re-measurement using the same items cannot distinguish improved judgment from recall of a previously disclosed answer. Alternate forms are in development.