Rumbo · methodology
How the measurement works
A pilot report makes claims about your team. This page explains where each one comes from and how far it can be pushed before it breaks.
Three things, measured three different ways
The single most common mistake in this category is collapsing capability, adoption and productivity into one number. They are different claims with different evidence behind them, and mixing them is how a report loses a CFO in one question. Everything we produce carries one of three labels.
What the platform observed
- Example:
- Assessment scores, task completion, time from opening a task to submitting it, which skills moved.
- Strength:
- Strongest. No assumption, no extrapolation, no self-report.
- Limit:
- It is about our tasks, not your month-end.
What a learner told us about their own work
- Example:
- “Variance commentary used to take me 60 minutes; it took 25 this time.”
- Strength:
- It is the only way to see outside the platform at all.
- Limit:
- Recall, not measurement. Always reported with the sample size.
A manager confirmed a learner's claim was directionally reasonable
- Example:
- The same claim, after the learner's manager reviewed it.
- Strength:
- The strongest workplace evidence we produce.
- Limit:
- Still an estimate, and the confirmer is not disinterested — the manager sponsoring the pilot is the person confirming it worked. Corroboration, not audit.
What the AI Score measures
On a path with professional missions, such as Finance, the AI Score is a weighted estimate across seven skills, built from how a person performed on realistic tasks with an AI assistant available. Each task is graded on up to seven dimensions — the result, professional accuracy, how AI was used, verification, efficiency, communication and judgement — against a rubric written when the task was written. On the department paths, and before a person’s first graded mission, the score shown reads their practice instead, on the same scale from 0 to 100.
Harder tasks count for more. Recent evidence counts for more than old evidence. A skill stays close to its starting point until enough evidence exists to move it, so one lucky mission cannot swing a score.
What the AI Score does not measure
- Whether someone is good at their job.
- Accounting knowledge in general. The tasks assume competence and measure what happens when AI enters the workflow.
- Anything about work done outside the platform.
- Anything comparable to another company’s score. There is no norming population.
Baseline and final
The baseline is three tasks, about fifteen minutes, taken before any training. There are no hints and the assistant will do work but not coach. It runs on a company that appears nowhere in the learning path.
The final runs on the same company as the baseline, deliberately — a different period, different documents, different figures, different causes and different wording, but Aldabra Foods both times. Holding the company constant removes an obvious confound: if the final ran on a new company, a rise in score could be a learner finding that company easier to read rather than the learner getting better.
What the build check enforces is that both sets cover the same competencies at the same difficulties for the same total length. It checks structure, not psychometric difficulty equivalence — those are different claims and we only make the first.
Taking an assessment does not move the score it measures. Assessment attempts award no XP, write no skill scores, build no streak and produce no score history. That is enforced in the database rather than in application code, and a test removes the guard and confirms the suite fails. Without it, the act of measuring would inflate the thing being measured and every improvement figure would be wrong.
The final assessment’s content is withheld — from the API, not merely from the screen — until an administrator opens it. A measurement a learner can revise for measures revision.
Transfer tasks
In week five a third set runs on a third company. These are deliberately unlike training: the request does not say what it wants, the pack has a genuine gap in it, an AI first pass is fluent and partly wrong, and at least one question cannot be answered from the files at all.
Saying plainly that something cannot be concluded scores well. Producing a confident answer anyway does not, however good the prose. That distinction is most of what separates someone who passed a course from someone you can leave alone with a board pack.
Retries, practice and evidence
Path missions can be retried as often as a learner likes — that is learning. A retry carries a third of the evidence weight of a first attempt, so repetition improves skill without manufacturing a score.
Assessments are measurements, not practice. A second attempt at an assessment task is recorded as an integrity flag and surfaced on the report in aggregate. We also flag submissions written implausibly fast, text identical to an earlier submission, and answers that are long verbatim spans of the assistant’s own output. These qualify a score; they never punish a person, and no individual is named to a manager.
Confidence
Nothing is reported without a confidence level, and confidence is deliberately hard to raise.
One thing this measurement does not do: separate learning from practice. There is no control group. A learner who improves between baseline and final has had six weeks of training and also six weeks of doing similar tasks, and this design cannot tell you how much of the movement is which. If that distinction matters to you, hold a third of the cohort back, baseline everyone at the same time and train them a month later — it costs nothing and it is a much stronger result.
- Capability is preliminary below about five completed missions per learner, and below three paired learners nothing is reported at all.
- Productivity is LOW on one or two learners’ estimates; MEDIUM on repeated evidence across several learners or some manager review; HIGH only with repeated evidence and manager verification of a meaningful share. Volume of self-reports alone can never reach HIGH.
Money
No euro figure exists anywhere in the product unless you supply your own fully-loaded hourly rate. When you do, the arithmetic is shown in full and the result is labelled estimated potential annual value — not ROI generated, not savings delivered. It assumes the time freed is redeployed, which is your assumption to make and not ours.
What the AI Score is not
The AI Score is a proprietary performance measure that exists inside this platform. It is not a regulated professional qualification, it is not accredited by any professional body, and it is not a standardised psychometric instrument — it has not been validated against job performance, and we will not describe it as predictive until it has been.
It measures what a person did on realistic tasks, under consistent conditions, before and after. That is a real thing to know and a narrow one. Certificates issued by the platform say the same on their face.