Case study
Drafting question banks with AI, and learning where it helps and where it doesn't
A tool that turns a competency matrix and course material into tiered assessment questions in under a minute, ready for expert review.

The challenge
Technical subject matter experts needed question banks for their courses but had no time to write them alongside their other course-building work. Writing ten questions by hand took an expert two to three hours. Then a deadline arrived that needed more than 400 questions in three days, and writing them by hand was not an option.
My role
I was asked whether it was possible to build something to help. I designed and built the tool myself, in under two days.
Key decisions and why
- A dedicated tool, not a chat window. A general AI chat gives different output each time you run the same prompt. A question bank has to be consistent, so I built a tool where the question-writing guidelines and output structure are fixed, and every run follows the same standard whoever operates it.
- Start from the competency framework. The tool takes the competency matrix and the course source material, so every question is tied to a competency learners need, not just to what the text says.
- Three levels of difficulty. Foundation questions test recall of taught content, Applied questions ask learners to use it in real work situations, and Advanced questions ask them to transfer it to situations the material doesn't cover.
- Built for the reviewer. Each question comes with its competency code, tier, four options, the correct answer and a plain-language rationale, so the expert can check it quickly.
- Experts review everything. The tool drafts; the subject expert verifies and corrects before anything is used.
What we built
A web application, built with Abacus AI, used by a learning team without a technical background:
- Upload the competency matrix (an Excel file).
- Upload the course source material, such as technical documents and design guides. Several documents can be loaded at once.
- Choose one or more difficulty tiers and click Generate.
- In under a minute, the questions appear on screen.
- One click exports them to a formatted Word document, ready for the technical team to review and sign off.


The screens above show sample content. The client's competencies and course material have been replaced.
“The subject matter expert's job is no longer to write from scratch. It is to read, verify, and correct where needed. That is a fundamentally different — and far more efficient — use of expert time.”
Results
- Built and in use in under two days.
- More than 400 draft questions produced in days rather than weeks, the equivalent of 80 to 100 hours of expert writing time.
- On expert review, 10% were usable as they stood, 20% needed light editing and 40% gave a useful starting point. The remaining 30% were not usable.
- Expert effort moved from writing questions to checking them.
- MSX International's internal promotion of the tool invited other teams to adapt it for their own projects.
What I learned
The weak point was the wrong answer options. Good distractors depend on knowing which misconceptions learners actually hold, and that knowledge lives in learner data and experts' experience, not in the source material. Building the tool showed me clearly where AI speeds up learning design and where human expertise still does the work.