Question bank — Item Analysis (psychometric metrics)
Goal: Review psychometric quality metrics for questions that have real candidate response data, and prioritise which questions to keep, monitor, or retire.
Who can do this: Platform administrators and Directory administrators only. The Item Analysis tab is not visible to any other role — MCA users, company admins, staff with browse access, and Staff viewers can open the Question Bank but only see the Question List. They cannot see Item Analysis.
Before you start: Real candidate response data must exist for live assessments (minimum 250 response rows). If the tab is empty, select an assessment from the dropdown and click Run Analysis.
/question-bank → Item Analysis tab
What this tab is for
Item Analysis shows Classical Test Theory (CTT) quality signals for questions that have enough real attempts—similar to what psychometrics teams review in spreadsheets:
| Idea | On screen |
|---|---|
| Difficulty | p-value (proportion who answered correctly) with a 5-band label (excellent, good, fair, poor, negative) |
| Discrimination | D-index (how well the item separates stronger vs weaker performers) |
| Overall recommendation | Recommendation chip per row (Bank It, Monitor, Review, Mis-keyed, Remove) |
Use it to prioritise review before promoting bank items or after a high-stakes deployment—not as the only approval gate (operators still use Review Queue and status lifecycle).
A. Open Item Analysis
- Sidebar → Question Bank.
- Select the Item Analysis tab (fourth tab after Question List, Review Queue, Usage Dashboard).
- The top stat cards (Total Questions, Live Unvalidated, Review Queue) are hidden while this tab is active — they don't apply here.
- Wait for the summary cards and table to load.
B. Read the summary cards
Four counters reflect all analysed items across every assessment — they do not change when you filter the table:
| Card | Meaning |
|---|---|
| Total Analysed | Total question-level analytics rows in the system |
| Bank It | Items the engine recommends keeping as strong candidates |
| Review | Items flagged for human review — combines review and review distractors verdicts |
| Remove | Items recommended for removal — combines remove and mis-keyed verdicts |
These help you size a validation sprint before drilling into rows.
C. Select an assessment and run analysis
The Assessment dropdown lists every assessment that has at least 1,000 candidate response rows (from both Build and Match platforms), identified by their human-readable name where available.
- Select an assessment from the dropdown.
- The table filters to only questions from that assessment.
- An Assessment Reliability Card appears below the dropdown.
- Difficulty and Discrimination distribution tables appear, showing the band breakdown for that assessment.
- Use the Run Analysis button (right of the dropdown) to compute CTT metrics for that assessment.
- If no results exist yet, the button shows Run Analysis (filled style).
- Once results exist, it changes to Re-run Analysis (outlined style) — use this to refresh after new response data arrives.
- The table, stat cards, and reliability card all update automatically when the run completes.
- To see all assessments together, select All assessments from the dropdown (hides the reliability card and distribution tables).
:::info Match platform assessments
Assessments imported from the Match platform appear in the dropdown alongside Build assessments. They are labelled with their assessment name if available, or "Match Assessment #<id>" as a fallback.
:::
D. Assessment Reliability Card
Visible when a single assessment is selected. It shows three sections:
Header
Items count, test-taker count, max/min score, and banked item count, plus when the metrics were last computed.
Reliability table
| Metric | Meaning |
|---|---|
| Cronbach's Alpha | Internal consistency reliability for all item types. Scale: excellent (≥ 0.90) · good (≥ 0.80) · acceptable (≥ 0.70) · questionable (≥ 0.60) · poor (≥ 0.50) · unacceptable (< 0.50). |
| KR-20 | Kuder-Richardson formula 20 — reliability for MCQ-only items, with the MCQ item count shown. Same 6-level scale. |
If either value is ≥ 1.0, a ⚠ Degenerate badge appears instead of an interpretation — this signals a pathological item set (e.g. all items perfectly correlated or sample too small). Do not act on interpretation until response data is larger.
Central tendency
| Metric | Meaning |
|---|---|
| Mean | Average candidate total score |
| Median | Middle score — less sensitive to outliers than mean |
| Variance | Spread of scores |
| Std Dev | Standard deviation — typical distance from the mean |
Verdict distribution
Five chips show how questions in the assessment are distributed: Bank It, Monitor, Review (combines review + review distractors), Mis-keyed, Remove.
E. Difficulty and Discrimination distribution tables
Visible when a single assessment is selected. Each table shows how many items fall into each quality band.
Difficulty (p-value) bands
| Band | p-value range | Action |
|---|---|---|
| Excellent | 0.30 – 0.70 | Retain — optimal discrimination zone |
| Good | 0.20–0.30 or 0.70–0.85 | Retain — slightly off-optimal |
| Fair | 0.15–0.20 or 0.85–0.90 | Review — worth monitoring |
| Poor | 0.05–0.15 or 0.90–0.95 | Remove — too extreme |
| Negative | < 0.05 or > 0.95 | Remove — essentially all wrong or all right |
Discrimination (D-index) bands
| Band | D-index range | Action |
|---|---|---|
| Excellent | ≥ 0.40 | Strong discriminator |
| Good | 0.30 – 0.40 | Good discriminator |
| Fair | 0.20 – 0.30 | Marginal — review |
| Poor | < 0.20 | Weak or negative — remove |
F. Distractor summary (MCQ items, visible page)
When MCQ items are in the current page, a summary shows how many items have fully functional distractors versus items with weak, non-functional, or likely mis-keyed options.
:::note Page scope The distractor summary reflects only the items visible on the current page (up to 25). For a full picture, use the detail modal to inspect individual items, or filter to a verdict chip and page through results. :::
G. Filter by recommendation
Below the reliability card, filter chips narrow the table:
- All — no verdict filter
- Bank It, Monitor, Review, Mis-keyed, Remove
Review covers both the review and review_distractors verdicts — items with distractor issues appear under Review, and the detail modal shows the specific distractor finding.
Selecting a chip reloads the list and resets pagination to page 1. The assessment filter and recommendation filter work together. The summary cards at the top always show global totals regardless of these filters.
H. Work the table
Columns:
| Column | What to look for |
|---|---|
| Question | Question text (truncated to two lines). Falls back to a short bank ID if text isn't available. |
| p-value | Difficulty index + coloured band (green ≈ retain band, orange ≈ review/remove band) |
| D-index | Discrimination index + label (e.g. excellent, good, fair, poor) |
| Recommendation | Automated recommendation chip |
Click any row to open the Item Analysis Detail modal.
I. Detail modal (per question)
The modal shows:
- Header — full question text, assessment name, date computed.
- Difficulty (p-value) and Discrimination (D-index) — numeric value, interpretation band, retain/review/remove decision.
- Group sizes — total responses, upper 27%, lower 27% (standard CTT slicing).
- Distractor analysis (multiple-choice only) — for options A–D: % chosen in upper vs lower groups and a status (functional, weak, likely mis-keyed, non-functional).
- Recommendation — overall verdict label with detail text explaining the driver.
- Edit Question button — opens the edit modal for that question's bank item so you can correct content or metadata without leaving Item Analysis.
The assessment is shown as its human-readable name. If only an ID is available (e.g. for older Match imports before name storage was added), it falls back to a short truncated ID.
Close the modal to return to the table. Use these details when escalating to psychometrics or content authors.
J. Editing a question from Item Analysis
The detail modal includes an Edit Question button. Clicking it closes the modal and opens the standard Edit Question form for that item. Changes to content, status, or classification take effect immediately. Re-run analysis after edits to see updated metrics.
K. Pagination
When more than 25 rows match your filter, use the pager at the bottom. Recommendation summary cards still reflect global counts from the server.
L. Relationship to other Question Bank tabs
| Tab | How Item Analysis differs |
|---|---|
| Question List | Catalogue CRUD, status, filters on metadata |
| Review Queue | Human approve/reject workflow |
| Usage Dashboard | Where unvalidated / flagged bank statuses cluster |
| Item Analysis | Performance psychometrics after real attempts |
An item can be live (validated) in the bank but still show Review here if field data says difficulty or discrimination is weak — treat both signals.
Common questions
| Situation | Guidance |
|---|---|
| Tab loads but table is empty | Select an assessment and click Run Analysis. |
| Assessment not in dropdown | Fewer than 1,000 response rows exist for it — wait for more completions, or check with engineering. |
| Reliability card shows degenerate (≥ 1.0) | Pathological result — typically too few items or near-perfect correlation. Wait for more data before acting. |
| Reliability card shows 0 for all verdict chips | Analysis was run before the new verdict columns were added to the schema. Re-run analysis to repopulate. |
| Very low response counts | Metrics are unstable; wait for more completions before retiring items. |
| "Remove" / "Mis-keyed" verdict | Open detail modal; check distractor table for swapped keys or dead options. |
| Staff viewer | Can open the Question Bank and browse the Question List, but cannot see the Item Analysis tab. |
For catalogue operations (create, import, review queue), see Question Bank — operator runbook.