A quality score can drift because the work changed or because the reviewer grew tired. The two causes look alike in a monthly report. An audit of the review sample separates them before the score is used to coach an agent. The audit does not assume that every low score is fatigue, and it does not excuse a real defect.
Map the review session
Record when each review happened and where it sat in the reviewer's session. A score from the first case of the morning is not directly comparable with a score from the twentieth case after lunch. The session map lets you compare like with like. It also shows whether reviews are clustered at the end of a long block.
Count the minutes of continuous review before each case. Include the time spent reading the transcript, listening to the call, and writing the feedback. A reviewer who has been scoring for three hours may be slower and harsher than one who started twenty minutes ago. The count should come from the review tool or a simple time log.
Note the type of case in each position. Complex cases may be saved for the end of a session, which confuses case difficulty with reviewer fatigue. Randomize the case order or stratify the sample before comparing positions. A fair audit uses the same mix at the start and the end.
Keep the session map for the next audit. A single month cannot show a pattern, but three months can show whether fatigue appears at the same point. The quality reviewer capacity plan can connect the pattern to staffing. The next audit can then start with the same map instead of rebuilding it.
Build a repeat sample
Select a small set of cases that were already scored. Re-score them in a fresh session without showing the original score. Keep the same rubric and the same evidence, but change the order. The difference between the two scores is the review error for that case.
Choose cases across the score range, not only the low scores. A reviewer may be consistent on clear failures and inconsistent on borderline work. Include cases with missing evidence, ambiguous policy, and a clean interaction. The mix shows where fatigue changes judgment.
Use a second reviewer for part of the repeat sample. The second reviewer does not need to be a manager, but they should be calibrated on the rubric. Compare the two reviewers on the same case before comparing each reviewer with themselves. That separates personal drift from rubric ambiguity.
Record the time and position of the repeat score as well. If the second score is lower at the end of a long session, the session length is a likely cause. If the second score is lower at the start, the rubric or the evidence may be the problem. The calibration routine can provide a shared reference.
Separate severity from error
A harsher score can mean the reviewer found a real defect that the first pass missed. It can also mean the reviewer applied the rubric more strictly without new evidence. Read the comments for each changed score and classify the change. Use categories such as new evidence, stricter interpretation, arithmetic error, or missed defect.
Do not treat every change as a quality failure by the reviewer. A reviewer who catches a missed defect is doing the job. The audit should ask whether the first pass had enough time and evidence to catch it. That question leads to a process fix rather than a blame conversation.
Check whether the rubric allows the same case to be scored in two ways. If it does, the audit has found a rubric problem, not a fatigue problem. Rewrite the rule with an example before the next sample. A rubric that depends on the reviewer's mood will produce drift even in a short session.
Share the classification with the reviewers. They need to know which changes were counted as errors and which were counted as improvements. A transparent classification keeps the audit from feeling like a secret test. The case ownership model can clarify who owns the final score.
Cap continuous review time
Set a maximum continuous review block, such as forty-five or sixty minutes. The exact number should fit the work and the reviewer's other duties. Put the cap in the schedule so it is not left to individual discipline. A cap that exists only in a policy document will be the first thing dropped in a busy week.
Build a short reset between blocks. The reset can be a walk, a queue check, or a different task that does not require the same judgment. It should be long enough to change the reviewer's attention, not just to refill a coffee. Record the reset in the time log so the audit can see whether it happened.
Rotate reviewers across case types and teams. A reviewer who scores the same queue all day may develop a narrow view of what good looks like. Rotation also spreads the emotional load of difficult cases. The rotation should be planned, not improvised when someone is absent.
Review the cap after a month. If scores still drift at the same point, the cap may be too long or the sample may be too large. If scores are stable, keep the cap and document why it works. The agent workload guide can help compare review time with other work.
Report the finding without overclaiming
State the size of the repeat sample and the number of changed scores. A finding based on six cases is a signal, not a proof. Use a confidence interval or a simple range to show the uncertainty. Readers should be able to see how much weight the result can carry.
Report the direction of the change and the category. A lower score at the end of a session is different from a higher score at the end. A missed defect is different from a stricter interpretation. The category tells the team what to fix.
Recommend one change and one check. The change might be a shorter review block, a randomized case order, or a clearer rubric example. The check should be a repeat audit after the change. Do not recommend three changes at once, because the next result will be impossible to interpret.
Publish the audit to the reviewers before the managers use it. The reviewers should have a chance to correct the record and add context. A quality system that hides its own error will lose trust faster than one that reports it. The training program review can turn the finding into a shared lesson.
