In one week, one auditor failed the fire safety check at four of the five bars he visited. The two other auditors who use the same checklist haven’t failed it once in eight visits.
So which is it? Are those four bars a fire risk or does one auditor read the question differently from everyone else?
You can’t tell from the scores. And the question matters more than it seems. Every decision you make from audit data, which sites get attention, who gets praised, where the training budget goes, assumes that a pass means roughly the same thing wherever it appears. When it doesn’t, the dashboards still look perfectly confident. They just aren’t telling you the truth.
Two quiet ways audit data goes wrong
Most audit programs don’t have a data-quality problem in any dramatic sense. Nobody is inventing results. The issues are quieter than that, and they tend to come in two forms.
Audits done too fast. An auditor running late ticks through a 34-question checklist in the parking lot or fills in a week’s worth of audits from memory on Sunday night. Safety teams have a name for this, pencil whipping, and it happens in digital checklists just as easily as on paper. The audit is complete, the score is recorded and nobody actually checked most of it.
Auditors who score differently. One auditor reads “tables wiped and sanitized between guests” as “no visible spills.” Another expects to see the sanitizer bottle in use. Both are doing their job honestly. Over a year, though, every bar they visit inherits their standard, and the scores stop being comparable.
Neither problem shows up on a dashboard because both produce normal-looking audits. You have to look underneath the scores to find them, which is exactly the kind of tedious cross-checking an assistant is good at.
<<cta>>
Audits finished too fast to be real
Completed audits in monitorQA record how long they took and when they were submitted, alongside every answer. With monitorQA connected to an assistant like Claude (see the setup guide if you haven’t connected yet), you can put those things together:
“Which audits from the last three months were done too fast to be real? Use 5 seconds per question as the cut-off and show me the three most suspicious batches and why.”
Two things about that request. It asks for the worst few batches rather than every fast audit so you get something to act on instead of a long table. And it sets a floor per question rather than asking what’s “typical.” If rushing has become normal on your team, typical is rushed too, and comparing against it flags nothing. A floor of a few seconds per question is generous. Nobody checks a fire extinguisher and an exit sign in five seconds, let alone thirty-four things in a minute.
The grouping is where it gets interesting. One fast audit can be an honest re-check of a single area. Four audits for the same bar, submitted back to back days after they were due, look a lot more like catching up on paperwork. Five bars in four states in an hour and a half is not a route anyone drove.
A short duration still isn’t proof of anything on its own. Before drawing conclusions, a couple of things are worth checking:
- Is it a pattern? One fast audit is a question. The same auditor appearing in batch after batch is a conversation.
- How do your auditors work? If some fill in a paper copy on site and enter it later, their recorded time will be short by design, and their submissions will come in batches. Find out before you flag them. Note that the recorded time isn’t simply the gap between opening and submitting: an audit opened on one day and submitted two weeks later can still show sixteen minutes.
<<cta>>
Same checklist, different auditors
The second check is about consistency. Here, the trick is to compare auditors on the same thing, rather than comparing their average scores across different sites.
“Compare how each auditor answers the Bar Hygiene & Readiness Audit. Where do they disagree most and how many audits is each figure based on?”
Average scores alone can mislead here. An auditor who mostly covers your best bars will naturally post higher scores. Looking question by question, and where possible at bars that more than one auditor has visited, gives a much fairer picture.
Notice the two findings aren’t equally strong. The fire safety gap is striking, but that auditor covers different bars from the other two, so it could be him or it could be the bars. The second finding is cleaner: same bar, same checklist, different readings of the same two questions. That’s the kind of result that tells you exactly what to talk about.
Two cautions keep the comparison honest. First, small numbers swing wildly: four or five audits per auditor can make anyone look like an outlier by chance. That’s why the request asks for the number of audits behind each figure. Second, a difference only tells you someone is out of line with the group. It doesn’t tell you who’s right. Sometimes the strict auditor is the one applying the standard properly.
Calibrate, don’t catch out
It would be easy to turn this into a list of auditors to reprimand. That’s rarely the right use for it, and it tends to make the data worse, not better: people under suspicion get cautious, not consistent.
The more useful responses are practical:
- Clarify the question. If two auditors read a question two ways, the wording is probably the problem. Rewriting it fixes the data for everyone, which is the subject of Let your results rewrite your checklists, later in this series.
- Run a joint audit. Send two auditors to the same bar together, compare answers on the spot and agree what a pass looks like. An hour of this does more than any memo.
- Share the comparison with the team. When everyone can see where they sit against the group, most people adjust on their own.
If you do plan to use these figures in individual performance reviews, check with HR first. Expectations around monitoring and using employee data vary by company policy and by location.
<<cta>>
When you want a second opinion
Sometimes the right next step is simply another look. When one auditor’s results stand out and you can’t tell whether it’s the auditor or the site, send someone else:
“Book a hygiene audit at Minneapolis next week with a different auditor from last time. Say it’s a calibration visit and ask for photos of any failures. Show me before you create it.”
If the two results land close together, you’ve gained confidence in the original. If they don’t, you have two independent audits to compare, question by question, instead of a hunch.
Make it a regular check
Data quality drifts. New auditors join, checklists change, busy seasons squeeze time on site. Running both checks once a quarter is usually enough to catch drift before it distorts your trends. If your assistant supports scheduled tasks, you can put it on the calendar and have the results waiting (Stop checking dashboards covers how).
Before you trust the trend, check the data
Pick the checklist your organization relies on most, the one whose scores end up in management reports. Run the speed check and the auditor comparison on it once.
If nothing stands out, you can present those numbers with more confidence than before. If something does, you’ve found it before anyone else did.
