What is an acceptance test in a vision inspection pilot?
An acceptance test is the test in which you decide, against criteria written in advance, on images never used in training and under real line conditions, whether the system goes live. A successful demo shows that the system can work; an acceptance test shows under which conditions it works, under which it does not, and what the line does then.
A good score on a public dataset does not answer that question either. The paper introducing the MVTec AD 2 dataset reports that current methods stay below 60% average AU-PRO on scenarios with transparent and overlapping objects, dark-field and back-light illumination, high variance in normal samples and extremely small defects. The dataset also includes test scenarios with changed lighting to measure robustness to real-world distribution shifts. Your line's lighting, product variety and shift pattern match none of these scenarios exactly; the acceptance evidence has to come from your own line.
Lot acceptance sampling in quality control names two risks separately. According to the NIST/SEMATECH e-Handbook of Statistical Methods, producer's risk is the probability of rejecting a lot at the acceptable quality level, and consumer's risk the probability of accepting a lot at the unacceptable defect level. For a pilot the equivalents are plain: rejecting a system that is good enough, and accepting one that is not. The size of the test set and the wording of the criteria decide how large those two risks are.
How do you write acceptance criteria?
Write the criteria before the model is trained; give each one a decision unit, a defect class, a measurement method and a consequence of failure, and freeze them before the final test. Fix the decision unit at the start: image, part, tray or production batch? For a part seen by four cameras, "recall per image" and "recall per part" are not the same number.
| Criterion | How it is stated | Where it is measured | What happens on failure |
|---|---|---|---|
| Critical defect recall | At most K misses among N confirmed defects, per class | Held-out test set and shadow mode | That class is removed from automatic rejection; human review continues |
| False alerts | At most X alerts per 1,000 parts or per shift | Shadow mode, at the real production rate | Threshold or imaging setup is revisited |
| Ungradable images | Share of blurred, glare-hit or out-of-frame captures | Shadow mode | Camera, lighting or triggering is fixed |
| Decision time | Including the slowest 1%, before the reject gate deadline | At line speed, during peak production | The defined safe path for late results takes over |
| Review workload | Review minutes per shift | Shadow mode | Staffing plan or threshold changes |
| Safe stop | Behaviour on camera loss, full queue and late result | Planned fault trial | Automatic rejection is not switched on |
Do not average a miss on a critical defect away with good results on cosmetic defects; every criterion has to pass on its own. Write down who signs the acceptance too: on most lines that is the quality manager, not the person who trained the model. Next to each criterion, add which data, which period and which person will measure it; a criterion without a measurement plan turns into an argument at the end of the pilot.
How many defect examples does an acceptance test need?
The number of confirmed critical defects in the test set decides how strong a claim your criterion can carry. Seeing no misses does not mean the miss rate is zero. The one-sided 95% upper bounds from the binomial distribution are:
| Confirmed critical defects | Misses | 95% upper bound on the miss rate |
|---|---|---|
| 12 | 0 | about 22% |
| 20 | 0 | about 14% |
| 30 | 0 | about 9.5% |
| 59 | 0 | about 5% |
| 100 | 0 | about 3% |
As a rule of thumb, with zero misses the upper bound is around 3/n. To say "the miss rate is below 5%", you need to catch all of at least 59 confirmed critical defects. You can compute these bounds with the Clopper-Pearson ("beta") method in statsmodels' proportion_confint. Its documentation notes that the method uses alpha/2 in each tail and that coverage becomes 1 − alpha/2 when the count is zero, so the default two-sided 95% calculation gives a slightly more conservative upper bound than the table. For the one-sided bounds in the table, use the function's alternative parameter.
If you cannot collect that many defects, write the criterion honestly: "12 of 12 confirmed cracks caught; upper bound on the miss rate about 22%." With a result like that, keeping human review for the class and accumulating confirmed defects is a more defensible decision than wiring it to automatic rejection.
How do you build the test set shift by shift?
Collect test images after the training data, on separate days and from every shift, and keep all images of the same part in the same set. scikit-learn's cross-validation guide explains that with dependent groups of samples, validation samples must come from groups that are not in training at all, and that data changing over time should be evaluated on "future" observations. In an acceptance test, that becomes this checklist:
- Time: test images were collected after the last day of training data.
- Shift: day, night and, if there is one, weekend shift are counted as separate slices, each with enough parts to interpret.
- Camera and station: each camera is reported separately; a good result on one camera does not cover another.
- Product and material: product revisions, suppliers and material batches are listed.
- Defect type and severity: every critical class is counted; a class missing from the test is reported as "not tested".
- Difficult conditions: glare, a dirty lens and partly hidden parts are kept in a separate slice.
- Labels: test images were double-labelled, disagreements were adjudicated by an expert, and the set was frozen with a version.
- Access: the people developing the model did not see the test images while tuning; the set is used for one final measurement.
How do you run shadow mode?
In shadow mode the model scores and records every part on the line but does not control the reject mechanism; the existing inspection keeps making the decision. Amazon SageMaker's documentation on shadow tests describes the same principle in software: the new variant receives a copy of live requests, only the production variant's responses go back to the caller, and the shadow variant's responses are logged for comparison. On the line, the new system is likewise measured on real production without affecting anyone.
Record the operator's decision without showing them the model output; an operator who sees the alert looks accordingly, and the comparison loses its meaning. For every part, store the model score, the decision, processing time, image quality information and the operator's decision under the same part ID. Have a quality engineer resolve disagreements and record the reason. Two kinds of disagreement answer different questions:
- Model alerted, operator passed: the part is physically re-inspected. If a real defect turns up, the current inspection was missing something; if not, the cause of the false alert (reflection, dust, surface texture) is recorded.
- Operator rejected, model passed: a possible miss. The image is checked for whether the defect is visible to the camera at all; if it is not, the problem is the imaging setup, not the model.
Define the length of shadow mode by coverage, not by a number of days: it should not end before every shift, every product variant and at least one cleaning or maintenance cycle have been seen. For the wider context of shadow mode, thresholds and the review flow, see our article on vision-based quality control in production.
Hypothetical example: a shift-by-shift acceptance sheet for a cast part
The figures below are invented to illustrate the method. For crack inspection on a cast housing, the criteria were written and frozen in advance; results from the test set and two weeks of shadow mode are split by shift:
| Criterion | Target | Day | Night | Result |
|---|---|---|---|---|
| Critical crack recall | No misses | 18/18 | 13/14 | Fail |
| False alerts (per 1,000 parts) | At most 15 | 9 | 22 | Fail |
| Ungradable images | At most 1% | 0.4% | 1.8% | Fail |
| Slowest 1% decision time | At most 120 ms | 84 ms | 86 ms | Pass |
| Review workload (minutes/shift) | At most 45 | 25 | 58 | Fail |
Overall, 31 of 32 cracks were caught, or 96.9%; reported as a single average, the pilot could have looked like a success. The shift split tells a different story: the night shift fails four of the five criteria, while the day shift passes all of them. The crack missed at night and the rise in ungradable captures may point to the same cause: ambient light that behaves differently at night. The right decision is not to tune the threshold on the night data; add light shielding and repeat the measurement with a new, untouched night test set. Until then, the system is not accepted for automatic rejection.
How do you monitor the first month after acceptance?
Make approval conditional on monitoring. Sample accepted and rejected parts every day, track the alert rate by shift and review every suspected critical miss individually. Write the rollback triggers in advance: the camera shifting (checked against a stored reference image), a new product revision, the review queue exceeding staffing, a sudden change in alert rate. Keep the manual inspection route open until real operating evidence has built up; a system that cannot be paused safely is not ready for automatic rejection.
If the threshold changes after the test set has been seen, the old result no longer holds; a new validation cycle and a new, untouched test set are needed. Store the model version, the threshold, the camera and lighting settings and the test set version together. Our visual quality inspection service page shows where the acceptance test sits among the pilot steps and which breakdowns the report uses.
Common mistakes
- Writing or changing the criteria after seeing the test results.
- Reporting a single average without breakdowns by shift, camera and product.
- Using images of the same part in both training and test.
- Showing operators the model alert in shadow mode and still expecting an independent decision.
- Presenting "no misses" on a handful of defects as a firm guarantee.
- Never trying failure cases such as camera loss or a late result.