What would make this state wrong?
The first snooker article showed how a perception pipeline can turn broadcast frames into an inspectable visual story. The September work asks a narrower question: how reliably can the current candidate identify balls, pockets and the playing surface in a still frame, and which failures remain hidden by the aggregate score?
That is an important shift in emphasis. A table can look convincingly annotated while containing one extra pink or a missed red. An aggregate detection measure may barely move, but the resulting state can be materially wrong for a downstream application. Counting the right objects, with the right colours, is a more demanding task than producing a plausible overlay.
I ran a bounded model campaign and retained explicit comparison receipts, selection settings and error diagnostics. The resulting September candidate is private research work. The existing service was not replaced by the campaign, and the candidate is not a production-certified tracking or tactical model. Those boundaries let us discuss the progress precisely without turning a research result into a broader deployment claim.
The interesting outcome is mixed. The candidate misses fewer labelled balls, improves pocket localisation and still produces excess detections in difficult scenes. That mixture is useful. It tells us what improved, what did not and where the next investigation should go.
Compare the errors, not just the headline
The frozen campaign completed fourteen trials, including five evaluation-only runs. Validation choices were fixed before the final pilot evaluation. The retained comparison covers 215 validation images and 95 pilot images. Both baseline and candidate are evaluated through the same campaign evaluator with their respective frozen settings.
On validation, false negatives fall from fourteen to eight and false positives from thirteen to twelve. On the pilot, false negatives fall from eleven to three while false positives remain at fourteen. Class-aware ball F1 rises from 99.13% to 99.41% on that pilot. The change is small in headline percentage terms, but the error counts reveal a much clearer improvement in missed detections.
The count diagnostics complicate the story in a productive way. Pilot frames with too many predicted balls increase from seven to eleven. Frames with too few fall from four to zero. Mean absolute total-count error remains unchanged. Equal totals can conceal an extra detection and a miss in the same frame, so a count alone cannot replace colour-aware matching.
Pocket localisation also improves: mean pilot error falls from 10.15 to 6.32 pixels when scaled to a 1080p image. No labelled pockets are missed in either split. The playing-surface component is retained from the incumbent because a newer candidate improved average validation overlap slightly while worsening the weakest case. Choosing the stronger overall system meant declining one apparently attractive component update.
| Measure | Baseline | Candidate |
|---|---|---|
| False positives | 14 | 14 |
| False negatives | 11 | 3 |
| Exact per-colour counts | 86.32% | 88.42% |
| Frames with excess total count | 7 | 11 |
| Mean pocket error at 1080p | 10.15 px | 6.32 px |
Pink is a different problem from red
The candidate's pilot errors are concentrated. Pink contributes seven false positives and no false negatives. Red contributes four false positives and two misses. Brown contributes one of each, while black contributes two false positives. White, yellow, green and blue have no recorded errors in this particular pilot.
These are object-detection errors, not a conventional confusion matrix over every pixel or possible background region. There is no defined population of true-negative ball boxes, so a false-positive rate using that denominator would not be meaningful. Precision, recall and raw counts are clearer descriptions of what the evaluator actually measured.
The distinction matters for the next development task. A system that misses tightly clustered reds may need different evidence from one that creates extra pink detections. Folding both into one score risks sending the team towards a generic improvement campaign when targeted diagnosis would be more useful.
It also changes how I would present confidence to a downstream product. A high overall score should not erase class-specific weaknesses. The output can carry diagnostics about implausible colour counts and excess detections without silently clipping the prediction to a desired total. The scene must remain an observation that can be challenged, rather than a tidy state manufactured to satisfy a rule.
Sometimes the disagreement is in the label
A read-only audit of all 1,000 unique images found one impossible labelled colour count: a frame with two browns. Visual inspection showed that the right-hand baulk ball was green but its box was labelled brown. A separate inspected image contained a visible pink with no corresponding annotation. The evaluator therefore counted a prediction at that location as a false positive.
Those findings change the interpretation of individual disagreements. They do not justify rewriting the entire result as if the model had been proven correct. Not every disputed label has been independently inspected, and a model's confidence is not evidence that its annotation should be accepted as truth.
The campaign retained the delivered labels unchanged and claimed no corrected score. That is the useful discipline: preserve the original evaluation, record the disputed cases and require an independently reviewed annotation revision before publishing a revised benchmark. It keeps model development from quietly becoming a process of moving the answer key.
This is one of the most important lessons in practical machine learning. Evaluation data is a product with its own quality requirements. Its categories, geometry, exclusions and review history shape the result. Treating it as an unquestionable spreadsheet at the end of training leaves a large part of the system unmanaged.
For a data leader, the response should be organisational as well as technical. Label review needs an owner, a version and a rule for when corrections enter the evaluation. The model developer should be able to challenge a label without also being the sole authority on the replacement. That separation protects both the research and the credibility of its results.
A good still frame is not a complete game model
The pilot was inspected during earlier development. It is not an untouched hidden benchmark. Filename-based sequence groups also do not establish independent match identities; correlated views may remain across partitions. These limitations matter more than the number of decimal places in the final F1 score.
The input population is mainly selected stationary frames. Motion, camera changes, severe occlusion and unusual crops need separate evaluation. A still frame cannot determine whether an absent ball was potted, hidden or outside the image. It does not establish fouls, breaks, outcomes or win probabilities.
The earlier visual demonstration remains useful as a product concept and explanation surface. Its temporal and illustrative overlays should not be confused with acceptance evidence for this narrower September still-image candidate. Each artifact answers a different question, and linking them should make that distinction clearer.
The work also keeps public evaluation contracts separate from private model implementation. A common task needs visible rules and repeatable scoring; it does not need private weights or hidden evaluation answers distributed to participants. Fairness depends on maintaining that boundary and testing the end-to-end contract before making any launch claim.
Use the result to choose the next experiment
The next useful benchmark would use fresh, development-disjoint matches and independently reviewed labels. It should retain difficult scenes rather than selecting only clean frames, report performance by relevant conditions and keep the final assessment separate from threshold selection. The aim is a stronger answer about generalisation, not simply a larger collection of familiar pictures.
The existing diagnostics already suggest focused work on excess detections, colour-count consistency and pocket geometry. Each should have a clear acceptance question. Reducing misses must not hide an unacceptable rise in extra objects; improving average geometry should not disguise a worse tail.
I see this as progress in both the model and the way the work is led. The campaign produced a better candidate on several measured dimensions, preserved a component that remained preferable and exposed uncertainty in the evaluation itself. Those are concrete outcomes that help the next person choose a sensible experiment.
The strongest technical story is not that the model is almost perfect. It is that we can now describe more precisely how it fails, which improvements survived a frozen comparison and which evidence is still missing. That is what turns a promising perception system into a programme that can keep improving.
Evidence: frozen 16 September 2026 campaign receipts and model card. Public chart inputs are in the aggregate evidence file. Labels were not changed; no corrected accuracy or production acceptance is claimed.