Marine Debris Roadmap v4

OPTIONAL · ROBUSTNESS LAYER

Optional — Pseudo-Private / Shadow Holdout

A bounded shadow evaluation that can reduce internal-validation and public-leaderboard selection overfitting when time and data budget permit.

OPTIONALNEW SPLIT REQUIREDSKIP IF DEADLINE RISK

If this optional branch is activated

Create a NEW grouped shadow holdoutDraw from the development/training pool and remove those samples from subsequent finalist training.
Keep shadow metrics hiddenDo not inspect them during routine tuning.
Select and freeze finalistsUse normal development evidence.
Retrain finalists without shadow samples
Evaluate the frozen shortlistOnce or very rarely on the shadow holdout.
Use for robustness / sanityNever as an iterative tuning target.

Alternative robustness evidence

  • multiple seeds
  • grouped folds / OOF
  • slice consistency
  • conservative interpretation of small validation differences

Public leaderboard policy

  • Treat the public leaderboard as a submission sanity signal, not a hyperparameter oracle.
  • Do not repeatedly change thresholds, models, or ensembles from small public movements; that converts the public subset into another validation set and increases private-leaderboard risk.