Evaluation Pipeline

This page documents how to utilize scripts/evaluation.py to evaluate the MooVision baseline cross-sucking detection pipeline by comparing model predictions against ground truth annotations.

Summary

To assess how well the trained YOLO model detects cross-sucking events, this script compares baseline model predictions (JSON) against ground truth annotations (the processed clips index CSV) across three levels:

Frame level — bounding box IoU for every predicted frame vs. ground truth frame, independent of temporal matching. Measures spatial detection accuracy of the YOLO model directly.
Event level — bounding box IoU, precision, recall, F1, and F2 for matched events (only events that passed the temporal IoU threshold).
Sequence level — temporal IoU, measuring how well predicted event time windows overlap with labeled event time windows.

Results are also stratified by pen and weaning stage to support research questions about whether model performance differs across pens or developmental stages.

Input Requirements

Property	Details
Format	Directory of `.json` files
Content	Prediction metadata produced by running inference (e.g. `baseline.py` or `seq_NMS.py`) on source videos

Property	Details
Format	`all_clips_index_raw.csv`
Content	Ground truth clip index produced by `read_all_clips_index.py`, containing confirmed cross-sucking events with timing, pen, weaning stage, day, and CVAT annotation zip paths

Before running the evaluation script, ensure your local workspace satisfies the following dependencies:

Prediction JSON files: generated by running inference upstream.
Ground truth CSV: generated by the upstream data indexing pipeline. If not yet generated, run:

uv run scripts/read_all_clips_index.py

(Optional) CVAT annotation zip files, if bounding box IoU metrics are required. Without these, --labelled_clips_dir can be omitted and all bbox IoU values will be 0.0.

Output

By default, results are printed to stdout. If --output is provided, results are saved as a structured JSON report:

results/evaluation_report.json

{
  "thresholds": {...},
  "frame_level": {...},
  "event_level": {...},
  "sequence_level": {...},
  "by_pen": [...],
  "by_weaning_stage": [...],
  "matched_pairs": [...]
}

How it works

load_predictions reads all prediction JSON files from the predictions directory and flattens them into a DataFrame, one row per predicted event.
load_ground_truth reads the ground truth CSV and renames columns to align with prediction column names.
match_predictions_to_ground_truth matches each predicted event to the ground truth event with the highest temporal IoU, classifying matches as true positives, false positives, or false negatives based on --temporal_iou_threshold and --confidence_threshold.
compute_frame_level_bbox_iou independently computes bounding box IoU across all predicted frames against ground truth frames loaded from CVAT annotation zip files, regardless of event-level temporal matching.
evaluate_by_stratum repeats the matching and metric computation separately for each pen and weaning stage.
generate_evaluation_report assembles all of the above into a single report, optionally saving it to --output.

Edge Cases

"Missing --labelled_clips_dir" If no CVAT annotation directory is provided, all bounding box IoU values (frame-level and event-level) default to 0.0. Temporal IoU and precision/recall/F-score metrics are unaffected, since they only depend on event start/end times.

"Frame Number Misalignment" Predicted frame numbers and CVAT-exported frame numbers rarely align exactly, due to frame skipping during inference or small fps rounding differences. Bounding box matching uses nearest-frame matching within a tolerance window (default 10 frames, ~0.3s at 30fps) rather than requiring an exact frame match. Each ground truth frame can only be matched once.

"Missing or Corrupted CVAT Zip Files" If a CVAT zip file referenced in the ground truth CSV cannot be found or read, load_gt_boxes_from_zip returns an empty list and prints a warning rather than raising an error. Affected events will have a bbox IoU of 0.0.

Core Pipeline Concepts

Temporal IoU vs. Bounding Box IoU

Temporal IoU (compute_temporal_iou) measures overlap between predicted and ground truth time windows, exactly like bounding box IoU but in 1D. Bounding box IoU (compute_bbox_iou) measures spatial overlap between predicted and ground truth boxes, but is normalized by the smaller box's area rather than the union — this corrects for camera distance bias, since calves closer to the camera produce larger bounding boxes that would otherwise inflate standard IoU scores.

Matching Strategy

Within a source video, each predicted event above --confidence_threshold is greedily matched to the ground truth event with the highest temporal IoU, provided that IoU meets --temporal_iou_threshold. Each ground truth event can only be matched once, preventing a single real event from being counted as multiple true positives.

F2 Score

In addition to F1, the report computes F2, which weights recall twice as heavily as precision. This reflects the project's risk tolerance: missing a real cross-sucking event is considered more costly than occasionally flagging a false one.

Usage

Basic

By default, the script requires --predictions and --ground_truth paths.

uv run scripts/evaluation.py \
    --predictions results/metadata/baseline/ \
    --ground_truth data/raw/all_clips_index_raw.csv

Saving a Report

Pass --output to save the full evaluation report as JSON.

uv run scripts/evaluation.py \
    --predictions results/metadata/baseline/ \
    --ground_truth data/raw/all_clips_index_raw.csv \
    --output results/evaluation_report.json

Including Bounding Box IoU

Pass --labelled_clips_dir to compute frame-level and event-level bounding box IoU using CVAT annotation zip files.

uv run scripts/evaluation.py \
    --predictions results/metadata/baseline/ \
    --ground_truth data/raw/all_clips_index_raw.csv \
    --labelled_clips_dir /path/to/cross_sucking_labelled \
    --output results/evaluation_report.json

Adjusting Thresholds

Use --confidence_threshold to control the minimum prediction confidence considered, and --temporal_iou_threshold to control the minimum temporal IoU required to count as a match. Both default to 0.5.

uv run scripts/evaluation.py \
    --predictions results/metadata/baseline/ \
    --ground_truth data/raw/all_clips_index_raw.csv \
    --confidence_threshold=0.6 \
    --temporal_iou_threshold=0.4

Changing Source Video FPS

Use --fps to set the frames per second of the source videos, used when converting clip start times to frame numbers for CVAT zip offsetting. Defaults to 30.0.

uv run scripts/evaluation.py \
    --predictions results/metadata/baseline/ \
    --ground_truth data/raw/all_clips_index_raw.csv \
    --fps=25.0