CS Detection Evaluation

This page documents how to use scripts/predict_bboxes.py to evaluate the spatial accuracy of the fine-tuned YOLO model's cross-sucking detection, independently of event-level temporal matching.

Summary

While evaluation.py measures whether the model correctly identifies when cross-sucking events occur, predict_bboxes.py measures whether the model correctly identifies which frames contain cross-sucking and where in the frame it is occurring. It does this by:

Running the fine-tuned YOLO model frame-by-frame on each video clip
Saving predicted bounding boxes as .txt files in YOLO normalized format, mirroring the CVAT ground truth annotation folder structure
Optionally comparing predicted .txt files against CVAT ground truth zip annotations to compute frame-level precision, recall, F1, F2, and average bounding box IoU

Frames with no detections are skipped — no empty .txt file is written, matching the behaviour of CVAT ground truth exports which only contain frames where a bounding box was annotated.

Input Requirements

Property	Details
Format	`.csv` file
Content	Processed clips index CSV containing `clip_relative_path` and `labelled_clip_relative_path` columns. Must be `processed_clips_index.csv`, not `all_clips_index_raw.csv` — only the processed index contains the `labelled_clip_relative_path` column needed for ground truth bounding box comparison

Property	Details
Format	`.pt` file
Content	Fine-tuned YOLO model weights produced by `training_yolo.py`

Property	Details
Format	Directory of CVAT annotation `.zip` files
Content	Ground truth bounding box annotations exported from CVAT — required only if frame-level evaluation metrics are needed. Without this, `--labelled_clips_dir` can be omitted and only predicted `.txt` files will be produced

Output

Predicted bounding box .txt files are saved to ROOT_DIR/output_path/, using the same naming convention as CVAT ground truth annotations:

<numeric_id>_<part_id>_frame_<XXXXXX>.txt

Each .txt file contains one line per detected bounding box in YOLO normalized format:

class_id  center_x  center_y  width  height

If --labelled_clips_dir is provided, a frame-level evaluation report is also saved to the output directory:

results/predicted_bboxes/<split_name>/frame_level_evaluation.json

{
  "true_positives": 0,
  "false_positives": 0,
  "false_negatives": 0,
  "precision": 0.0,
  "recall": 0.0,
  "f1": 0.0,
  "f2": 0.0,
  "avg_bbox_iou": 0.0
}

How it works

extract_predicted_bboxes reads the input CSV, loads the fine-tuned YOLO model once, then loops through each clip frame by frame.
save_bbox_as_yolo_txt converts raw YOLO model output for each frame into normalized YOLO format and saves it as a .txt file. Frames with no detections above --conf_threshold are skipped entirely.
If --labelled_clips_dir is provided, evaluate_frame_level is called after inference completes. It loads ground truth boxes from CVAT zip files and compares them against predicted .txt files frame by frame, counting true positives, false positives, and false negatives.

Edge Cases

"No detections on a frame" If the model makes no detections above --conf_threshold on a given frame, no .txt file is written for that frame. This matches the CVAT ground truth convention where unannotated frames have no corresponding .txt file.

"Missing clip files" If a clip listed in the input CSV cannot be found on disk, a warning is printed and that clip is skipped. All other clips continue processing.

"Missing CVAT zip files" If a CVAT zip file referenced in the ground truth CSV cannot be found, load_gt_boxes_from_zip returns an empty dict and that clip contributes no ground truth frames to the evaluation.

Core Pipeline Concepts

Frame-Level vs Event-Level Evaluation

This script evaluates at the frame level — each individual frame is either a true positive, false positive, or false negative independently. This is different from evaluation.py which evaluates at the event level, grouping consecutive detections into events and comparing event time windows using temporal IoU.

Frame-level evaluation directly measures the YOLO model's raw CS detection accuracy: when the model draws a bounding box, how accurate is it spatially, and how often does it detect CS in frames that actually contain CS?

Bounding Box Matching

A predicted frame is counted as a true positive if the model detected a bounding box on that frame AND the ground truth has a bounding box for that frame, with IoU >= --iou_threshold. If the model detects something but the ground truth has no box (or IoU is too low), it is a false positive. If the ground truth has a box but the model detected nothing, it is a false negative.

F2 Score

In addition to F1, the evaluation computes F2, which weights recall twice as heavily as precision. This reflects the project's risk tolerance: missing a real cross-sucking frame is considered more costly than occasionally flagging a false one.

Usage

Inference Only

Run inference and save predicted bounding boxes without evaluation:

uv run python scripts/predict_bboxes.py \
    --input_path data/processed/pipeline_demo/test.csv \
    --model_path data/yolo_training_runs/pipeline_demo/demo_01/weights/best.pt \
    --output_path results/predicted_bboxes/pipeline_demo/ \
    --skip 10

Inference + Evaluation

Run inference and evaluate against CVAT ground truth annotations:

uv run python scripts/predict_bboxes.py \
    --input_path data/processed/<split_name>/test.csv \
    --model_path data/yolo_training_runs/<split_name>/<run_name>/weights/best.pt \
    --output_path results/predicted_bboxes/<split_name>/<run_name>/ \
    --labelled_clips_dir /path/to/cross_sucking_labelled \
    --skip 10

Overwriting Existing Results

By default the script skips frames that already have a predicted .txt file. Pass --FORCE to overwrite existing files:

uv run python scripts/predict_bboxes.py \
    --input_path data/processed/pipeline_demo/test.csv \
    --model_path data/yolo_training_runs/pipeline_demo/demo_01/weights/best.pt \
    --output_path results/predicted_bboxes/pipeline_demo/ \
    --skip 10 \
    --FORCE

Adjusting Thresholds

Use --conf_threshold to control the minimum YOLO confidence score required to save a detection, and --iou_threshold to control the minimum bounding box IoU required to count a frame as a true positive during evaluation:

uv run python scripts/predict_bboxes.py \
    --input_path data/processed/pipeline_demo/test.csv \
    --model_path data/yolo_training_runs/pipeline_demo/demo_01/weights/best.pt \
    --output_path results/predicted_bboxes/pipeline_demo/ \
    --labelled_clips_dir /path/to/cross_sucking_labelled \
    --conf_threshold 0.25 \
    --iou_threshold 0.5 \
    --skip 10

Arguments

Argument	Type	Default	Description
`--input_path`	str	required	Path to input CSV relative to `ROOT_DIR`
`--model_path`	str	required	Path to YOLO model weights relative to `ROOT_DIR`
`--output_path`	str	required	Output directory relative to `ROOT_DIR`
`--labelled_clips_dir`	str	None	Path to CVAT annotation zip files. If provided, runs frame-level evaluation after inference
`--skip`	int	1	Process every Nth frame. Should match skip used in preprocessing
`--conf_threshold`	float	0.0	Minimum YOLO confidence score to save a detection
`--iou_threshold`	float	0.5	Minimum bbox IoU to count as a True Positive during evaluation
`--FORCE`	flag	False	Overwrite existing output files

Function Reference

`scripts.evaluation.predict_bboxes`

predict_bboxes.py

Runs a fine-tuned YOLO model frame-by-frame on cross-sucking video clips, saves predicted bounding boxes as .txt files in YOLO format mirroring the CVAT ground truth annotation folder structure, and optionally evaluates frame-level CS detection accuracy against ground truth annotations.

Output files use the same YOLO format as ground truth annotations: class_id center_x center_y width height (all values normalized between 0 and 1)

Frames with no detections are skipped — no empty .txt file is written, matching the behavior of CVAT ground truth exports which only contain frames where a bounding box was annotated.

Evaluation (optional): When --labelled_clips_dir is provided, computes frame-level precision, recall, F1, F2 and avg bbox IoU by comparing predicted .txt files against CVAT ground truth zip annotations. A TP is a frame where the model predicted a bbox AND ground truth has a bbox with IoU >= iou_threshold.

How to run (inference only): uv run python scripts/evaluation/predict_bboxes.py --input_path data/processed/pipeline_demo/test.csv --model_path data/yolo_training_runs/pipeline_demo/demo_01/weights/best.pt --output_path results/predicted_bboxes/pipeline_demo/ --skip 1

How to run (inference + evaluation): uv run python scripts/evaluation/predict_bboxes.py --input_path data/processed//test.csv --model_path data/yolo_training_runs///weights/best.pt --output_path results/predicted_bboxes/// --labelled_clips_dir /path/to/cross_sucking_labelled --skip 1

`compute_bbox_iou_normalized(pred, gt)`

Compute IoU between two bounding boxes in normalized YOLO format.

Both boxes are (cx, cy, w, h) with values between 0 and 1. Converts to (x1, y1, x2, y2) format for intersection computation.

Parameters:

Name	Type	Description	Default
`pred`	`tuple(cx, cy, w, h)`	Predicted bounding box in normalized YOLO format.	required
`gt`	`tuple(cx, cy, w, h)`	Ground truth bounding box in normalized YOLO format.	required

Returns:

Type	Description
`float`	IoU score between 0.0 and 1.0.

`evaluate_frame_level(df, output_dir, labelled_clips_dir, skip=1, iou_threshold=0.5)`

Evaluate frame-level CS detection accuracy by comparing predicted bounding box .txt files against ground truth CVAT zip annotations.

For each frame in each clip: - TP: model predicted a bbox AND ground truth has a bbox, and their IoU >= iou_threshold - FP: model predicted a bbox but ground truth has no bbox (or IoU < iou_threshold) - FN: ground truth has a bbox but model predicted nothing

Aggregates TP/FP/FN across all clips and all frames to compute a single set of frame-level precision, recall, F1, and F2 scores. F2 is prioritized since missing a real CS frame is more costly than a false positive.

Parameters:

Name	Type	Description	Default
`df`	`DataFrame`	Input CSV DataFrame containing clip paths and labelled_clip_relative_path.	required
`output_dir`	`Path`	Directory containing predicted .txt files from extract_predicted_bboxes().	required
`labelled_clips_dir`	`Path`	Root directory of CVAT annotation zip files (cross_sucking_labelled/).	required
`skip`	`int`	Frame skip value — must match the skip used during prediction so predicted and ground truth frame numbers align. Default 1.	`1`
`iou_threshold`	`float`	Minimum IoU between predicted and ground truth box to count as TP. Default 0.5.	`0.5`

Returns:

Type	Description
`dict`	Frame-level evaluation metrics: {precision, recall, f1, f2, avg_bbox_iou, tp, fp, fn, total_frames_with_gt, total_frames_with_pred}

`extract_predicted_bboxes(input_path, model_path, output_path, labelled_clips_dir=None, skip=1, conf_threshold=0.0, iou_threshold=0.5, FORCE=False)`

Run YOLO inference frame-by-frame on video clips, save predicted bounding boxes as .txt files, and optionally evaluate frame-level detection accuracy against CVAT ground truth annotations.

For each video clip in the input CSV: 1. Opens the video file from UNLABELLED_CLIPS_DIR 2. Reads every Nth frame (controlled by skip) 3. Passes each frame to the fine-tuned YOLO model 4. Saves predicted bounding boxes as a .txt file in YOLO format using the same naming convention as CVAT ground truth exports 5. Skips frames with no detections (no empty .txt file written)

If labelled_clips_dir is provided, runs frame-level evaluation after inference completes, comparing predicted .txt files against CVAT ground truth zip annotations to compute precision, recall, F1, F2, and average bbox IoU across all frames.

Output folder structure mirrors the ground truth zip file structure: ROOT_DIR/output_path/ _frame.txt

Parameters:

Name	Type	Description	Default
`input_path`	`str`	Path to CSV file containing clip paths and labels. Relative to ROOT_DIR (e.g. data/processed/pipeline_demo/test.csv). Must contain 'clip_relative_path' and 'labelled_clip_relative_path' columns.	required
`model_path`	`str`	Path to fine-tuned YOLO model weights (.pt file). Relative to ROOT_DIR (e.g. data/yolo_training_runs/pipeline_demo/demo_01/weights/best.pt).	required
`output_path`	`str`	Output directory for predicted bounding box .txt files. Relative to ROOT_DIR (e.g. results/predicted_bboxes/pipeline_demo/).	required
`labelled_clips_dir`	`str`	Path to root directory of CVAT annotation zip files (cross_sucking_labelled/). If provided, frame-level evaluation metrics are computed after inference. If None, skips evaluation.	`None`
`skip`	`int`	Process every Nth frame. skip=1 processes every frame, skip=5 processes every 5th frame. Should match the skip value used in preprocessing so frame numbers align with ground truth. Default 1.	`1`
`conf_threshold`	`float`	Minimum YOLO confidence score to save a detection. Default 0.0 (save all detections regardless of confidence).	`0.0`
`iou_threshold`	`float`	Minimum bbox IoU between predicted and ground truth box to count as a True Positive during evaluation. Default 0.5.	`0.5`
`FORCE`	`bool`	If True, overwrite existing output files. Default False.	`False`

Returns:

Type	Description
`None`	Saves .txt prediction files to ROOT_DIR/output_path/ and optionally prints and saves a frame-level evaluation report.

`load_gt_boxes_from_zip(zip_path, skip=1)`

Load ground truth bounding boxes from a CVAT annotation zip file.

Reads all frame_XXXXXX.txt files from the obj_train_data folder inside the zip, parses YOLO normalized format, and returns a dict mapping frame number to bounding box coordinates (still normalized).

Parameters:

Name	Type	Description	Default
`zip_path`	`Path`	Full path to the CVAT annotation zip file.	required
`skip`	`int`	Frame skip value — only loads frames that match the skip pattern so ground truth frames align with predicted frames. Default 1.	`1`

Returns:

Type	Description
`dict`	{frame_num: (cx, cy, w, h)} in normalized YOLO format. Empty dict if zip not found or no annotations.

`load_pred_boxes_from_txt(txt_path)`

Load predicted bounding boxes from a saved .txt file.

Parameters:

Name	Type	Description	Default
`txt_path`	`Path`	Path to the predicted .txt file in YOLO format.	required

Returns:

Type	Description
`list of tuple (cx, cy, w, h)`	All predicted boxes for this frame. Empty list if file missing.

`save_bbox_as_yolo_txt(path, results, frame_shape, conf_threshold=0.0)`

Save YOLO model predictions for a single frame as a .txt annotation file.

Uses YOLO normalized format matching CVAT ground truth exports: class_id center_x center_y width height All coordinate values are normalized between 0 and 1 relative to frame dimensions.

Class ID is forced to 0 (cross-sucking) regardless of the raw model output, since the fine-tuned model was trained on a single class and this guarantees exact format matching with ground truth annotation files.

Parameters:

Name	Type	Description	Default
`path`	`Path`	Output path for the .txt file.	required
`results`	`ultralytics Results`	Raw output from model(frame) — contains predicted boxes.	required
`frame_shape`	`tuple`	Shape of the video frame (height, width, channels) from frame.shape. Used to normalize pixel coordinates.	required
`conf_threshold`	`float`	Minimum confidence score to save a detection. Default 0.0 (save all).	`0.0`

Returns:

Type	Description
`bool`	True if at least one detection was saved, False if no detections above threshold (caller should skip writing the file).