Data Preprocessing Pipeline
This page documents how to utilize scripts/training/preprocessing_yolo.py to compile video data and bounding box annotations into structured training configurations optimized for fine-tuning YOLO object detection models.
Summary
To transition from abstract train.csv and val.csv splits to actual model optimization, the pipeline slices video clips into discrete images while simultaneously extracting compressed bounding box files.
This script processes videos alongside zipped annotation folders (e.g., CVAT output archives), synchronizes their frames, and produces a standardized YOLO-compliant architecture:
my_yolo_dataset/
├── dataset.yaml <-- Defines paths to train/val subsets & class names
├── images/ <-- Main directory for image files (.jpg)
│ ├── train/
│ └── val/
└── labels/ <-- Main directory for text annotations (.txt)
├── train/
└── val/
To optimize for parallel executions, we makes use of UBC's high performance computing cluster (HPC), UBC ARC Sockeye.
A note on HPC's:
- HPC's are optimized for transfering large files across storage space, however they are notoriously slow at transferring many small files. Since, preprocessing for YOLO models requires the creation and transfer of many small files, there are slight differences in workflows when running locally vs running on Sockeye. This script will automatically detect when it is running locally or on a cluster, and run the optimal workflow for each scenario.
Intput Requirements
| Property | Details |
|---|---|
| Format | train.csv, val.csv |
| Content | Upstream data indeces of a train/val split |
Before running the preprocessing execution block, ensure your local workspace satisfies the following dependencies:
train.csv/val.csvPaths: An upstream data index mapping individual clips to their corresponding annotation zip files.- Environment File: A fully populated local .env configuration mapping your local volume paths for ROOT_DIR and LOCAL_DIR.
- Downloaded videos and zipped annotations (else preprocessing time will be significantly extended by file downloads).
If your data indexes and splits are not yet generated, execute the upstream data layer pipeline sequentially via your terminal:
uv run scripts/read_all_clips_index.py
uv run scripts/splitting.py
Output
Local: Outputs the file structure layed out below. UBC ARC Sockeye: Outputs a tar file containing the dataset structure below to avoid transfer of many small files.
By default results are saved to ROOT_DIR/data/training/yolo/split_name.
data/training/yolo/split_name/
├── dataset.yaml
├── images/
│ ├── train/
│ │ ├── 0001_None_frame_000001.jpg
│ │ ├── 0001_None_frame_000002.jpg
│ │ ├── ...
│ │ ├── 0002_part01_frame_000001.jpg
│ │ ├── 0002_part01_frame_000002.jpg
│ │ ├── ...
│ └── val/
│ ├── 0003_part02_frame_000001.jpg
│ └── 0003_part02_frame_000002.jpg
│ └── ...
└── labels/
├── train/
│ ├── 0001_None_frame_000001.txt
│ ├── 0001_None_frame_000002.txt
│ ├── ...
│ ├── 0002_part01_frame_000001.txt
│ ├── 0002_part01_frame_000002.txt
│ ├── ...
└── val/
├── 0003_part02_frame_000001.txt
└── 0003_part02_frame_000001.txt
data.yaml
"path": "./dataset",
"train": "images/train",
"val": "images/val",
"nc": 1,
"names": "cross-sucking"
How it works
Local Workflow: Working directory is the root directory
1) Read in train.csv and val.csv from the root directory.
2) Extract a list of relative paths to cross-sucking videos, and pass this to extract_frames. Extract a list of relative paths to zipped annotation folder, and pass this to extract_labels. Note, relative paths of cross-sucking videos are stored in the clips_relative_path column, and relative paths of annotations are stored in the labelled_clips_relative_path column.
3) extract_frames reads in each video in the passed list, extracts frames as jpg images using MultiThreadPooling and Semaphore attributes, and saves frames to either an images/train/ or an images/val/ subfolder within the working directory depending on the split source.
4) extract_labels opens each .zip file in the list, extracts bounding box annotations as .txt files from 'obj_train_data', removes the prefix 'obj_train_data', and saves these to either a labels/train/ or a labels/val/ subfolder within the output directory depending on the split source.
5) build_yaml creates a yaml file at data/training/yolo/split_name/ within the working directory with required classes and paths. Note that the dataset path in the dataset.yaml file is set within training_yolo.py to accomodate running jobs on Sockeye where this path cannot be predetermined.
UBC ARC Sockeye Workflow: Working directory is a high-speed tmp/ folder on the compute node.
6) Build a tar file from the dataset and transfer this to ROOT_DIR/data/training/yolo/split_name/dataset.tar e.g. ROOT_DIR/data/training/yolo/random/dataset.tar.
note: This is specified to be transferred back to the compute node as a .tar file and unpacked there for training. This workflow avoids the transportation of many small files across the HPC and makes use of the fast transfer system for large files. In this way, we are able to optimize preprocessing time.
Edge Cases
"Crucial Synchronization Requirement"
YOLO models link images to annotations by replacing sections of file paths: (images -> labels, .jpg -> .txt). For the dataset to sync correctly, video processing frame rates must exactly match the frame indexes exported in the annotation bounding boxes. If there is a frame rate or naming mismatch, your labels will misalign with your target images. It is assumed that both videos and CVAT outputs do not have missing intermediary frames in this regard. Preprocessing_yolo.py handles frame matching with unique naming conventions based on the numeric ID and part ID for each video, and a customizable skip parameter that is passed to both frame and label extraction functions. This ensures each frame and label pair has a unique naming convention and can be passed safely downstream to training.
"Corrupted Videos, and Misaligned Annotations" When extracting frames, video corruption may result in a premature exit of a video extraction loops. If not dealt with, this will result in unaligned annotation where extracted labels do not have matching video frames. Moreover, CVAT removes non-event labels (.txt files) leading to the end of a video clip. This leads to scenarios where videos extend past annotaion labels and some frames do not have matching labels. To account for this, preprocessing_yolo.py has functionality to compare and remove both labels and frames which are missing the other part. This ensures all frames and labels exist as matching pairs, and training sets will not break or cause downstream disruptions when training YOLO models.
"Missing Videos, or Missing Annotation" This is handled when reading in data, if not if is handled during data validation where frames or labels without matching counterparts will be removed to training sets.
Core Pipeline Concepts
File Structure and Naming Mechanics
To match names across video frame to annotation labels, and prevent namespace collision across overlapping frame numbers, both extract_frames and extract_labels use parsing functions from scripts/matching.py to extract the numeric ID and part ID of each video/zipped file. Adding the numericID and partID ensures there are no repeated names, and allows functionality for reading in multiple videos and zipped files.
Files are uniformly written out using a deterministic unique index:
Format:
numericID_partID_frame_frameNum.jpg (or.txt)
Example:
0001_None_frame_000001.jpg, 0001_None_frame_000001.txt.
0003_part01_frame_000645.jpg, 0003_part01_frame_000645.txt
Challenges
extract_labels assumes that within zipped annotation folders bounding boxes exist in a obj_train_data folder as .txt files, along with images highlighted with said bounding boxes. extract_labels extracts only the .txt files in this folder, forcibly carrying the hierarchy obj_train_data/frame_000001.txt with it as well. The file paths of .txt files are then rewritten to remove the obj_train_data hierarchy and flatten the output. i.e. this changes filepaths from labels/train/obj_train_data/0001_None_frame_000001.txt to labels/train/0001_None_frame_000001.txt
Temporal Frame Skipping
The skip parameter controls downsampling density. For instance, setting skip=5 extracts every 5th frame. Using larger step intervals accelerates dataset generation and reduces spatial autocorrelation (redundant data), but excessive downsampling introduces temporal tracking errors across fast-moving targets.
Frame and Label Matching Validation
The skip parameter controls downsampling density. For instance, setting skip=5 extracts every 5th frame. Using larger step intervals accelerates dataset generation and reduces spatial autocorrelation (redundant data), but excessive downsampling introduces temporal tracking errors across fast-moving targets.
Overwrite Behaviours (force)
By default, the pipeline preserves existing targets to save disk I/O time. If an output target directory is present, the script skips parsing. To discard stale data matrices and completely rebuild your dataset structures from scratch, pass the explicit overwrite flag: force=True.
Usage
| Argument | Type | Required / Default | Description |
|---|---|---|---|
--train_path |
str |
Required | Path to the training data file. |
--val_path |
str |
Required | Path to the validation data file. |
--output_path |
str |
Required | Output directory where the train/val splits will be saved. |
--skip |
int |
1 |
Downsampling density. For example, 5 means the script will read every 5th frame. |
--force |
flag |
False |
Overwrite existing files in the output directory if specified. |
Examples
Basic
By default, the script requires a train path, val path and output directory specified as strings.
uv run scripts/training/preprocessing.py --input_path="data/processed/pipeline_demo/train.csv" --val_path="data/processed/pipeline_demo/val.csv" --output_dir="data/training/pipeline_demo/"
Skip Frames
Passing the --skip= argument controls the downsampling density. skip=5 reads every 5th frame.
uv run scripts/training/preprocessing.py --input_path="data/processed/pipeline_demo/train.csv" --val_path="data/processed/pipeline_demo/val.csv" --output_dir="data/training/pipeline_demo/" --skip=5
Overwriting Files
To overwrite files use the --force argument.
uv run scripts/training/preprocessing.py --input_path="data/processed/pipeline_demo/train.csv" --val_path="data/processed/pipeline_demo/val.csv" --output_dir="data/training/pipeline_demo/" --force
Function Reference
scripts.preprocessing.preprocessing_yolo
Preprocessing functionality for creating training sets to fine-tune YOLO object detection models.
WIP NOTES. NOTE 2: Look into sampling frames at x/second rather than a defined skip amount. NOTE 3: Examples are not finished and need to be properly updated.
package_tar_file(base_local_dir, working_dir, output_dir)
Package the dataset into a tar archive and transfer it to destination storage.
This function compresses the node-local staging dataset workspace into a single
uncompressed tarball (dataset.tar) inside the local scratch directory. It then
atomically relocates the resulting tarball to the final cluster network or project
scratch storage space and sweeps the temporary local directory to free up disk space.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
base_local_dir
|
Path
|
The root temporary directory allocated for the job on the local computing node
(e.g., fallback path or |
required |
working_dir
|
Path
|
The location of the structured dataset folder ( |
required |
output_dir
|
Path
|
The primary targeting directory on target storage where the final |
required |
Returns:
| Type | Description |
|---|---|
None
|
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
FileNotFoundError
|
If |
Notes
This pattern is optimized for high-performance computing (HPC) environments where node-local NVMe scratch structures minimize network bottlenecks during compilation.
process_single_split(df, working_dir, split, skip, force, clips_root_dir, labels_root_dir)
Process image frames, label batches, and purge orphans for a single dataset split.
This function executes an end-to-end extraction pipeline for a single data split (e.g., 'train' or 'val'). It extracts frames from source videos, extracts bounding box annotations from corresponding zip files, and cross-references them. Any frame generated by the video extractor that lacks a corresponding entry in the label registry is treated as an orphan and is unlinked from the file system.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Driving DataFrame containing tracking metadata for the target split. Must include the columns 'clip_relative_path' and 'labelled_clip_relative_path'. |
required |
working_dir
|
Path
|
The root destination directory where the workspace structure ('images' and 'labels' folders) is built. |
required |
split
|
str
|
The dataset split identifier name. Typically 'train' or 'val'. |
required |
skip
|
int
|
Frame downsampling step factor parameter passed directly down to the frame extractor engine. |
required |
force
|
bool
|
If True, forces re-extraction of video assets and labels even if the target storage folders already exist. |
required |
clips_root_dir
|
Path
|
The root path pointing to where the raw original video files are located. |
required |
labels_root_dir
|
Path
|
The root path pointing to where the raw original compressed annotation archive zips are located. |
required |
Returns:
| Type | Description |
|---|---|
None
|
|
Raises:
| Type | Description |
|---|---|
KeyError
|
If the driving |
Examples:
>>> import pandas as pd
>>> from pathlib import Path
>>> data = {"clip_relative_path": ["v1.mp4"], "labelled_clip_relative_path": ["l1.zip"]}
>>> df = pd.DataFrame(data)
>>> process_single_split(
... df=df, working_dir=Path("./work"), split="train", skip=2,
... force=False, clips_root_dir=Path("./clips"), labels_root_dir=Path("./labels")
... )
resolve_working_directory(output_dir)
Resolve the optimal node-local staging folder based on the host system.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output_dir
|
Path
|
The default fallback compilation directory path. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
working_directory |
Path
|
The verified, isolated local scratch workspace directory path. |
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
run_yolo_preprocessing(train_path, val_path, output_path, skip, force=False)
Execute the end-to-end YOLO preprocessing pipeline on a tracking index.
Reads a processed dataset file, splits the data into training and
validation subsets via a randomized train/test split, and systematically
calls `extract_frames` and `extract_labels` to generate a YOLO-compliant
object detection directory structure.
validate_dataset(working_dir)
Validate that train and validation dataset splits have matched file counts.
This function scans the specified working directory to count images (supporting .jpg, .jpeg, and .png formats) and labels (.txt format) separately across the 'train' and 'val' split directories. It ensures the bounding box annotations perfectly pair with available target images before feeding data into a YOLO training model pipeline.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
working_dir
|
Path
|
The root path containing the dataset workspace tree layout. Expected
internal subdirectory architecture requires:
- |
required |
Returns:
| Type | Description |
|---|---|
None
|
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
AssertionError
|
If there is a structural file count mismatch between the image assets and text annotation assets inside either the 'train' or 'val' split. |
Examples:
>>> from pathlib import Path
>>> workspace = Path("/home/user/yolo_dataset")
>>> validate_dataset(workspace)
Processing complete. Checking file counts...
Verification Counts (Local NVMe):
- Train Images: 1250 | Labels: 1250
- Val Images: 150 | Labels: 150
Update reccomendations
- Scale Out Compute: Update to chunk the dataset and run preprocessing across multiple nodes, where each node returns a .tar file of its share of the dataset. These can then be unpacked on the compute node during training. Cuts down on current preprocessing time of 1-4hrs and adds scalability if the number of videos increases.