YOLO Model Training
Frame-by-frame object detection training using Ultralytics YOLO models. Wraps the Ultralytics training API with a fixed interface and CLI entry point, allowing models to be trained from the command line or imported directly into a pipeline.
Summary
As with preprocessing, we specify two workflows for training models: local and Sockeye (HPC). Local workflows run linearly, where each training process must be called individually. HPC workflows parallelize training jobs to run each of the eight experimental splits simultaneously.
| Property | Detail |
|---|---|
| Script | training_yolo.py |
| Framework | Ultralytics YOLO |
| Supported Models | YOLOv8, YOLOv11 (YOLO26+) |
| Input | Dataset YAML + pretrained weights / Dataset.tar + pretrained weights |
| Output | Model weights + training artifacts written to disk |
| CLI Support | Yes |
See Also:
04_train_yolo.shfor the YOLO training Sockeye job submission script andrun_training_pipeline.shfor the Sockeye orchestrator script.
Inputs
- Local: Reads training data directly from the local
dataset.yamlpath, loads the pretrained YOLO weights, updates the path references, and runs training. - Sockeye (HPC): Unpacks a compressed
dataset.tarfile locally to the compute node to avoid numerous slow file transfers over the HPC network. It updatesyaml_pathto the current node directory, loads pre-downloaded YOLO weights, and runs training against the freshly unpacked files.
Output
This script writes directly to disk and returns None. All outputs are organized by experimental split and written to project/name/split.
project/
└── name/
├── weights/
│ ├── best.pt # Best checkpoint evaluated by validation metric
│ └── last.pt # Final checkpoint from the last epoch
├── results.csv # Per-epoch training and validation metrics
└── args.yaml # Resolved runtime training configuration
How It Works
-
Environment Detection: The script detects whether the execution context is local or on Sockeye.
-
On Sockeye, the
dataset.tarfile is unpacked directly on the local compute node, and theyaml_pathvariable insidedataset.yamlis updated dynamically to target the node location. -
On a Local machine,
yaml_pathinsidedataset.yamlis updated to your current project path before training execution. -
Weight Allocation:
-
On Sockeye, a pretrained YOLO model is loaded from a local, pre-downloaded weights directory (configurable via
.env_sockeyeand prepared by runningbash 01_setup.sh). Ensure the model version downloaded and the variables specified in04_train_yolo.shmatch before running. -
Locally, weights are fetched directly from Ultralytics using the version and size flags passed via the CLI (e.g., resolving to
yolov8n.ptoryolo11n.pt). -
Training Execution: The model is trained on the dataset described in the generated YAML file.
-
Parameter Forwarding: Core training arguments—such as epochs, batch size, image size, patience, and device constraints—are passed directly to the underlying Ultralytics
model.train()execution call.
Core Pipeline Functionality
YAML Path Configuration & Dataset Unpacking
Because absolute paths on shared HPC clusters change depending on the assigned compute node, YAML target paths are initially specified as relative links (./dataset). This script resolves and updates paths dynamically at runtime to prevent pathing failures during training.
On Sockeye, datasets are bundled into a single dataset.tar archive to maximize filesystem network efficiency. This script unpacks the dataset onto the local compute node, updates the underlying configuration paths, and validates folder lengths and file counts to ensure complete data integrity before initializing the model.
Image Handling
Training images are dynamically resized to the target img_size resolution. When rect=True (default behavior), the longest side of the frame is restricted to img_size while fully preserving the original aspect ratio. This configuration is highly recommended for high-resolution footage like the 1920×1800 MooVision clips utilized in this pipeline.
Early Stopping
If the chosen validation metrics fail to improve for patience consecutive epochs, training terminates automatically to save compute cycles. You can also pass the time parameter to hard-cap the absolute wall-clock runtime in hours, regardless of how many epochs have finished.
Batch Size
Supports flexible, variable batch configuration. The default batch size on Sockeye is optimized at 64 to maximize throughput; allocating exceptionally large batch values on certain compute nodes may trigger Out Of Memory (OOM) faults.
Epoch Setting
The epochs parameter specifies the maximum number of full training passes over the dataset. While higher epoch caps allow for deep feature optimization, they significantly increase the probability of model overfitting.
Cache Optimization
Allows caching of dataset images and labels directly into memory using --cache. Caching is generally discouraged for this pipeline as the video frame datasets are exceptionally large and risk flooding memory limits. Given the high-speed local disk nodes on Sockeye, caching yields negligible performance benefits.
Overwriting Existing Runs
To prevent accidental data loss, the pipeline prevents overwriting existing directories by default. To replace an existing run folder with a new experiment, pass the --exist_ok flag.
Hardware & GPU Compatibility
The architecture is designed to scale across single or multi-GPU environments using the device and workers parameters. By default, device is set to "cpu" to ensure safe, error-free execution on machines lacking dedicated acceleration.
- To target a specific GPU, pass a single string identifier:
--device="0" - To run parallelized multi-GPU training, pass a collection list:
--device=[0,1] - The
workersargument (defaults to 8) regulates the number of parallel CPU data-loading threads used to feed data to the GPU.
Model Selection
Pretrained base weights are dynamically resolved at execution as yolo{version}{size}.pt. For example, setting model=26 (MooVision internally aliases 26 for v11 architectures) and model_size="m" will resolve to yolo11m.pt. Ultralytics automatically handles fetching remote files if matching weights are not discovered locally.
Command Line Arguments
For complete information regarding downstream engine configurations, visit the Ultralytics Training Reference Documentation.
| Argument | Type | Default | Required | Description |
|---|---|---|---|---|
--dataset |
str |
None | Yes | Path to the dataset configuration target file. |
--device |
str |
"cpu" |
No | Hardware acceleration choice. Examples: '0' or '0,1' for specific GPUs, 'cpu', 'cuda', or 'mps'. |
--workers |
int |
8 |
No | Number of parallel data loader worker processes. |
--name |
str |
"cross-sucking" |
No | Name identifier for the specific training run/experiment. |
--project |
str |
"MooVision" |
No | Name of the output project directory where results are saved. |
--weights_dir |
str |
"" |
No | Path to a local folder containing model weights. Leave completely blank to automatically download weights from the internet. |
--model |
int |
26 |
No | YOLO model version integer (e.g., 8 for YOLOv8, 26 for YOLO11/26+). |
--model_size |
str |
"n" |
No | Model scale/variant size configuration options: n (nano), s (small), m (medium), l (large), or x (xlarge). |
--epochs |
int |
50 |
No | Maximum number of full validation passes over the entire dataframe during training. |
--time |
float |
None |
No | Maximum absolute training duration allowed, specified in hours. |
--batch |
int |
32 |
No | Training batch size per iteration. |
--patience |
int |
20 |
No | Early stopping threshold. Training stops early if no evaluation improvements are seen after this many consecutive epochs. |
--img_size |
int |
640 |
No | Target image size (resolution pixel width/height) used for model training. |
--not_rect |
Flag | True |
No | Disables rectangular training. Including this flag switches the underlying configuration (rect) to False, forcing the system to stop maintaining native image aspect ratios. |
--do_not_save |
Flag | True |
No | Disables tracking saves. Including this flag switches the underlying configuration (save) to False, preventing training checkpoints and finalized model weights from saving to disk. |
--exist_ok |
Flag | False |
No | Enables project overwriting. Including this flag sets the parameter to True, allowing the script to safely overwrite existing project directories without throwing errors. |
--cache |
Flag | False |
No | Enables image caching. Including this flag sets the parameter to True, caching raw images in RAM/storage memory for significantly quicker batch step testing. |
Usage
Command Line (Local) Example
uv run scripts/training/training_yolo.py \
--dataset="data/training/pipeline_demo/dataset/dataset.yaml" \
--project="pipeline_demo" \
--name="demo_01" \
--device="cpu" \
--epochs=1 \
--batch=8
Function Reference
Frame-by-frame object detection training using Ultralytics YOLO models.
setup_node_dataset(dataset, base_name='dataset')
Extract and isolate datasets to process-specific local compute node directories.
This function sets up a task-isolated staging workspace inside high-speed
node-local storage (e.g., SSD/NVMe scratch spaces mapped via SLURM_TMPDIR)
to prevent file collisions when multi-task array jobs run concurrently on the
same physical hardware node. It extracts the raw dataset tarball via native system
utilities, checks the data structure for YOLO formatting compatibility, and
dynamically overwrites the dataset's internal configuration file (dataset.yaml)
with absolute pathing strings.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset
|
str
|
The path string targeting the dataset. On an HPC cluster loop, this maps
to the target |
required |
base_name
|
str
|
An identifier appended to the directory isolation signature naming convention
( |
"dataset"
|
Returns:
| Name | Type | Description |
|---|---|---|
local_yaml_path |
Path
|
The absolute path pointing to the dynamically updated and isolated configuration
|
on_cluster |
bool
|
Flags whether execution environments evaluated as an HPC cluster loop or a local fallback run. |
Raises:
| Type | Description |
|---|---|
FileNotFoundError
|
If the target dataset tar file is missing on a cluster, or if the internal
|
ValueError
|
If cluster execution conditions are true but |
RuntimeError
|
If native system |
Source code in scripts/training/training_yolo.py
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 | |
train_yolo_model(on_cluster, yaml_path, name, project, weights_dir, model, model_size, device, workers, exist_ok=False, epochs=100, time=None, patience=50, batch=16, img_size=640, save=True, rect=True, cache=False, **kwargs)
Train a YOLO model.
Takes in training set from preprocessing.py and trains a YOLOv8 model of the specified size.
Parameters are taken from ultralytics docs. For more information and arguments see: https://docs.ultralytics.com/modes/train.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
yaml_path
|
str
|
Path to data.yaml configuration file. File contains dataset paths, and classes (e.g. cross-sucking) |
required |
name
|
str
|
Experiment name. e.g. cross-sucking-yolo-v26-m |
required |
project
|
Path | str
|
Output directory. This is where output of model training is stored, including model weights and predictions. |
required |
weights_dir
|
str
|
Path to local pretrained YOLO weights (raw weights) |
required |
model
|
int
|
YOLO model version. i.e. 26 for v26, 8 for v8. |
required |
model_size
|
str
|
Model size (n=nano, s=small, m=medium, l=large, x=xlarge). A larger model may take longer to train or use, but it may also provide increased performance. |
required |
device
|
str
|
device to train on (0 for GPU, 'cuda' for CUDA, 'cpu' for CPU, 'mps' for GPU on Mac). |
required |
workers
|
int
|
Number of parallel threads to run on sockeye. |
required |
exist_ok
|
bool
|
If True, overwrite existing project/name directory. |
False
|
epochs
|
int
|
Number of training epochs. Each epoch represents one full pass of the dataest. A larger epoch value will increase training duration, but may also increase accuracy. |
100
|
time
|
float
|
Maximum training time in hours. This will override the number of epochs if reached first. Note, this may affect model performance and training times. |
None
|
patience
|
int
|
Number of epochs to wait until stopping the training due no improvements in validation metrics. |
50
|
batch
|
int or float
|
Batch size. See ultrlytics docs for more info: https://www.ultralytics.com/glossary/batch-size |
16
|
img_size
|
int
|
Image size for training. Resized to squares of img_size x img_size if rect = False, else keeps aspect ratio. This may affect model accuracy and computational complexity. |
640
|
save
|
bool
|
Enables saving of training checkpoints and final model weights. Set to True by default to save results of model training. |
True
|
rect
|
bool
|
If True, keeps image ratio when rescaling images. The longest side is equal to img_size. Can improve efficiency and speed but may affect accuracy. |
True
|
**kwargs
|
named arguemnts
|
See ultralytics docs for additional arguments. |
{}
|
Returns:
| Type | Description |
|---|---|
None
|
This function reads to disk and does not return anything. |
Examples:
.. code-block:: python
# After running scripts read_all_clips_index.py, and splitting.py run...
yaml_path = "data/processed/pipeline_testing/yolo_format" #/path/to/yaml/file
train_yolo_model(
yaml_path=yaml_path,
model_size="n",
epochs=3,
imgsz=640,
batch=8,
rect=True,
device="mps",
save=True,
patience=50,
name="cross_sucking",
project="MooVision",
)
Source code in scripts/training/training_yolo.py
250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 | |
options: show_source: false show_root_heading: true
Work In Progress (WIP)
- Flexible Parameter Forwarding: Integrate
kwargscapability into parsed argument behaviors. This will unlock direct passthrough optimization, allowing users to leverage any latent arguments supported natively within the Ultralytics API straight from this interface wrapper.
Target Interface Concept
Pass additional Ultralytics arguments via --kwargs (feature implementation pending):
python train_yolo_model.py --yaml_path dataset.yaml --kwargs lr0=0.01 cos_lr=True