Extract Labels Function References

Functions documentation for extract_labels.py

Function Reference

`scripts.preprocessing.extract_labels`

Functions for extracting labels from CVAT zip folders

`extract_labels(label_paths, working_dir, labels_root, frame_registry, split, skip, force=False)`

Extract labels from annotated data folders and match them to frames extracted from corresponding videos.

Extracts bounding box labels from .zip files (CVAT Output folders), and saves files to either a train, or val folder in the specified working directory.

If force = True, and the output directory structure already exists, this will overwrite the existing folders and rebuild the data from scratch.

extract_labels outputs data into a labels/ folder within the output directory. This is to create the data format required for fine-tuning YOLO models.

This function assumes that bounding box labels are saved as .txt files in obj_train_data folders within .zip files, and that label_paths lists relative paths to these .zip files within the labels_root directory.

Parameters:

Name	Type	Description	Default
`label_paths`	`List[str]`	List of relative paths within the labelled clips directory to zipped folders containing bounding box annotations for cross-sucking events.	required
`working_dir`	`Path`	Path to output directory. Points to node's local temp workspace when running on Sockeye.	required
`labels_root`	`Path`	Path to the labelled clips directory containing zipped folders with bounding box annotations for cross-sucking events.	required
`frame_registry`	`dict`	A dictionary holding sets of frames that have been saved for each video. Uses the unique numeric ID and part ID of each video as keys, to ensure all labels have a matching frame.	required
`split`	`str`	One of `train`, `val`. Dictates which split folder, train/ or val/, the labels should be extracted to.	required
`skip`	`int`	Controls the downsampling density; number of frames to skip. `skip=5` will read every 5th frame.	required
`force`	`bool`	If True, overwrites the existing data. Defaults to False.	`False`

Returns:

Type	Description
`dict[tuple, set] or None`	Returns a dictionary mapping video tracking keys to sets of saved frame labels if processing occurs. This is used via set subtraction to remove any video frames which do not have an associated label. Returns None if execution is skipped via early exit bypass.

Raises:

Type	Description
`ValueError`	If `label_paths` is an empty list. If `split` is not one of 'train' or 'val'. If `skip` is less than or equal to zero.
`TypeError`	If runtime argument types do not match specified parameter declarations.
`FileNotFoundError`	If any relative paths do not lead to files for labelled data. If `labels_root` does not exist on disk. If the target folder does not exist within the .zip files. If there are no `.txt` files found in the target folder.

Notes

extract_labels is meant to be used in conjunction with extract_frames to build datasets for training YOLO models. YOLO requires frames and labels are split into train/val sets with corresponding file names.

`extract_single_zip(zip_path, frame_registry, skip, file_prefix, video_key)`

Extract downsampled annotation files matching a valid frame registry from one zip.

Parses a single zipped CVAT export file, extracting text annotation data for frames that match downsampling strides and exist within the provided tracking registry.

Parameters:

Name	Type	Description	Default
`zip_path`	`Path`	The filesystem path leading to the target zip file archive.	required
`frame_registry`	`dict`	A multi-video tracking lookup dictionary. Keys are tuples matching `(numeric_id, part_id)` and values are sets of integers representing successfully extracted video frames.	required
`skip`	`int`	The downsampling stride value (e.g., `5` extracts every 5th frame).	required
`file_prefix`	`str`	The standardized name prefix string generated for the specific video file clip.	required
`video_key`	`tuple of (int, str or None)`	A pair composed of `(numeric_id, part_id)` used as a unique identifier for logging and checking against registries.	required

Returns:

Name	Type	Description
`label_batch`	`dict of {str : bytes}`	A mapping of target destination filenames to their raw, unwritten text-file binary data bytes.
`saved_frames`	`set of int`	A set tracking the sequential frame numbers successfully extracted and batched from this specific archive.

Raises:

Type	Description
`TypeError`	If `zip_path` is not a Path object, `frame_registry` is not a dict, `skip` is not an integer, `file_prefix` is not a string, or `video_key` is not a tuple.
`ValueError`	If `skip` is less than or equal to zero, or if a parsed file inside the zip archive contains an invalid frame index structure.
`FileNotFoundError`	If the file at `zip_path` does not exist on disk.

`generate_label_metadata(zip_name)`

Extract lookup keys and file prefixes from a CVAT export zip filename.

Parameters:

Name	Type	Description	Default
`zip_name`	`str`	The base name of the zip file archive.	required

Returns:

Name	Type	Description
`video_key`	`tuple of (int, str or None)`	A tracking tuple pair composed of (numeric_id, part_id) used to index and align annotations with processed frames.
`file_prefix`	`str`	The standardized prefix format used for outputting dataset files, formatted as `"{numeric_id:04}_{part_id_str}_"`.

Raises:

Type	Description
`TypeError`	If `zip_name` is not passed as a string.
`ValueError`	If `zip_name` is an empty string.

`parse_zip_annotations(validated_paths, frame_registry, skip)`

Parse zipped CVAT annotation exports into in-memory byte batches and matching registries.

Iterates through a list of validated paths to zip files containing bounding box annotations. Extracts text streams for downsampled frames that possess a valid corresponding entry in the provided frame registry.

Parameters:

Name	Type	Description	Default
`validated_paths`	`list of pathlib.Path`	A list of verified absolute paths to zipped CVAT data folders.	required
`frame_registry`	`dict`	A lookup dictionary tracking extracted video frames. Keys are tuples containing the numeric ID and part ID (`(int, str/None)`), and values are sets of integers representing successfully saved frame numbers.	required
`skip`	`int`	The sequence interval step size used for downsampling frame data (e.g., passing 5 filters and extracts every 5th sequential frame annotation).	required

Returns:

Name	Type	Description
`label_batch`	`dict of {str : bytes}`	An in-memory batch mapping target destination filenames (formatted for YOLO) to their raw text binary streams extracted directly from the zip file archives.
`saved_labels_registry`	`dict of {tuple(int, str or None) : set of int}`	A mapping of video keys to sets of frame indexes that were successfully processed and queued for extraction.

Raises:

Type	Description
`ValueError`	If a file within the target archive violates standard naming conventions preventing safe extraction of its sequential frame index.

`save_label_batch(label_batch, final_output_dir)`

Commit a batch of in-memory annotation byte streams to disk.

Iterates over a dictionary of pre-filtered filenames and raw text byte arrays, writing them directly to the specified local workspace directory (e.g., node-local NVMe staging or local repository folders). Prints periodic progress updates for large batch operations.

Parameters:

Name	Type	Description	Default
`label_batch`	`dict of {str : bytes}`	A mapping of target destination filenames (e.g., `'0001_part01_frame_000000.txt'`) to their raw annotation contents stored as unwritten text bytes.	required
`final_output_dir`	`Path`	The destination directory path where the text files will be saved.	required

Returns:

Type	Description
`None`	This function writes directly to the filesystem and does not return a value.

Raises:

Type	Description
`TypeError`	If `label_batch` is not a dictionary or if `final_output_dir` is not a pathlib.Path object.