Integrating a new module

This tutorial aims to integrate a new pose detector in the framework. [Yolov8](https://docs.ultralytics.com/tasks/pose/) proposes a pose detector which takes as input an image and provides as output the positions of the detected joints and the bounding boxes. To do this, we will start by extending the abstract class ImageLevelModule.

Note

The first thing we need to do is determine at what level we want to run the module :

VideoLevelModule : You get all the detections from a video at once.

ImageLevelModule : You get all the detections from a single image.

DetectionLevelModule : You get the information of a single detection

ImageLevelModule extension

The abstract class ImageLevelModule is defined in the file pipeline/imagelevel_module.py. It allows to define a common interface for the different detectors and the automation of the aggregation of the results in the pipeline.

Basically, its role is to take as input an image and to provide as output the detections objects that the model will have made.

It will be necessary to define the functions __init__, preprocess and process whose expected behavior is described in the docstring of the class.

The __init__ function will be called once in main.py when the model is initialized.

The preprocess function is called at each iteration of the pipeline, it allows to define what is the expected behavior of the dataloader. It is in this function that we load the image, the metadata and apply the processing (if required) before providing it to the process function. The default behavior of the collate function is that of Pytorch, but it can of course be modified by changing the self.collate_fn attribute. Finally, it takes as argument a pd.Series object which contains the metadata related to the image, and a pd.DataFrame which contains the detections from the previous steps (if applicable).

The process function is called after the batching of the dataloader via the self.collate_fn function. It takes as argument the batch and the metadata linked to the input images. This is where the processing by the model must be done. This function must return the results of the detections in one of the following forms: Union[pd.Series, List[pd.Series], pd.DataFrame]. The mechanism for updating the results during the pipeline is based on the index of the detections. This is why it is important to assign a unique id to each object. This can be done by incrementing the self.id variable. This id should then be added as a name argument in the case of a pd.Series or as an index in the case of a pd.DataFrame. Finally, it is expected that at least the following attributes are assigned in the detections: image_id, video_id, category_id, bbox_ltwh and bbox_conf. Of course other elements can be added at this stage if needed.

An example of the implementation for Yolov8-Pose is available in the file wrappers/detect_multi/yolov8_pose.py. You will also need to add this new class to the wrappers/detect_multi/__init__.py file.

Config file

Our framework works with the [Hydra](https://hydra.cc/) configuration system which takes advantage of a hierarchical configuration via files. This is very convenient to easily change modules.

You will have to create a new .yaml configuration file, add the required arguments in the cfg field and link the _target_ to the new object for the instantiation. The completed example is available in configs/modules/pose_bottomup/yolov8_pose.yaml.

You will also have to change in the main config file configs/config.yaml the field modules/pose_bottomup: yolov8_pose (name of the new file) to infer with the new model.