How does data augmentation improve performance in Computer Vision?
This article introduces "image blending", an image augmentation method. It explains seamless image blending to generate realistic training data for object detection. A practical application is illustrated in the detection of starfish in images of the Great Barrier Reef.
Machine learning models are playing an increasingly important role in research and industry, thanks to impressive progress, particularly in tasks previously reserved for human experts. To learn, models go through a training phase for which they generally require labeled data. A general rule is that the more data available, the better the model, as it will be able to capture more patterns during its training. This is even more true as the problem and the data become more complex.
In the context of object recognition, it is not uncommon for the training set to be small, or even non-existent. It is indeed costly to manually label large volumes of images, especially when a certain level of expertise is required (in the field of medical imaging, for example).
One of the most widely used methods to address this problem is Data Augmentation (DA), which involves artificially increasing the size of the dataset using image manipulation techniques. This augmentation is typically performed by applying operations that modify the appearance of the image without altering its semantics: for example, by changing the brightness, performing a rotation/mirror effect, changing the scale, or adding noise.
In this article, we will focus on a more advanced augmentation method: image blending. Image blending is a process of transferring an image from a source domain to a target domain while ensuring that the transformed pixels conform to the target domain to maintain consistency.
The idea is to increase the number of images available for training an object detection model by directly adding the target object into different backgrounds. This makes it very easy to obtain labeled images: since we know where the object has been inserted, we can simultaneously create the corresponding bounding box (which serves as a label).
More specifically, we will look at Gradient Domain Blending, which allows for seamless image blending. The goal is to generate images without visible boundaries between the background and the added object, so that they are realistic enough to be useful for training a detection model.
The principle, inspired by this research, consists of solving the Poisson equation associated with the image gradient in the area defined by a mask where the blending is to take place. The mask helps to more precisely define where the object of interest is located in the image and to define a gradient to smooth the transition between the two images.
The algorithm takes four elements as input:
The output is a new, already labeled image containing the object of interest. By repeating this process on a sufficiently large number of background images, we can create a new database ready to be used for training an object detection algorithm like YOLO.
Shown opposite is an example of an object of interest and its corresponding mask: the white area more or less defines the object's position in the image.
The application example we will study comes from a dataset provided for a Kaggle competition focused on detecting a family of starfish (Crown-of-Thorns Starfish, or CoTS) in video footage of the Great Barrier Reef in Australia.
The available dataset contains 23,000 images, about 5,000 of which include the objects to be detected. The idea is to use the 18,000 seafloor images that do not contain useful information as a base for this augmentation method.
The first step is to build a dataset—in our case, about fifty entries—of objects to add to the backgrounds to simulate a variety of scenarios. The bulk of the manual work happens at this stage, as you collect images and define the corresponding masks. This dataset is then augmented using standard techniques (mirroring, rotation, contrast, etc.) to obtain the most varied object base possible.
However, generating images by randomly positioning an object on a background is not enough: to obtain a model that can be used in real-world scenarios, the generated images must be plausible. For example, imagine we want to develop a bicycle detection algorithm for roads. If we use this method to augment an existing dataset by inserting bicycle images onto background images containing a road, we want to insert the "bicycle" objects on the road and not, for instance, at the top of a tree on the side of the road.
Therefore, the second step consists of analyzing the real images in the dataset to use them as a reference. We are particularly interested in the size of the objects to be detected and their spatial distribution in the actual images.
The distributions obtained in this way will allow us to build images whose diversity is as close as possible to reality.
Finally, we move on to generating the new images:
Below are two before/after examples of augmented images with the corresponding label, which a detection model could use as a training set.
Even after analyzing real images to understand their context, it is still possible to generate aberrant images. For example, in the sample images above, one might inadvertently introduce objects to be detected in areas where it is impossible for real objects to exist (such as areas containing only water). Without additional verification, introducing this unrealistic data into the training set risks teaching the model incorrect patterns.
One solution is to start by training an initial model on the non-augmented dataset. Then, using a principle similar to GANs, we can use this model to detect the objects present in our new images: if the model based on real data manages to spot the new "fake" objects, we can infer that they are realistic enough to be used as a training base.
In conclusion, data augmentation is an essential component of the process for improving predictive models. More specifically, image blending allows us to go further in fields where few labeled data are available. However, one must not fall into the trap of augmenting without understanding the context of the data and the goal of the machine learning model, so as not to introduce aberrant data into our model.
Do you have an idea for a use case involving automatic image analysis but feel your training dataset is too limited? Contact us to determine how data augmentation could be useful.
Lorem ipsum dolor sit amet consectetur. Ipsum penatibus viverra feugiat amet varius pharetra porta risus lectus. Turpis enim massa non tincidunt in faucibus.
