An Approach to Granular Testing of Deep Neural Networks
Issues with aggregated metrics
Let us consider a DNN tasked with detecting pedestrians on images of traffic situations. We can obtain the model’s performance on the test dataset by using Intersection over Union (IoU), a commonly used semantic segmentation metric. Let us consider that we obtain a score of 80%, which is better than existing classical methods. Using only the aggregated score, one would consider deploying such a model in some real-world application. However, there could be subregions in the data space where the model performs worse than the average. Such regions of low performance could be potential weak spots learnt from the training data. We propose to consider human understandable semantic features that describe the scene as subregions of the data and calculate the performance metric by grouping over such semantic features.
Figure 1 shows a representation sketch of our example where we see the IoU along the semantic dimensions – skin color of pedestrian, which indicates a bias in the data. The color of the points in the image indicates the pedestrian’s skin color, and we can quite quickly notice that the model performs consistently poorly for dark-skinned pedestrians. This test can be extended for all possible semantic dimensions, which are safety-critical according to a human expert. While we expect independent and identically distributed sampling in test datasets, real-world does have temporal and semantic consistency. Suppose models are systematically weak for specific semantic dimensions in one scene. In that case, they will probably be weak for future scenes, and therefore, it is essential to identify such weak spots and fix the underlying issue. We call this semantic testing.
Semantic Testing
As one can imagine, generating labelled data for supervised learning is a time and cost-intensive task. Obtaining scene and semantic information in addition to labels would further complicate the labelling process. Certain semantic features can be extracted from automated classical image processing techniques using the raw images and labels, but there is a limit on how elaborate this approach can get. However, we can automate the data generation, labelling, and scene descriptions extraction processes using computer simulators.
For our work, we use one such open-source simulator for autonomous driving called the CARLA simulator. While it is possible to already extract raw images and labels directly, we had to make some modifications to the source code to extract the metadata of the scenes (code available on GitHub). Using our modified CARLA simulator, we create a large dataset of raw scenes, semantic segmentation labels and metadata of the scenes and use these to train a semantic segmentation DNN called DeepLabv3+. We perform various evaluations across semantic dimensions to study where weak spots in the model could occur. We can extend it further by generating special test sets as shown in Video 1, where we change a single semantic dimension in each image, for example the pedestrian position and evaluate the performance as shown in Figure 2. We can see that the performance is not uniform. As expected, pedestrians closer are detected better than pedestrians farther away. However, we can notice that there are other factors
influencing performance apart from distance because the performance is not symmetric.
Takeaways
Semantic testing offers a more granular and interpretable approach to testing DNNs. With semantic testing, one can answer questions of the type: Is the pedestrian detection model going to work for (i) all real positions a pedestrian could be in or are there certain locations where there is a higher chance of failure. (ii) are there clothing types which are difficult for a model. However, semantic testing when evaluated at single dimensions is still limited in its scope. Low performance along certain semantic features might not necessarily be due to that semantic feature itself but due to interplay between multiple semantic features, for example location + contrast + occlusion. In such situations, identifying which semantic features are important for the performance and which are not important is not trivial. Our future work looks at generating such an “importance matrix” of semantic dimensions that influence the performance.