|
| 1 | +{ |
| 2 | + "cells": [ |
| 3 | + { |
| 4 | + "cell_type": "markdown", |
| 5 | + "metadata": {}, |
| 6 | + "source": [ |
| 7 | + "# How single-shot detector (SSD) works?" |
| 8 | + ] |
| 9 | + }, |
| 10 | + { |
| 11 | + "cell_type": "markdown", |
| 12 | + "metadata": {}, |
| 13 | + "source": [ |
| 14 | + "## Image classification and object detection" |
| 15 | + ] |
| 16 | + }, |
| 17 | + { |
| 18 | + "cell_type": "markdown", |
| 19 | + "metadata": {}, |
| 20 | + "source": [ |
| 21 | + "Image classification in computer vision takes an image and predicts the object in an image, while object detection not only predicts the object but also finds their location in terms of bounding boxes. For example, when we build a swimming pool classifier, we take an input image and predict whether it contains a pool, while an object detection model would also tell us the location of the pool.\n", |
| 22 | + "\n", |
| 23 | + "<img src=\"../../static/img/classVdetection.png\" height=\"500\" width=\"500\">\n", |
| 24 | + "<center>Figure 1. Difference between classification and object detection</center>" |
| 25 | + ] |
| 26 | + }, |
| 27 | + { |
| 28 | + "cell_type": "markdown", |
| 29 | + "metadata": {}, |
| 30 | + "source": [ |
| 31 | + "For illustrative purpose, assuming there is at most one class and one object in an image, the output of an object detection model should include:\n", |
| 32 | + "- Probablity that there is an object, \n", |
| 33 | + "- Height of the bounding box, \n", |
| 34 | + "- Width of the bounding box, \n", |
| 35 | + "- Horizontal coordinate of the center point of the bounding box,\n", |
| 36 | + "- Vertical coordinate of the center point of the bounding box.\n", |
| 37 | + "\n", |
| 38 | + "This is just one of the conventions of specifying output. Different models and implementations may have different formats, but the idea is the same, which is to output the probablity and the location of the object." |
| 39 | + ] |
| 40 | + }, |
| 41 | + { |
| 42 | + "cell_type": "markdown", |
| 43 | + "metadata": {}, |
| 44 | + "source": [ |
| 45 | + "## Why sliding window approach wouldn't work?" |
| 46 | + ] |
| 47 | + }, |
| 48 | + { |
| 49 | + "cell_type": "markdown", |
| 50 | + "metadata": {}, |
| 51 | + "source": [ |
| 52 | + "It's natural to think of building an object detection model on the top of an image classification model. Once we have a good image classifier, a simple way to detect objects is to slide a 'window' across the image and classify whether the image in that window (cropped out region of the image) is of the desired type. Sounds simple! Well, there are at least two problems: \n", |
| 53 | + "- (1) How do you know the **size of the window** so that it always contains the object? Different types of objects (palm tree and swimming pool), even the same type of objects (e.g. a small building and a large buidling) can be of varying sizes as well. \n", |
| 54 | + "- (2) **Aspect ratio** (the ratio of height to width of a bounding box). A lot of objects can be present in various shapes like a building footprint will have a different aspect ratio than a palm tree.\n", |
| 55 | + "\n", |
| 56 | + "To solve these problems, we would have to try out different sizes/shapes of sliding window, which is very computationally intensive, especially with deep neural network. \n", |
| 57 | + "\n", |
| 58 | + "<img src=\"../../static/img/slidingwindow.gif\" height=\"500\" width=\"500\">\n", |
| 59 | + "<center>Figure 2. Example of sliding window approach</center>" |
| 60 | + ] |
| 61 | + }, |
| 62 | + { |
| 63 | + "cell_type": "markdown", |
| 64 | + "metadata": {}, |
| 65 | + "source": [ |
| 66 | + "In practice, there are two types of mainstream object detection algorithms. Algorithms like R-CNN and Fast(er) R-CNN use a two-step approach - first to identify regions where objects are expected to be found and then detect objects only in those regions using convnet. On the other hand, algorithms like YOLO (You Only Look Once) [1] and SSD (Single-Shot Detector) [2] use a fully convolutional approach in which the network is able to find all objects within an image in one pass (hence ‘single-shot’ or ‘look once’) through the convnet. The region proposal algorithms usually have slightly better accuracy but slower to run, while single-shot algorithms are more efficient and has as good accuracy and that's what we are going to focus on in this section." |
| 67 | + ] |
| 68 | + }, |
| 69 | + { |
| 70 | + "cell_type": "markdown", |
| 71 | + "metadata": {}, |
| 72 | + "source": [ |
| 73 | + "> To follow the guide below, we assume that you have some basic understanding of the convolutional neural networks (CNN) concept. You can refresh your CNN knowledge by going through this short paper “[A guide to convolution arithmetic for deep learning](https://arxiv.org/pdf/1603.07285.pdf)”." |
| 74 | + ] |
| 75 | + }, |
| 76 | + { |
| 77 | + "cell_type": "markdown", |
| 78 | + "metadata": {}, |
| 79 | + "source": [ |
| 80 | + "## Single-Shot Detector (SSD)" |
| 81 | + ] |
| 82 | + }, |
| 83 | + { |
| 84 | + "cell_type": "markdown", |
| 85 | + "metadata": {}, |
| 86 | + "source": [ |
| 87 | + "SSD has two components: a __backbone__ model and __SSD head__. _Backbone_ model usually is a pre-trained image classification network as a feature extractor. This is typically a network like ResNet trained on ImageNet from which the final fully connected classification layer has been removed. We are thus left with a deep neural network that is able to extract semantic meaning from the input image while preserving the spatial structure of the image albeit at a lower resolution. For ResNet34, the backbone results in a 256 7x7 feature maps for an input image. We will explain what feature and feature map are later on. The _SSD head_ is just one or more convolutional layers added to this backbone and the outputs are interpreted as the bounding boxes and classes of objects in the spatial location of the final layers activations. \n", |
| 88 | + "\n", |
| 89 | + "In the figure below, the first few layers (white boxes) are the backbone, the last few layers (blue boxes) represent the SSD head.\n", |
| 90 | + "<img src=\"https://cdn-images-1.medium.com/max/1000/1*GmJiirxTSuSVrh-r7gtJdA.png\">\n", |
| 91 | + "<center>Figure 3. Architecture of a convolutional neural network with a SSD detector [2]</center>" |
| 92 | + ] |
| 93 | + }, |
| 94 | + { |
| 95 | + "cell_type": "markdown", |
| 96 | + "metadata": {}, |
| 97 | + "source": [ |
| 98 | + "Next, let's go through the important concepts/parameters in SSD." |
| 99 | + ] |
| 100 | + }, |
| 101 | + { |
| 102 | + "cell_type": "markdown", |
| 103 | + "metadata": {}, |
| 104 | + "source": [ |
| 105 | + "### Grid cell\n", |
| 106 | + "\n", |
| 107 | + "Instead of using sliding window, SSD divides the image using a grid and have each grid cell be responsible for detecting objects in that region of the image. Detection objects simply means predicting the class and location of an object within that region. If no object is present, we consider it as the background class and the location is ignored. For instance, we could use a 4x4 grid in the example below. Each grid cell is able to output the position and shape of the object it contains.\n", |
| 108 | + "\n", |
| 109 | + "<img src=\"../../static/img/gridcell.png\" height=\"300\" width=\"300\">\n", |
| 110 | + "<center>Figure 4. Example of a 4x4 grid</center>\n", |
| 111 | + "\n", |
| 112 | + "Now you might be wondering what if there are multiple objects in one grid cell or we need to detect multiple objects of different shapes. There is where anchor box and receptive field come into play." |
| 113 | + ] |
| 114 | + }, |
| 115 | + { |
| 116 | + "cell_type": "markdown", |
| 117 | + "metadata": {}, |
| 118 | + "source": [ |
| 119 | + "### Anchor box\n", |
| 120 | + "\n", |
| 121 | + "Each grid cell in SSD can be assigned with multiple anchor/prior boxes. These anchor boxes are pre-defined and each one is responsible for a size and shape within a grid cell. For example, the swimming pool in the image below corresponds to the taller anchor box while the building corresponds to the wider box.\n", |
| 122 | + "\n", |
| 123 | + "<img src=\"../../static/img/anchorbox.png\" height=\"480\" width=\"480\">\n", |
| 124 | + "<center>Figure 5. Example of two anchor boxes</center>\n", |
| 125 | + "\n", |
| 126 | + "\n", |
| 127 | + "SSD uses a matching phase while training, to match the appropriate anchor box with the bounding boxes of each ground truth object within an image. Essentially, the anchor box with the highest degree of overlap with an object is responsible for predicting that object’s class and its location. This property is used for training the network and for predicting the detected objects and their locations once the network has been trained. In practice, each anchor box is specified by an aspect ratio and a zoom level.\n", |
| 128 | + "\n", |
| 129 | + "#### Aspect ratio\n", |
| 130 | + "\n", |
| 131 | + "Not all objects are square in shape. Some are longer and some are wider, by varying degrees. The SSD architecture allows pre-defined aspect ratios of the anchor boxes to account for this. The ratios parameter can be used to specify the different aspect ratios of the anchor boxes associates with each grid cell at each zoom/scale level.\n", |
| 132 | + "\n", |
| 133 | + "<img src=\"../../static/img/aspect_ratio.png\" height=\"350\" width=\"350\">\n", |
| 134 | + "<center>Figure 6. The bounding box of building 1 is higher, while the bouding box for building 2 is wider</center>\n", |
| 135 | + "\n", |
| 136 | + "\n", |
| 137 | + "#### Zoom level\n", |
| 138 | + "\n", |
| 139 | + "It is not necessary for the anchor boxes to have the same size as the grid cell. We might be interested in finding smaller or larger objects within a grid cell. The zooms parameter is used to specify how much the anchor boxes need to be scaled up or down with respect to each grid cell. Just like what we have seen in the anchor box example, the size of building is generally larger than swimming pool.\n" |
| 140 | + ] |
| 141 | + }, |
| 142 | + { |
| 143 | + "cell_type": "markdown", |
| 144 | + "metadata": {}, |
| 145 | + "source": [ |
| 146 | + "### Receptive Field\n", |
| 147 | + "\n", |
| 148 | + "Receptive field is defined as __the region in the input space that a particular CNN’s feature is looking at (i.e. be affected by)__. We will use \"feature\" and \"activation\" interchangeably here and treat them as the linear combination (sometimes applying an activation function after that to increase non-linearity) of the previous layer at the corresponding location [3]. Because of the the convolution operation, features at different layers represent different sizes of region in the input image. As it goes deeper, the size represented by a feature gets larger. In this example below, we start with the bottom layer (5x5) and then apply a convolution that results in the middle layer (3x3) where one feature (green pixel) represents a 3x3 region of the input layer (bottom layer). And then apply the convolution to middle layer and get the top layer (2x2) where each feature corresponds to a 7x7 region on the input image. These kind of green and orange 2D array are also called __feature maps__ which refer to a set of features created by applying the same feature extractor at different locations of the input map in a sliding window fastion. Features in the same feature map have the same receptive field and look for the same pattern but at different locations. This creates the spatial invariance of ConvNet.\n", |
| 149 | + "<img src=\"../../static/img/receptive1.png\" height=\"500\" width=\"500\">\n", |
| 150 | + "<center>Figure 7. Visualizing CNN feature maps and receptive field</center>\n", |
| 151 | + "\n", |
| 152 | + "Receptive field is the central premise of the SSD architecture as it enables us to detect objects at different scales and output a tighter bounding box. Why? As you might still remember, the ResNet34 backbone outputs a 256 7x7 feature maps for an input image. If we specify a 4x4 grid, the simplest approach is just to apply a convolution to this feature map and convert it to 4x4. This approach can actually work to some extent and is exatcly the idea of YOLO (You Only Look Once). The extra step taken by SSD is that it applies more convolutional layers to the backbone feature map and has each of these convolution layers output a object detection results. __As earlier layers bearing smaller receptive field can represent smaller sized objects, predictions from earlier layers help in dealing with smaller sized objects__.\n", |
| 153 | + "\n", |
| 154 | + "Because of this, SSD allows us to define __a hierarchy of grid cells__ at different layers. For example, we could use a 4x4 grid to find smaller objects, a 2x2 grid to find mid sized objects and a 1x1 grid to find objects that cover the entire image. " |
| 155 | + ] |
| 156 | + }, |
| 157 | + { |
| 158 | + "cell_type": "markdown", |
| 159 | + "metadata": {}, |
| 160 | + "source": [ |
| 161 | + "## SSD implementation in `arcgis.learn`\n", |
| 162 | + "\n", |
| 163 | + "Armed with these fundamental concepts, we are now ready to define a SSD model. `arcgis.learn` allows us to define a SSD architecture just through a single line of code. For example:\n", |
| 164 | + "\n", |
| 165 | + " ssd = SingleShotDetector(data, grids=[4], zooms=[1.0], ratios=[[1.0, 1.0]])\n", |
| 166 | + "\n", |
| 167 | + "The grids parameter specifies the size of the grid cell, in this case 4x4. Additionally, we are specifying a zoom level of 1.0 and aspect ratio of 1.0:1.0. What this essentially means is that the network will create an anchor box for each grid cell, which is the same size as the grid cell (zoom level of 1.0) and is square in shape with an aspect ratio of 1.0:1.0. The output activations along the depth of the final feature map are used to shift and scale (within a reasonable limit) this anchor box so it can approach the actual bounding box of the object even if it doesn’t exactly match with the anchor box. \n", |
| 168 | + "\n", |
| 169 | + "For more information about the API, please go to the [API reference](https://esri.github.io/arcgis-python-api/apidoc/html/arcgis.learn.html#singleshotdetector)." |
| 170 | + ] |
| 171 | + }, |
| 172 | + { |
| 173 | + "cell_type": "markdown", |
| 174 | + "metadata": {}, |
| 175 | + "source": [ |
| 176 | + "## References\n", |
| 177 | + "- [1] Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi: “You Only Look Once: Unified, Real-Time Object Detection”, 2015; <a href='https://arxiv.org/abs/1506.02640'>arXiv:1506.02640</a>.\n", |
| 178 | + "- [2] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu: “SSD: Single Shot MultiBox Detector”, 2016; <a href='http://arxiv.org/abs/1512.02325'>arXiv:1512.02325</a>.\n", |
| 179 | + "- [3] Zeiler, Matthew D., and Rob Fergus. \"Visualizing and understanding convolutional networks.\" In European conference on computer vision, pp. 818-833. springer, Cham, 2014.\n", |
| 180 | + "- [4] Dang Ha The Hien. A guide to receptive field arithmetic for Convolutional Neural Networks. https://medium.com/mlreview/a-guide-to-receptive-field-arithmetic-for-convolutional-neural-networks-e0f514068807" |
| 181 | + ] |
| 182 | + } |
| 183 | + ], |
| 184 | + "metadata": { |
| 185 | + "kernelspec": { |
| 186 | + "display_name": "Python 3", |
| 187 | + "language": "python", |
| 188 | + "name": "python3" |
| 189 | + }, |
| 190 | + "language_info": { |
| 191 | + "codemirror_mode": { |
| 192 | + "name": "ipython", |
| 193 | + "version": 3 |
| 194 | + }, |
| 195 | + "file_extension": ".py", |
| 196 | + "mimetype": "text/x-python", |
| 197 | + "name": "python", |
| 198 | + "nbconvert_exporter": "python", |
| 199 | + "pygments_lexer": "ipython3", |
| 200 | + "version": "3.7.2" |
| 201 | + }, |
| 202 | + "toc": { |
| 203 | + "base_numbering": 1, |
| 204 | + "nav_menu": {}, |
| 205 | + "number_sections": true, |
| 206 | + "sideBar": true, |
| 207 | + "skip_h1_title": false, |
| 208 | + "title_cell": "Table of Contents", |
| 209 | + "title_sidebar": "Contents", |
| 210 | + "toc_cell": false, |
| 211 | + "toc_position": {}, |
| 212 | + "toc_section_display": true, |
| 213 | + "toc_window_display": false |
| 214 | + } |
| 215 | + }, |
| 216 | + "nbformat": 4, |
| 217 | + "nbformat_minor": 2 |
| 218 | +} |
0 commit comments