Advances in application of deep learning in computer vision are often highly focalized on the kind of classification problems that can be summarized by challenges such as ImageNet (but also, for instance, PASCAL VOC - http://host.robots.ox.ac.uk/pascal/VOC/voc2012/) and the ConvNets suitable to crack it (Xception, VGG16, VGG19, ResNet50, InceptionV3, and MobileNet, just to quote the ones available in the well-known package Keras: https://keras.io/applications/).

Though deep learning networks based on ImageNet data are the actual state of the art, such networks can experience difficulties when faced with real-world applications. In fact, in practical applications, we have to process images that are quite different from the examples provided by ImageNet. In ImageNet the elements to be classified are clearly the only clear element present...

As a way of boosting the capabilities of the research community, Google research scientists and software engineers often develop state-of-the-art models and make them available to the public instead of keeping them proprietary. As described in the Google research blog post, https://research.googleblog.com/2017/06/supercharge-your-computer-vision-models.html , on October 2016, Google's in-house object detection system placed first in the COCO detection challenge, which is focused on finding objects in images (estimating the chance that an object is in this position) and their bounding boxes (you can read the technical details of their solution at https://arxiv.org/abs/1611.10012).

The Google solution has not only contributed to quite a few papers and been put to work in some Google products (Nest Cam - https://nest.com/cameras/nest-aware...

Given such a powerful tool made available by TensorFlow, our plan is to leverage its API by creating a class you can use for annotating images both visually and in an external file. By annotating, we mean the following:

• Pointing out the objects in an image (as recognized by a model trained on MS COCO)
• Reporting the level of confidence in the object recognition (we will consider only objects above a minimum probability threshold, which is set to 0.25, based on the speed/accuracy trade-offs for modern convolutional object detectors discussed in the paper previously mentioned)
• Outputting the coordinates of two opposite vertices of the bounding box for each image
• Saving all such information in a text file in JSON format
• Visually representing the bounding box on the original image, if required

In order to achieve such objectives, we need to:

1. Download...

We start scripting our project in the file tensorflow_detection.py by loading the necessary packages:

import os
import numpy as np
import tensorflow as tf
import six.moves.urllib as urllib
import tarfile
from PIL import Image
from tqdm import tqdm
from time import gmtime, strftime
import json
import cv2

In order to be able to process videos, apart from OpenCV 3, we also need the moviepy package. The package moviepy is a project that can be found at http://zulko.github.io/moviepy/ and freely used since it is distributed with an MIT license. As described on its home page, moviepy is a tool for video editing (that is cuts, concatenations, title insertions), video compositing (non-linear editing), video processing, or to create advanced effects.

The package operates with the most common video formats, including the GIF format. It needs the FFmpeg converter...

Everything related to this project started from the following paper: Speed/accuracy trade-offs for modern convolutional object detectors(https://arxiv.org/abs/1611.10012) by Huang J, Rathod V, Sun C, Zhu M, Korattikara A, Fathi A, Fischer I, Wojna Z, Song Y, Guadarrama S, Murphy K, CVPR 2017. Concluding this chapter, we have to thank all the contributors of the TensorFlow object detection API for their great job programming the API and making it open-source and thus free and accessible to anyone: Jonathan Huang, Vivek Rathod, Derek Chow, Chen Sun, Menglong Zhu, Matthew Tang, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, Jasper Uijlings, Viacheslav Kovalevskyi, Kevin Murphy. We also cannot forget to thank Dat Tran for his inspirational posts on medium of two MIT licensed projects on how to use the TensorFlow object...

This project has helped you to start immediately classifying objects in images with confidence without much hassle. It helps you to see what a ConvNet could do for your problem, focusing more on the wrap up (possibly a larger application) you have in mind,and annotating many images for training more ConvNets with fresh images of a selected class.

During the project, you have learned quite a few useful technicalities you can reuse in many projects dealing with images. First of all, you now know how to process different kinds of visual inputs from images, videos, and webcam captures. You also know how to load a frozen model and put it to work, and also how to use a class to access a TensorFlow model.

On the other hand, clearly, the project has some limitations that you may encounter sooner or later, and that may spark the idea to try to integrate your code and make it shine...