Skip to content

Commit 3338fac

Browse files
scottamaincopybara-github
scottamain
authored and
copybara-github
committed
Add object detection example for TF Lite with Pi Camera. Draws bounding box around each detected object on the camera preview (above a given score threshold). Includes dependency downloads and walkthru Readme with option to accelerate using Coral Edge TPU.
PiperOrigin-RevId: 270932545
1 parent 711b11d commit 3338fac

8 files changed

Lines changed: 464 additions & 5 deletions

File tree

‎lite/README.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ The following samples demonstrate the use of TensorFlow Lite in mobile applicati
77

88
## Image classification
99

10-
This app performs image classification on a live camera feed and displays the inference output in realtime on the screen.
10+
This app performs image classification on a live camera feed and displays the
11+
inference output in realtime on the screen.
1112

1213
<!-- TODO(b/124116863): Add app screenshot and model details. -->
1314

@@ -21,7 +22,10 @@ This app performs image classification on a live camera feed and displays the in
2122

2223
## Object detection
2324

24-
This app performs object detection on a live camera feed and displays the results in realtime on the screen. The app displays the confidence scores, classes and detected bounding boxes for multiple objects. A detected object is only displayed if the confidence score is greater than a defined threshold.
25+
This app performs object detection on a live camera feed and displays the
26+
results in realtime on the screen. The app displays the confidence scores,
27+
classes and detected bounding boxes for multiple objects. A detected object is
28+
only displayed if the confidence score is greater than a defined threshold.
2529

2630
<!-- TODO(b/124116863): Add app screenshot and model details. -->
2731

@@ -31,6 +35,8 @@ This app performs object detection on a live camera feed and displays the result
3135

3236
[iOS object detection](examples/object_detection/ios/README.md)
3337

38+
[Raspberry Pi object detection](examples/object_detection/raspberry_pi/README.md)
39+
3440

3541
## Speech command recognition
3642

‎lite/examples/image_classification/raspberry_pi/classify_picamera.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/usr/bin/python3
1+
# python3
22
#
33
# Copyright 2019 The TensorFlow Authors. All Rights Reserved.
44
#
@@ -57,7 +57,8 @@ def classify_image(interpreter, image, top_k=1):
5757

5858

5959
def main():
60-
parser = argparse.ArgumentParser()
60+
parser = argparse.ArgumentParser(
61+
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
6162
parser.add_argument(
6263
'--model', help='File path of .tflite file.', required=True)
6364
parser.add_argument(

‎lite/examples/image_classification/raspberry_pi/download.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ else
77
fi
88

99
# Install required packages
10-
pip install -r requirements.txt
10+
python3 -m pip install -r requirements.txt
1111

1212
# Get TF Lite model and labels
1313
curl -O https://storage.googleapis.com/download.tensorflow.org/models/tflite/mobilenet_v1_1.0_224_quant_and_labels.zip

‎lite/examples/object_detection/raspberry_pi/README.md‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
# TensorFlow Lite Python object detection example with Pi Camera
2+
3+
This example uses [TensorFlow Lite](https://tensorflow.org/lite) with Python
4+
on a Raspberry Pi to perform real-time object detection using images
5+
streamed from the Pi Camera. It draws a bounding box around each detected
6+
object in the camera preview (when the object score is above a given threshold).
7+
8+
Although the TensorFlow model and nearly all the code in here can work with
9+
other hardware, the code uses the [`picamera`](
10+
https://picamera.readthedocs.io/en/latest/) API to capture images from the Pi
11+
Camera. So you can modify those parts of the code if you want to use a different
12+
camera input.
13+
14+
At the end of this page, there are extra steps to accelerate the example using
15+
the Coral USB Accelerator, which increases the inference speed by ~10x.
16+
17+
18+
## Set up your hardware
19+
20+
Before you begin, you need to [set up your Raspberry Pi](
21+
https://projects.raspberrypi.org/en/projects/raspberry-pi-setting-up) with
22+
Raspbian (preferably updated to Buster).
23+
24+
You also need to [connect and configure the Pi Camera](
25+
https://www.raspberrypi.org/documentation/configuration/camera.md).
26+
27+
And to see the results from the camera, you need a monitor connected
28+
to the Raspberry Pi. It's okay if you're using SSH to access the Pi shell
29+
(you don't need to use a keyboard connected to the Pi)—you only need a monitor
30+
attached to the Pi to see the camera stream.
31+
32+
33+
## Install the TensorFlow Lite runtime
34+
35+
In this project, all you need from the TensorFlow Lite API is the `Interpreter`
36+
class. So instead of installing the large `tensorflow` package, we're using the
37+
much smaller `tflite_runtime` package.
38+
39+
To install this on your Raspberry Pi, follow the instructions in the
40+
[Python quickstart](https://www.tensorflow.org/lite/guide/python).
41+
Return here after you perform the `pip install` command.
42+
43+
44+
## Download the example files
45+
46+
First, clone this Git repo onto your Raspberry Pi like this:
47+
48+
```
49+
git clone https://github.com/tensorflow/examples --depth 1
50+
```
51+
52+
Then use our script to install a couple Python packages, and
53+
download the MobileNet model and labels file:
54+
55+
```
56+
cd examples/lite/examples/object_detection/raspberry_pi
57+
58+
# The script takes an argument specifying where you want to save the model files
59+
bash download.sh /tmp
60+
```
61+
62+
63+
## Run the example
64+
65+
```
66+
python3 detect_picamera.py \
67+
--model /tmp/detect.tflite \
68+
--labels /tmp/coco_labels.txt
69+
```
70+
71+
You should see the camera feed appear on the monitor attached to your Raspberry
72+
Pi. Put some objects in front of the camera, like a coffee mug or keyboard, and
73+
you'll see boxes drawn around those that the model recognizes, including the
74+
label and score for each. It also prints the amount of time it took
75+
to perform each inference in milliseconds at the top-left corner of the screen.
76+
77+
For more information about executing inferences with TensorFlow Lite, read
78+
[TensorFlow Lite inference](https://www.tensorflow.org/lite/guide/inference).
79+
80+
81+
## Speed up the inferencing time (optional)
82+
83+
If you want to significantly speed up the inference time, you can attach an
84+
ML accelerator such as the [Coral USB Accelerator](
85+
https://coral.withgoogle.com/products/accelerator)—a USB accessory that adds
86+
the [Edge TPU ML accelerator](https://coral.withgoogle.com/docs/edgetpu/faq/)
87+
to any Linux-based system.
88+
89+
If you have a Coral USB Accelerator, follow these additional steps to
90+
delegate model execution to the Edge TPU processor:
91+
92+
1. First, be sure you have completed the [USB Accelerator setup instructions](
93+
https://coral.withgoogle.com/docs/accelerator/get-started/).
94+
95+
2. Now open the `detect_picamera.py` file and add the following import at
96+
the top:
97+
98+
```
99+
from tflite_runtime.interpreter import load_delegate
100+
```
101+
102+
And then find the line that initializes the `Interpreter`, which looks like
103+
this:
104+
105+
```
106+
interpreter = Interpreter(args.model)
107+
```
108+
109+
And change it to specify the Edge TPU delegate:
110+
111+
```
112+
interpreter = Interpreter(args.model,
113+
experimental_delegates=[load_delegate('libedgetpu.so.1.0')])
114+
```
115+
116+
The `libedgetpu.so.1.0` file is provided by the Edge TPU library you
117+
installed during the USB Accelerator setup in step 1.
118+
119+
3. Finally, you need a version of the model that's compiled for the Edge TPU.
120+
121+
Normally, you need to use use the [Edge TPU Compiler](
122+
https://coral.withgoogle.com/docs/edgetpu/compiler/) to compile your
123+
`.tflite` file. But the compiler tool isn't compatible with Raspberry
124+
Pi, so we included a pre-compiled version of the model in the `download.sh`
125+
script above.
126+
127+
So you already have the compiled model you need:
128+
`mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite`.
129+
130+
Now you're ready to execute the TensorFlow Lite model on the Edge TPU. Just run
131+
`classify_picamera.py` again, but be sure you specify the model that's compiled
132+
for the Edge TPU (it uses the same labels file as before):
133+
134+
```
135+
python3 classify_picamera.py \
136+
--model /tmp/mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite \
137+
--labels /tmp/coco_labels.txt
138+
```
139+
140+
You should see significantly faster inference speeds.
141+
142+
For more information about creating and running TensorFlow Lite models with
143+
Coral devices, read [TensorFlow modles on the Edge TPU](
144+
https://coral.withgoogle.com/docs/edgetpu/models-intro/).

‎lite/examples/object_detection/raspberry_pi/annotation.py‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
# python3
2+
#
3+
# Copyright 2019 The TensorFlow Authors. All Rights Reserved.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# https://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
"""An annotation library that draws overlays on the Pi camera preview.
17+
18+
Annotations include bounding boxes and text overlays.
19+
Annotations support partial opacity, however only with respect to the content in
20+
the preview. A transparent fill value will cover up previously drawn overlay
21+
under it, but not the camera content under it. A color of None can be given,
22+
which will then not cover up overlay content drawn under the region.
23+
Note: Overlays do not persist through to the storage layer so images saved from
24+
the camera, will not contain overlays.
25+
"""
26+
27+
from __future__ import absolute_import
28+
from __future__ import division
29+
from __future__ import print_function
30+
31+
from PIL import Image
32+
from PIL import ImageDraw
33+
34+
35+
def _round_up(value, n):
36+
"""Rounds up the given value to the next number divisible by n.
37+
38+
Args:
39+
value: int to be rounded up.
40+
n: the number that should be divisible into value.
41+
42+
Returns:
43+
the result of value rounded up to the next multiple of n.
44+
"""
45+
return n * ((value + (n - 1)) // n)
46+
47+
48+
def _round_buffer_dims(dims):
49+
"""Appropriately rounds the given dimensions for image overlaying.
50+
51+
As per the PiCamera.add_overlay documentation, the source data must have a
52+
width rounded up to the nearest multiple of 32, and the height rounded up to
53+
the nearest multiple of 16. This does that for the given image dimensions.
54+
55+
Args:
56+
dims: image dimensions.
57+
58+
Returns:
59+
the rounded-up dimensions in a tuple.
60+
"""
61+
width, height = dims
62+
return _round_up(width, 32), _round_up(height, 16)
63+
64+
65+
class Annotator:
66+
"""Utility for managing annotations on the camera preview."""
67+
68+
def __init__(self, camera, default_color=None):
69+
"""Initializes Annotator parameters.
70+
71+
Args:
72+
camera: picamera.PiCamera camera object to overlay on top of.
73+
default_color: PIL.ImageColor (with alpha) default for the drawn content.
74+
"""
75+
self._camera = camera
76+
self._dims = camera.resolution
77+
self._buffer_dims = _round_buffer_dims(self._dims)
78+
self._buffer = Image.new('RGBA', self._buffer_dims)
79+
self._overlay = None
80+
self._draw = ImageDraw.Draw(self._buffer)
81+
self._default_color = default_color or (0xFF, 0, 0, 0xFF)
82+
83+
def update(self):
84+
"""Draws any changes to the image buffer onto the overlay."""
85+
# For some reason, simply updating the current overlay causes
86+
# PiCameraMMALError every time we update. To avoid that, we create a new
87+
# overlay each time we want to update.
88+
# We use a temp overlay object because if we remove the current overlay
89+
# first, it causes flickering (the overlay visibly disappears for a moment).
90+
temp_overlay = self._camera.add_overlay(
91+
self._buffer.tobytes(), format='rgba', layer=3, size=self._buffer_dims)
92+
if self._overlay is not None:
93+
self._camera.remove_overlay(self._overlay)
94+
self._overlay = temp_overlay
95+
self._overlay.update(self._buffer.tobytes())
96+
97+
def clear(self):
98+
"""Clears the contents of the overlay, leaving only the plain background."""
99+
self._draw.rectangle((0, 0) + self._dims, fill=(0, 0, 0, 0x00))
100+
101+
def bounding_box(self, rect, outline=None, fill=None):
102+
"""Draws a bounding box around the specified rectangle.
103+
104+
Args:
105+
rect: (x1, y1, x2, y2) rectangle to be drawn, where (x1, y1) and (x2, y2)
106+
are opposite corners of the desired rectangle.
107+
outline: PIL.ImageColor with which to draw the outline (defaults to the
108+
Annotator default_color).
109+
fill: PIL.ImageColor with which to fill the rectangle (defaults to None,
110+
which will *not* cover up drawings under the region).
111+
"""
112+
outline = outline or self._default_color
113+
self._draw.rectangle(rect, fill=fill, outline=outline)
114+
115+
def text(self, location, text, color=None):
116+
"""Draws the given text at the given location.
117+
118+
Args:
119+
location: (x, y) point at which to draw the text (upper left corner).
120+
text: string to be drawn.
121+
color: PIL.ImageColor to draw the string in (defaults to the Annotator
122+
default_color).
123+
"""
124+
color = color or self._default_color
125+
self._draw.text(location, text, fill=color)

0 commit comments

Comments
 (0)