Building a Telepresence Robot
When building a robot you quickly work out that you have two choices with regards to controlling it: autonomous or some sort of remote control. We will develop both for Alexa M. We are going with remote control first because we are waiting for our ultrasonic mounting bracket to arrive from China.
As Alexa M has the Raspberry Pi camera fitted it makes sense to stream the video so we can have a view of what the robot is seeing. In effect a simple telepresence rover.
There are many different approaches for providing remote control to a robot (including wired, WiFi, Bluetooth, or RF). We wanted something wireless, with a Python API which could incorporate the video stream with minimal lag. That quickly narrowed things down and we chose control via WiFi.
Robot control via WiFi is pretty straight forward. You use a micro-framework like Bottle or Flask to set up the Pi as a web-server and then you can use your browser to access the associated web page. Well maybe it isn't that straight forward, but at least it is well documented. Streaming video to the same web page turned out to be a bit of a challenge - but not impossible. we were surprised that this wasn't a problem with an obvious solution given the numerous requests on the web for this functionality. The underlying issue seems to be that the Pi's camera outputs raw H.264, and what most browsers want is an MPEG transport stream. Given video was the tricky bit, we used this to decide which framework to use.
Video Streaming - The Options
The following is a list of the options that we came across when searching for a solution. No doubt there are many more, and if there are any we missed then let us know in the comments.
- picamera - was our first stop. It is s a pure Python interface to the Raspberry Pi camera module. Perfect! Except it doesn't do streaming. For anything else it is very good.
- RPi-Cam-Web-Interface - is a web interface for the Raspberry Pi Camera module that can be opened on any browser (smartphones included). Now we are cooking. Follow the link to install this on your Pi. It works very well, has zero lag and probably has the best video quality of the options we tried. However, server side coding, HTML, CSS and JavaScript are not an area of expertise so we need a pretty idiot proof guide to modding this. I'm sure you could add custom controls to the page served by RPi-Cam-Web-Interface but it wasn't obvious how to do this.
- bottle - is a fast, simple and lightweight WSGI micro web-framework for Python. It is distributed as a single file module and has no dependencies other than the Python Standard Library. The Raspberry Pi forums includes an example of how to stream video using bottle so this was definitely a contender. Electronut Labs provide a simple turn a LED on/off using bottle tutorial as well.
- flask - is another lightweight WSGI micro web-framework for Python. It is similar to bottle and you would probably choose flask over bottle if you had a more complicated application (over 1000 lines appears to be the consensus). Miguel has a tutorial on streaming video with flask and there is another guide provided by CCTV camera pros for the Raspberry Pi. Either flask or bottle would get the job done.
- Cayenne - helps you build a drag and drop web based dashboard for your IoT applications (i.e. Arduino and Raspberry Pi). It is pretty fancy but it cant do video streaming (yet).
- UV4L - was originally conceived as a modular collection of Video4Linux2-compliant, cross-platform drivers. It has evolved over the years and now includes a full-featured Streaming Server component. There is a module for single or dual Raspberry Pi CSI Camera boards but it is command line based and we would prefer a python API. At this stage there are easier options.
- pistreaming - provides low latency streaming of the Pi's camera module to any reasonably modern web browser. This is written by the same guy that did the picamera module, all the source code is provided and most importantly it is documented well enough for us to be able to modify the served page to do what we require. The video isn't as good as RPi-Cam-Web-Interface but there is no lag on our LAN. This is the option we ended up using.
PiStreaming
To get the pistreaming solution to work you will need 3 files:
- index.html - the html code for the page that you are serving;
- server.py - the python code which serves up the video stream; and
- jsmpg.js - Dominic Szablewski's Javascript-based MPEG1 decoder.
These can all be cloned from the
pistreaming repository. As a first step install the code by following the instructions at
pistreaming. Once you have that up and working you can tweak it for your purposes.
RS_Server - a Video Streaming Python Class
To make streaming compatible with our robot class we have turned server.py into a server class. We have made a few other tweaks like inverting the camera since ours is mounted upside down. The print(server) command will display the URL where you can view the stream. The Server class is designed to be imported into another class and usage should be obvious from the class documentation and instructions at pistreaming.
We have also changed the index.html file in preparation for controlling the robot via the website, but we will cover this in a subsequent post.
#!/usr/bin/env python
# RS_Server.py - Web Server Class for the Raspberry Pi
#
# Based on server.py from pistreaming
# ref: https://github.com/waveform80/pistreaming
# Copyright 2014 Dave Hughes <dave@waveform.org.uk>
#
# 06 March 2017 - 1.0 Original Issue
#
# Reefwing Software
# Simplified BSD Licence - see bottom of file.
import sys, io, os, shutil, picamera, signal
from subprocess import Popen, PIPE, check_output
from string import Template
from struct import Struct
from threading import Thread
from time import sleep, time
from http.server import HTTPServer, BaseHTTPRequestHandler
from wsgiref.simple_server import make_server
from ws4py.websocket import WebSocket
from ws4py.server.wsgirefserver import WSGIServer, WebSocketWSGIRequestHandler
from ws4py.server.wsgiutils import WebSocketWSGIApplication
###########################################
# CONFIGURATION
WIDTH = 640
HEIGHT = 480
FRAMERATE = 24
HTTP_PORT = 8082
WS_PORT = 8084
COLOR = u'#444'
BGCOLOR = u'#333'
JSMPEG_MAGIC = b'jsmp'
JSMPEG_HEADER = Struct('>4sHH')
###########################################
class StreamingHttpHandler(BaseHTTPRequestHandler):
def do_HEAD(self):
self.do_GET()
def do_GET(self):
if self.path == '/':
self.send_response(301)
self.send_header('Location', '/index.html')
self.end_headers()
return
elif self.path == '/jsmpg.js':
content_type = 'application/javascript'
content = self.server.jsmpg_content
elif self.path == '/index.html':
content_type = 'text/html; charset=utf-8'
tpl = Template(self.server.index_template)
content = tpl.safe_substitute(dict(
ADDRESS='%s:%d' % (self.request.getsockname()[0], WS_PORT),
WIDTH=WIDTH, HEIGHT=HEIGHT, COLOR=COLOR, BGCOLOR=BGCOLOR))
else:
self.send_error(404, 'File not found')
return
content = content.encode('utf-8')
self.send_response(200)
self.send_header('Content-Type', content_type)
self.send_header('Content-Length', len(content))
self.send_header('Last-Modified', self.date_time_string(time()))
self.end_headers()
if self.command == 'GET':
self.wfile.write(content)
class StreamingHttpServer(HTTPServer):
def __init__(self):
super(StreamingHttpServer, self).__init__(
('', HTTP_PORT), StreamingHttpHandler)
with io.open('index.html', 'r') as f:
self.index_template = f.read()
with io.open('jsmpg.js', 'r') as f:
self.jsmpg_content = f.read()
class StreamingWebSocket(WebSocket):
def opened(self):
self.send(JSMPEG_HEADER.pack(JSMPEG_MAGIC, WIDTH, HEIGHT), binary=True)
class BroadcastOutput(object):
def __init__(self, camera):
print('Spawning background conversion process')
self.converter = Popen([
'avconv',
'-f', 'rawvideo',
'-pix_fmt', 'yuv420p',
'-s', '%dx%d' % camera.resolution,
'-r', str(float(camera.framerate)),
'-i', '-',
'-f', 'mpeg1video',
'-b', '800k',
'-r', str(float(camera.framerate)),
'-'],
stdin=PIPE, stdout=PIPE, stderr=io.open(os.devnull, 'wb'),
shell=False, close_fds=True)
def write(self, b):
self.converter.stdin.write(b)
def flush(self):
print('Waiting for background conversion process to exit')
self.converter.stdin.close()
self.converter.wait()
class BroadcastThread(Thread):
def __init__(self, converter, websocket_server):
super(BroadcastThread, self).__init__()
self.converter = converter
self.websocket_server = websocket_server
def run(self):
try:
while True:
buf = self.converter.stdout.read(512)
if buf:
self.websocket_server.manager.broadcast(buf, binary=True)
elif self.converter.poll() is not None:
break
finally:
self.converter.stdout.close()
class Server():
def __init__(self):
# Create a new server instance
print("Initializing camera")
self.camera = picamera.PiCamera()
self.camera.resolution = (WIDTH, HEIGHT)
self.camera.framerate = FRAMERATE
# hflip and vflip depends on how you mount the camera
self.camera.vflip = True
self.camera.hflip = False
sleep(1) # camera warm-up time
print("Camera ready")
def __str__(self):
# Return string representation of server
ip_addr = check_output(['hostname', '-I']).decode().strip()
return "Server video stream at http://{}:{}".format(ip_addr, HTTP_PORT)
def start(self):
# Start video server streaming
print('Initializing websockets server on port %d' % WS_PORT)
self.websocket_server = make_server(
'', WS_PORT,
server_class=WSGIServer,
handler_class=WebSocketWSGIRequestHandler,
app=WebSocketWSGIApplication(handler_cls=StreamingWebSocket))
self.websocket_server.initialize_websockets_manager()
self.websocket_thread = Thread(target=self.websocket_server.serve_forever)
print('Initializing HTTP server on port %d' % HTTP_PORT)
self.http_server = StreamingHttpServer()
self.http_thread = Thread(target=self.http_server.serve_forever)
print('Initializing broadcast thread')
output = BroadcastOutput(self.camera)
self.broadcast_thread = BroadcastThread(output.converter, self.websocket_server)
print('Starting recording')
self.camera.start_recording(output, 'yuv')
print('Starting websockets thread')
self.websocket_thread.start()
print('Starting HTTP server thread')
self.http_thread.start()
print('Starting broadcast thread')
self.broadcast_thread.start()
print("Video Stream available...")
while True:
self.camera.wait_recording(1)
def cleanup(self):
# Stop video server - close browser tab before calling cleanup
print('Stopping recording')
self.camera.stop_recording()
print('Waiting for broadcast thread to finish')
self.broadcast_thread.join()
print('Shutting down HTTP server')
self.http_server.shutdown()
print('Shutting down websockets server')
self.websocket_server.shutdown()
print('Waiting for HTTP server thread to finish')
self.http_thread.join()
print('Waiting for websockets thread to finish')
self.websocket_thread.join()
def main():
server = Server()
print(server)
def endProcess(signum = None, frame = None):
# Called on process termination.
if signum is not None:
SIGNAL_NAMES_DICT = dict((getattr(signal, n), n) for n in dir(signal) if n.startswith('SIG') and '_' not in n )
print("signal {} received by process with PID {}".format(SIGNAL_NAMES_DICT[signum], os.getpid()))
print("\n-- Terminating program --")
print("Cleaning up Server...")
server.cleanup()
print("Done.")
exit(0)
# Assign handler for process exit
signal.signal(signal.SIGTERM, endProcess)
signal.signal(signal.SIGINT, endProcess)
signal.signal(signal.SIGHUP, endProcess)
signal.signal(signal.SIGQUIT, endProcess)
server.start()
if __name__ == '__main__':
main()
## Copyright (c) 2017, Reefwing Software
## All rights reserved.
##
## Redistribution and use in source and binary forms, with or without
## modification, are permitted provided that the following conditions are met:
##
## 1. Redistributions of source code must retain the above copyright notice, this
## list of conditions and the following disclaimer.
## 2. Redistributions in binary form must reproduce the above copyright notice,
## this list of conditions and the following disclaimer in the documentation
## and/or other materials provided with the distribution.
##
## THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
## ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
## WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
## DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
## ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
## (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
## LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
## ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
## (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
## SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
