Chapter 9 Get on the Road!

Use two terminals for raspberry

first teminal run jgp streamer

cd /home/pi/Sunfounder_Smart_Video_Car_Kit_for_RaspberryPi/mjpg-streamer/mjpg-streamer
sudo sh start.sh

second terminal run tcp_server.py:

cd ~/Sunfounder_Smart_Video_Car_Kit_for_RaspberryPi/server
sudo python tcp_server.py

on mac terminal in environment sunfounderPy27

Type in the following address at the address bar of your browser (Firefox is recommended):

http://192.168.178.67:8080/stream.html
source activate sunfounderPy27

or if openCV is needed

source activate carnd-term1 

9.1 install carnd-term1

git clone https://github.com/udacity/CarND-Term1-Starter-Kit.git
cd CarND-Term1-Starter-Kit
conda env create -f environment.yml
conda install -c anaconda tk 
 cd /Users/uwesterr/CloudProjectsUnderWork/ProjectsUnderWork/RoboCar/sunfounder/Sunfounder_Smart_Video_Car_Kit_for_RaspberryPi/client 
 sudo python client_App.py

9.2 Install Xbox 360 controller

install Xbox 360 controller drive on mac https://github.com/360Controller/360Controller/releases

NOTE!!!
it seems necessary to have the controller plugged into the USB port during boot up

According to http://www.philipzucker.com/python-xbox-controller-mac/ Install pygame

python -m pip install -U pygame --user

or

brew upgrade sdl sdl_image sdl_mixer sdl_ttf portmidi
python3.6 -m venv anenv
. ./anenv/bin/activate
pip install https://github.com/pygame/pygame/archive/master.zip

create a jupyter notebook “xboxControllerOnMac.ipynb”" with

import pygame
import sys
import time
import socket
import cPickle as pickle
 
UDP_IP = "127.0.0.1"
UDP_PORT = 5005
MESSAGE = "Hello, World!"
 
print "UDP target IP:", UDP_IP
print "UDP target port:", UDP_PORT
print "message:", MESSAGE
 
sock = socket.socket(socket.AF_INET, # Internet
                     socket.SOCK_DGRAM) # UDP
 
 
 
pygame.init()
 
pygame.joystick.init()
clock = pygame.time.Clock()
 
print pygame.joystick.get_count()
_joystick = pygame.joystick.Joystick(0)
_joystick.init()

gives:

UDP target IP: 127.0.0.1
UDP target port: 5005
message: Hello, World!
1

watch out for the “1” which indicates that the controller was identified

while 1:
    for event in pygame.event.get():
        if event.type == pygame.JOYBUTTONDOWN:
            print("Joystick button pressed.")
            print event
        if event.type == pygame.JOYAXISMOTION:
            #print _joystick.get_axis(0)
            #print event
            if event.axis == 0: # this is the x axis
                print event.value
            if event.axis == 5: # right trigger
                print event.value
    xdir = _joystick.get_axis(0)
 
    rtrigger = _joystick.get_axis(5)
    #deadzone
    if abs(xdir) < 0.2:
        xdir = 0.0
    if rtrigger < -0.9:
        rtrigger = -1.0
 
    MESSAGE = pickle.dumps([xdir,rtrigger])
    sock.sendto(MESSAGE, (UDP_IP, UDP_PORT))
 
    clock.tick(30)

when using controller the following output was created

0.00781273842586
-1.00003051851
Joystick button pressed.
<Event(10-JoyButtonDown {‘joy’: 0, ‘button’: 6})>
0.0
0.00781273842586
Joystick button pressed.
<Event(10-JoyButtonDown {‘joy’: 0, ‘button’: 12})>
0.0312509537034

9.3 Implement Xbox controller as input in client_App.py

In client_App.py the part for driving forward extracted

from Tkinter import *
from socket import *      # Import necessary modules

ctrl_cmd = ['forward', 'backward', 'left', 'right', 'stop', 'read cpu_temp', 'home', 'distance', 'x+', 'x-', 'y+', 'y-', 'xy_home']

top = Tk()   # Create a top window
top.title('Sunfounder Raspberry Pi Smart Video Car')

HOST = '192.168.178.67'    # Server(Raspberry Pi) IP address
PORT = 21567
BUFSIZ = 1024             # buffer size
ADDR = (HOST, PORT)

tcpCliSock = socket(AF_INET, SOCK_STREAM)   # Create a socket
tcpCliSock.connect(ADDR)                    # Connect with the server

# =============================================================================
# The function is to send the command forward to the server, so as to make the 
# car move forward.
# ============================================================================= 
def forward_fun(event):
    print 'forward'
    tcpCliSock.send('forward')

then keystrokes are binded to the forward function, this needs to be changed to bind Xbox controller values to the function

# =============================================================================
# Bind buttons on the keyboard with the corresponding callback function to 
# control the car remotely with the keyboard.
# =============================================================================
top.bind('<KeyPress-w>', forward_fun)   # Press down key 'w' on the keyboard and the car will drive forward.

from https://github.com/martinohanlon/XboxController/blob/master/XboxController.py

JOYAXISMOTION
event.axis event.value
0 - x axis left thumb (+1 is right, -1 is left)
1 - y axis left thumb (+1 is down, -1 is up)
2 - x axis right thumb (+1 is right, -1 is left)
3 - y axis right thumb (+1 is down, -1 is up)
4 - right trigger
5 - left trigger

JOYBUTTONDOWN | JOYBUTTONUP
event.button
A = 0
B = 1
X = 2
Y = 3
LB = 4
RB = 5
BACK = 6
START = 7
XBOX = 8
LEFTTHUMB = 9
RIGHTTHUMB = 10

9.3.1 Make steering proportional to remote control lever position

change code from

if event.axis == 0: # this is the x axis
        if event.value > thresSteerHigh:
           tcpCliSock.send('right')
      if event.value < thresSteerLow:
           tcpCliSock.send('left') 

to

if event.axis == 0: # this is the x axis
        if event.value > thresSteerHigh:
       tcpCliSock.send('right')
         angle = int(100*abs(event.value))
       data = tmp1 + str(angle)  
             print 'sendData = %s' % data
             tcpCliSock.send(data)  # Send the speed data to the server(Raspberry Pi)
        if event.value < thresSteerLow:
           tcpCliSock.send('left') 
           angle = int(-100*abs(event.value))
             data = tmp1 + str(angle)  
             print 'sendData = %s' % data
             tcpCliSock.send(data)  # Send the speed data to the server(Raspberry Pi)

9.4 Get IP adress of raspi in shack

Check in the router for the IP adress, procedure is dependent on router At shackspace go to http://leases.shack/#/ (only available from the shackspace network)

and then connect via ssh

if you get

Uwes-MBP:data uwesterr$ ssh pi@10.42.26.33
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:aYMRAzv3GpxqJNugz1oTi20m0QKVIVfVxszQkJNbNqg.
Please contact your system administrator.
Add correct host key in /Users/uwesterr/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/uwesterr/.ssh/known_hosts:20
ECDSA host key for 10.42.26.33 has changed and you have requested strict checking.
Host key verification failed.

then you have to remove the cached key for donkeypi-uwe (or the old IP adress) on the local machine:

ssh-keygen -R donkeypi-uwe

9.5 Store snapshots of video stream on local computer

Donkey car stores training data on raspi SD card. In this concept the XBox controller is connected via USB to a laptop (in my case a Mac) and it makes sense to store the traing data on the laptop as well since we anyway will train the NN on that machine.

The URL of the stream is:

url = “http://10.42.26.33:8080/?action=stream

for a snapshot the URL is url = “http://10.42.26.33:8080/?action=snapshot

check video https://youtu.be/2xcUzXataIk?t=556 for a good explanation of how to receive an IP video stream. Based on that tutorial the follwing code now stores a single frame and shows that frame as well.

import cv2
import numpy as np
import urllib
 
# based on example in https://www.youtube.com/watch?v=2xcUzXataIk 
url = "http://192.168.178.67:8080/?action=snapshot"  
imgNp = np.array(bytearray(imgResp.read()), dtype = np.uint8)
img = cv2.imdecode(imgNp,-1)
cv2.imshow("test",img)
cv2.imwrite( "Snapshot.jpg", img );
cv2.waitKey(10000)

9.5.1 Jie Hou’s alternative

Jie has another solution for the same task, see https://drive.google.com/drive/folders/10U8ZTr_2HVnBWrFqvVFGO0UyQn0vCmiE

The code is

import cv2
import numpy as np
try:
    from urllib.request import urlopen
except ImportError:
    from urllib2 import urlopen


print('# capture image from video #')
stream = urlopen('http://192.168.0.101:8080/?action=stream')
bytes = bytes()
FlagSaveImage = 0
while True:
    bytes += stream.read(1024)
    a = bytes.find(b'\xff\xd8')
    b = bytes.find(b'\xff\xd9')
    print(' #a: ', a, ' ,b: ', b)
    if a != -1 and b != -1:
        jpg = bytes[a: b + 2]
        bytes = bytes[b+2:]
        image = cv2.imdecode(np.fromstring(jpg, dtype=np.uint8), cv2.IMREAD_COLOR)
        cv2.imshow('image', image)
        if FlagSaveImage == 0:
            cv2.imwrite('test.jpg', image)
            FlagSaveImage = 1
        if cv2.waitKey(1) == 27:
            exit(0)