track-stereo-depth

x64 Systems

Installation - Anaconda + Pip

Create new environment:

conda create --name rs python=3.8
conda activate rs

Realsense install requirements:

pip install opencv-python pyrealsense2 python-dotenv

Detectron2 install requirements (tested on CUDA 11.1 + PyTorch v1.9):

1. Install PyTorch CPU or GPU version with pip or conda - https://pytorch.org/
2. Install appropriate detectron based on CUDA and PyTorch version - https://detectron2.readthedocs.io/en/latest/tutorials/install.html

Live Capture/Recording

• capture/measure_stream.py - demo for measuring distances on a live stream.
• capture/record_bag.py - records depth and colour streams into .bag file format (at full fps these become very large very quickly!).
• capture/keypoint_detection_measure.py - reads and aligns .bag colour and depth streams, runs keypoint detection, and measures 3D distance between two keypoints. (Set live_stream to true to measure from stream).
• capture/read_bag.py - aligns .bag colour and depth streams, and converts subsequent colour stream to .mp4 in preparation for keypoint detection.
• capture/detectron_keypoints_video.py - runs keypoint detection on .mp4 file. Outputs .json file of all detections. (NOTE - if "xcb" type error occurs, disable cv2.imshow and cv2.imwrite functions)
• capture/measure_keypoints_3d.py - reads .bag, aligns streams, reads .json file, and measures the 3d distances between keypoints.

RaspberryPi Data Processing Pipeline

1. preprocessing/unencrypt.py
2. preprocessing/csv_timeline.py - Produces a CSV file of when frames were captured.
3. preprocessing/capture_timeline.py - Produces a graph showing when frames were captured.
4. preprocessing/image_timelapse.py - Produces a timelapse video of images.
5. processing/detections.py - Produces CSV file of keypoint detections, occupation count, and distance measurements between detections.
6. plotting/occupation.py

RaspberryPi 4

For running capture remotely

Setup and Installation

Firstly prepare RaspberryPi and ssh using the following steps- https://ubuntu.com/tutorials/how-to-install-ubuntu-on-your-raspberry-pi#1-overview

Update and Install Dependencies:

$ sudo apt-get update && sudo apt-get upgrade
$ apt-get install -y python3 python3-dev python3-pip git libusb-1.0-0-dev usbutils libgl1 libsm6 libxext6 libxrender-dev libfontconfig1 libssl-dev xorg-dev libglu1-mesa-dev vim
$ sudo apt get cmake

Create and activate swap file:

$ sudo fallocate -l 4G /swapfile
$ sudo chmod 600 /swapfile
$ sudo mkswap /swapfile
$ sudo swapon /swapfile

Then add /swapfile swap swap defaults 0 0 to newline using $ sudo vim /etc/fstab

Clone Intel Realsense Repository:

$ git clone https://github.com/IntelRealSense/librealsense.git
$ cd librealsense
$ mkdir build

Build pyrealsense from source with python wrapper:

$ cd build
$ cmake ../ -DFORCE_RSUSB_BACKEND=true -DBUILD_PYTHON_BINDINGS=bool:true -DCMAKE_BUILD_TYPE=release -DBUILD_EXAMPLES=true -DBUILD_GRAPHICAL_EXAMPLES=true
$ make -j4
$ sudo make install

Build OpenCV:

$ cd
$ wget https://github.com/Qengineering/Install-OpenCV-Raspberry-Pi-64-bits/raw/main/OpenCV-4-5-5.sh
$ sudo chmod 755 ./OpenCV-4-5-5.sh
$ ./OpenCV-4-5-5.sh

Install pandas:

$ sudo pip3 install pandas

Running

Run python scripts using sudo:

$ sudo python3 RaspberryPi/run.py

Notes

• Change all python scripts import pyrealsense2 as rs to import pyrealsense2.pyrealsense2 as rs
• .bag files record at ~1.024MB/frame (30.72MB/s) at 30fps 640x480 (huge - 1.6GB+/minute)
• Ensure external usb drive is formatted as Ext4
• Install Real Time Clock module if not connected to internet

Resources

• https://intelrealsense.github.io/librealsense/doxygen/annotated.html
• https://github.com/IntelRealSense/librealsense/tree/master/wrappers/python#examples
• https://qengineering.eu/install-opencv-4.5-on-raspberry-64-os.html
• https://www.dexterindustries.com/howto/run-a-program-on-your-raspberry-pi-at-startup/
• https://medium.com/@yasuhirachiba/converting-2d-image-coordinates-to-3d-coordinates-using-ros-intel-realsense-d435-kinect-88621e8e733a
• https://www.mouser.com/pdfdocs/Intel_D400_Series_Datasheet.pdf
• realsenseai/librealsense#3473

Advanced Setup

Setting a static IP and connecting via ethernet:

• https://linuxize.com/post/how-to-configure-static-ip-address-on-ubuntu-20-04/
• https://nrsyed.com/2021/06/02/raspberry-pi-headless-setup-via-ethernet-and-how-to-share-the-host-pcs-internet-connection/

1. Disabled cloud-init by creating file /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg and putting in network: {config: disabled}
2. In /etc/netplan/50-cloud-init.yaml network:ethernets:eth0:
   • changed dhcp4: from true to false
   • removed optional:true
   • Added addresses: - 192.168.4.2/24 gateway4: 192.168.4.1 nameservers: addresses: [8.8.8.8, 1.1.1.1]
3. $ sudo netplan apply
4. Check if it's changed - $ ip addr show dev eth0
5. On host ubuntu (laptop) - $ nm-connection-editor
   • add a new ethernet connection
   • Input IPv4 Settings:
     • "Shared to other computers"
     • Address 192.168.4.1
     • Netmask 24
     • Gateway 192.168.4.1
     • NAME
     • Save and exit
6. Prevent from connecting thrrough internet from here - $ nmcli connection modify NAME ipv4.never-default true
7. Connect - $ ssh ubuntu@192.168.4.2

Cloning SD card:

• https://www.pragmaticlinux.com/2020/12/how-to-clone-your-raspberry-pi-sd-card-in-linux/
• Use raspberry Pi imager to write to new SD card

Real time clock:

if using ds3231 chipset also use 'ds1307' as outlined in these instructions

• https://forums.raspberrypi.com/viewtopic.php?t=330201

1. Enable i2c for ubuntu server
2. Check that ports GPIO pins are being used -> sudo i2cdetect -y 1. This should give a 68.
3. $ sudo modprobe rtc-ds1307
4. Enter bash $ sudo bash and enter $ echo ds1307 0x68 > /sys/class/i2c-adapter/i2c-1/new_device and then exit
5. Check sudo i2cdetect -y 1 has changed from 68 to UU.
6. Write internet time to device $ sudo hwclock -w
7. Check to see if RTC is working $ sudo hwclock -r
8. Enable at boot:

   • Navigate to /etc/modules-load.d. Edit the modules.conf file and add rtc-1307

   • In /etc/rc.local add: echo ds1307 0x68 > /sys/class/i2c-adapter/i2c-1/new_device sudo hwclock -s date

   • In /etc/systemd/system/ create file called rc-local.service and add: [Unit] Description=/etc/rc.local Compatibility ConditionPathExists=/etc/rc.local

     [Service] Type=forking ExecStart=/etc/rc.local start TimeoutSec=0 StandardOutput=tty RemainAfterExit=yes SysVStartPriority=99

     [Install] WantedBy=multi-user.target