A computer-vision pipeline that detects a rectangular object on a conveyor belt, extracts its position (x, y) and orientation (angle), and sends the coordinates to a Universal Robots (UR) manipulator via RTDE (Real-Time Data Exchange). Designed as the "eyes" of a robotic pick-and-place system.
- Overview
- Project Structure
- Architecture
- How It Works — Step by Step
- Getting Started
- Configuration
- Calibration Steps (When Hardware Is Ready)
- What Has Been Done
- What Still Needs to Be Done
The Flying Picker is a robotic pick-and-place system. A camera watches a conveyor belt, detects objects, and sends their position + angle to a robot so it can pick them mid-flight.
| # | Item | Purpose | Est. Cost |
|---|---|---|---|
| 1 | Raspberry Pi 5 (8 GB RAM) | Compute — runs the vision pipeline | €99 |
| 2 | Official USB-C Power Supply (27 W) | Power — Pi 5 requires the official 27 W PD supply | €20.20 |
| 3 | Active Cooler for Pi 5 | Cooling — prevents thermal throttling during vision processing | €10.10 |
| 4 | Raspberry Pi Global Shutter Camera (Sony IMX296) | The Eye — global shutter eliminates motion blur / jelly effect on moving conveyor | €85 |
| 5 | 6 mm CS-Mount Lens | The Optic — wide-angle lens to see the full belt width | €30 |
| 6 | Camera Cable (Mini → Standard) | Pi 5 uses a Mini CSI connector; adapter cable needed | €3 |
| 7 | MicroSD Card (32 GB, A2 Class) | Storage — fast boot + program loading | €25.30 |
| 8 | LED Strip (12 V Cool White) | Lighting — consistent illumination for reliable detection | €12.99 |
| Total | €285.59 |
Currently the vision system runs on a pre-recorded video (public/IMG_6256.MOV) of a light-coloured square card on a dark surface. No camera hardware is needed yet — the architecture is designed so the Pi Global Shutter Camera can be swapped in later (via picamera2) without changing the detection logic.
flying-picker-v1/
│
├── run_vision.py # ← Entry point — run the detection pipeline
├── run_camera_calibration.py # Calibrate lens distortion (checkerboard)
├── run_homography_calibration.py # Calibrate pixel → world mm transform
│
├── config/
│ ├── vision_config.yaml # All tunable parameters (thresholds, colors, paths)
│ ├── camera_calibration.json # (auto-generated by run_camera_calibration.py)
│ └── homography.json # (auto-generated by run_homography_calibration.py)
│
├── src/
│ ├── vision/
│ │ ├── __init__.py # Package exports
│ │ ├── frame_source.py # FrameSource class — reads frames from video/camera
│ │ ├── preprocess.py # Grayscale → blur → threshold → morphology
│ │ ├── detection.py # Contour detection → minAreaRect → (x, y, angle)
│ │ ├── detection_tracker.py # Single-send debounce — one coordinate set per part
│ │ ├── calibration.py # Camera intrinsic calibration (undistortion)
│ │ ├── coordinate_transform.py # Pixel→world homography + belt compensation
│ │ ├── RTDEsender.py # RTDE communication with UR robot
│ │ └── pipeline.py # Main loop — ties everything together + live overlay
│ │
│ └── robot/ # (empty — future robot communication module)
│
├── tests/ # (empty — future unit/integration tests)
│
├── public/
│ ├── IMG_6256.MOV # Sample video of the object on a surface
│ ├── 01_Flying picker.pdf # Project overview document
│ ├── Flying Picker research paper.pdf
│ └── Vision_System_BOM.pdf # Bill of Materials for vision hardware
│
└── requirements.txt # Python dependencies
┌──────────────────────────────────────────────────────────────────┐
│ run_vision.py │
│ (entry point) │
└──────────────────────┬───────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────┐
│ pipeline.py │
│ (main loop — orchestrator) │
│ │
│ Loads config + calibration files (if they exist) │
│ Opens FrameSource ──→ Loop: │
│ │ │
│ ┌──────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────┐ ┌──────────────┐ ┌──────────────────┐ │
│ │ FrameSource │──▶│ undistort() │──▶│ ROI crop │ │
│ │ │ │ (if calib │ │ (configurable │ │
│ │ Read next │ │ exists) │ │ edge removal) │ │
│ │ video frame │ └──────────────┘ └───────┬──────────┘ │
│ │ (loops at │ │ │
│ │ end of │ ▼ │
│ │ video) │ ┌──────────────┐ ┌──────────────────┐ │
│ └─────────────┘ │ preprocess() │──▶│ detect_object() │ │
│ │ │ │ │ │
│ │ BGR→Gray │ │ Find contours │ │
│ │ Gaussian blur│ │ Largest contour │ │
│ │ Binary thresh│ │ minAreaRect() │ │
│ │ Morph close │ │ → (x, y, θ) px │ │
│ └──────────────┘ └───────┬──────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────┐ ┌──────────────────┐ │
│ │ compensate │◀──│ pixel_to_world() │ │
│ │ _belt │ │ (if homography │ │
│ │ _motion() │ │ exists) │ │
│ │ (if belt │ │ → (X, Y, θ) mm │ │
│ │ enabled) │ └──────────────────┘ │
│ │ → pick pos │ │
│ └──────┬───────┘ │
│ │ │
│ ▼ │
│ ┌───────────────────────────┐ │
│ │ DetectionTracker │ │
│ │ • Debounce (3 frames) │ │
│ │ • Send once per part │ │
│ │ • Distance-based new part │ │
│ └────────────┬──────────────┘ │
│ │ should_send=True (once) │
│ ▼ │
│ ┌───────────────────────────┐ │
│ │ RTDEsender │ │
│ │ → Send (X, Y, θ) to robot │ │
│ └───────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌────────────────┐ │
│ │ Draw overlay │ │
│ │ • Green box │ │
│ │ • Red centroid │ │
│ │ • mm or px text│ │
│ │ cv2.imshow() │ │
│ └────────────────┘ │
└──────────────────────────────────────────────────────────────────┘
▲ ▲ ▲
│ config │ auto-loaded │ auto-loaded
│ │ if file exists │ if file exists
┌──────┴───────┐ ┌──────────────┴────────┐ ┌─────────┴──────────┐
│ vision_ │ │ camera_calibration │ │ homography.json │
│ config.yaml │ │ .json │ │ │
│ │ │ │ │ Pixel → mm │
│ thresholds │ │ Camera matrix + │ │ transform matrix │
│ ROI, belt │ │ distortion coeffs │ │ │
│ display opts │ │ (from checkerboard) │ │ (from ref points) │
└──────────────┘ └───────────────────────┘ └────────────────────┘
The pipeline gracefully degrades — without calibration files it works in pixel mode (exactly like before). Drop in camera_calibration.json → undistortion kicks in. Drop in homography.json → output switches to mm. Enable belt config → pick-position prediction appears.
FrameSource wraps OpenCV's VideoCapture. It opens the video file, reads one frame at a time, and loops back to frame 0 when the video ends (so you get a continuous stream for testing).
Video file (.MOV) ──▶ FrameSource.read() ──▶ BGR frame (numpy array)
Each raw BGR frame goes through four steps to produce a clean binary mask:
| Step | Function | Why |
|---|---|---|
| Grayscale | cv2.cvtColor(BGR2GRAY) |
Reduce 3 channels to 1 |
| Gaussian Blur | cv2.GaussianBlur(kernel=5) |
Remove noise / smooth edges |
| Binary Threshold | cv2.threshold(150) |
Light card → white (255), dark background → black (0) |
| Morphological Close | cv2.morphologyEx(MORPH_CLOSE) |
Fill small holes/gaps inside the object |
Input: full-colour frame → Output: binary mask (white object on black)
The binary mask is analysed to find the object's pose:
| Step | Function | Output |
|---|---|---|
| Find all contours | cv2.findContours() |
List of contour point arrays |
| Pick the largest | max(contours, key=cv2.contourArea) |
Single contour |
| Filter by min area | Area check (default 5000 px²) | Rejects noise |
| Fit rotated rectangle | cv2.minAreaRect() |
Center (x, y), size (w, h), angle (θ) |
| Get box corners | cv2.boxPoints() |
4 corner coordinates for drawing |
Returns a DetectionResult dataclass:
DetectionResult(
center_x, # centroid X in pixels
center_y, # centroid Y in pixels
width, # bounding box width
height, # bounding box height
angle, # rotation in degrees
contour, # raw contour points
box_points # 4 corners of the rotated box
)
Raw detections are jittery — the same part may be detected, lost for a frame, then detected again. Without tracking, this would send duplicate coordinates to the robot. The DetectionTracker solves this with a frame-based state machine:
IDLE → (3 consecutive detections) → CONFIRMING
CONFIRMING → (confirmed) → SENT (sends averaged coordinates ONCE)
SENT → (5 consecutive no-detections) → IDLE
| Parameter | Default | Purpose |
|---|---|---|
confirm_frames |
3 | Consecutive detection frames required before sending (filters jitter) |
exit_frames |
5 | Consecutive no-detection frames required before resetting (absorbs dropouts) |
distance_threshold_mm |
20 | If a new detection jumps more than this from the locked position, treat it as a new part |
All parameters are configurable in vision_config.yaml under tracking:.
The Sender class uses the RTDE (Real-Time Data Exchange) protocol to send coordinates to a Universal Robots manipulator. It sends:
| Register | Value |
|---|---|
input_double_register_0 |
X position (metres) |
input_double_register_1 |
Y position (metres) |
input_double_register_2 |
Angle (radians) |
input_double_register_3 |
Valid flag (1.0 = object, 0.0 = no object) |
Unit conversion (mm→m, deg→rad) happens inside send_pose() so the rest of the pipeline stays in mm/degrees.
The pipeline draws on the original frame and shows it:
- Green rotated bounding box — the detected rectangle outline
- Red dot — centroid (x, y)
- White text —
x=... y=... angle=...°
Two windows are shown:
- "Flying Picker — Detection" — original frame with overlay
- "Flying Picker — Mask" — the binary preprocessing result
Console output each frame:
[frame 42] px=(330,587) world=(280.7,-1226.1) mm angle=2.3 deg conf=0.93 (5.4 ms)
[SEND] Pose sent to robot: x=280.7 mm, y=-1226.1 mm, angle=2.3 deg
- Python 3.10+
- macOS / Linux / Windows
cd flying-picker-v1
# Create virtual environment (if not already done)
python -m venv .venv
source .venv/bin/activate # macOS/Linux
# .venv\Scripts\activate # Windows
# Install dependencies
pip install -r requirements.txtpython run_vision.pyTwo OpenCV windows will open showing the live detection. Press q to quit.
python run_vision.py path/to/custom_config.yaml# Step 1: Camera calibration (lens distortion) from saved checkerboard images
python run_camera_calibration.py --dir path/to/checkerboard_photos/ --board-cols 9 --board-rows 6
# ... or live capture (press SPACE to grab each image)
python run_camera_calibration.py --source 0 --num-captures 15
# Step 2: Homography calibration (pixel → mm) — interactive click mode
python run_homography_calibration.py --interactive --source 0 --world-points "0,0 400,0 400,300 0,300"
# ... or from known pixel/world point pairs
python run_homography_calibration.py \
--pixel-points "100,50 500,50 500,400 100,400" \
--world-points "0,0 400,0 400,300 0,300"Once calibration files exist in config/, the pipeline will automatically use them.
All tunable parameters are in config/vision_config.yaml:
input:
video_path: "public/IMG_6256.MOV" # Video source
preprocess:
blur_kernel_size: 5 # Gaussian blur kernel (must be odd)
threshold_value: 100 # Binary threshold cutoff (0–255)
threshold_max: 255 # Value for pixels above threshold
detection:
min_contour_area: 5000 # Ignore contours smaller than this (px²)
roi: # Crop frame edges to remove noise
enabled: true
x_start: 0.05
y_start: 0.05
x_end: 0.95
y_end: 0.90
calibration: # Checkerboard params for camera calibration
board_cols: 9
board_rows: 6
square_size_mm: 25.0
result_file: "config/camera_calibration.json"
homography: # Pixel → world mm transform
result_file: "config/homography.json"
belt: # Conveyor compensation
enabled: false
speed_mm_s: 0.0
detection_to_pick_delay_s: 0.0
direction: "x" # "x" or "y"
tracking: # Detection debounce (single-send per part)
confirm_frames: 3 # Consecutive detections before sending
exit_frames: 5 # Consecutive no-detections before resetting
distance_threshold_mm: 20 # Distance jump (mm) to consider a "new part"
display:
show_windows: true
show_mask: true
overlay_color: [0, 255, 0] # BGR green
centroid_color: [0, 0, 255] # BGR red
text_color: [255, 255, 255] # BGR whiteTuning tips:
- If the mask has noise (white specks in background) → increase
threshold_value - If the object is missing parts in the mask → decrease
threshold_value - If small contours are being detected as the object → increase
min_contour_area - If edges of the frame cause false detections → tighten the
roibounds
| Component | File | Status |
|---|---|---|
| Frame source (video/camera) | src/vision/frame_source.py |
✅ Done |
| Preprocessing (threshold) | src/vision/preprocess.py |
✅ Done |
| Object detection (contour) | src/vision/detection.py |
✅ Done |
| Detection tracker (debounce) | src/vision/detection_tracker.py |
✅ Done |
| Main pipeline + live overlay | src/vision/pipeline.py |
✅ Done |
| Configuration system | config/vision_config.yaml |
✅ Done |
| Entry point script | run_vision.py |
✅ Done |
| Dependencies | requirements.txt |
✅ Updated |
| Component | File | Status |
|---|---|---|
| Camera intrinsic calibration | src/vision/calibration.py |
✅ Done |
| Pixel → world homography | src/vision/coordinate_transform.py |
✅ Done |
| Belt motion compensation | src/vision/coordinate_transform.py |
✅ Done |
| Camera calibration script | run_camera_calibration.py |
✅ Done |
| Homography calibration script | run_homography_calibration.py |
✅ Done |
| Pipeline integration (undistort + homography + belt) | src/vision/pipeline.py |
✅ Done |
| Component | File | Status |
|---|---|---|
| RTDE sender (UR robot) | src/vision/RTDEsender.py |
✅ Done |
| RTDE config (registers) | rtde_config.xml |
✅ Done |
| Single-send detection tracker | src/vision/detection_tracker.py |
✅ Done |
| Auto-reconnect on disconnect | src/vision/pipeline.py |
✅ Done |
- Reads video at native FPS (~30 fps)
- Detects a single light rectangular object on a dark background
- Extracts centroid (x, y) and rotation angle (θ) per frame
- Single-send per part — each part triggers exactly one coordinate send (debounced)
- Sends coordinates to UR robot via RTDE with auto-reconnect
- Draws live overlay (bounding box + centroid + angle label)
- Loops video for continuous testing
- All parameters configurable via YAML (no code changes needed)
- ~5 ms processing time per frame
- Auto-loads calibration files if they exist (graceful degradation)
- Pixel → mm output when homography is calibrated
- Belt compensation predicts pick position when belt config is enabled
- CLI scripts for camera calibration (from images or live) and homography calibration (interactive click or direct points)
Before calibration and robot integration can happen, we need to physically measure and record these values from the actual conveyor setup. None of this can be done in software — it must be measured on-site.
| # | Measurement | Example Value | Why It's Needed | How to Measure |
|---|---|---|---|---|
| 1 | Conveyor belt width | e.g. 400 mm | Defines the X working area. Homography maps pixels → mm across this width. | Tape measure across the belt |
| 2 | Conveyor belt speed | e.g. 200 mm/s | Object moves between detection and robot pick. Robot must predict where the object will be. | Encoder on belt motor, or time a marked point over a known distance |
| 3 | Camera height above belt | e.g. 500 mm | Determines field of view and mm-per-pixel ratio. Higher = wider view, less resolution. | Measure from lens to belt surface |
| 4 | Camera position relative to robot origin | e.g. (ΔX=150, ΔY=0) mm | Robot's (0,0) ≠ camera's (0,0). Need an offset so coordinates make sense to the robot. | Measure or calibrate with a known point visible to both camera and robot |
| 5 | Object dimensions | e.g. 80×80 mm | Sanity check — compare detected size in mm to expected size to verify calibration accuracy. | Ruler on the object |
| 6 | Detection-to-pick delay | e.g. 50 ms | Time between "camera sees object" and "robot reaches object". Needed for belt compensation. | Measure or estimate from robot speed + communication latency |
These values will be stored in config/vision_config.yaml under a calibration: section once measured.
The 6 mm CS-mount lens distorts the image (barrel/pincushion distortion). Without correcting this, (x, y) measurements are inaccurate near the frame edges.
Procedure:
- Print a checkerboard pattern (e.g. 9×6 inner corners) on A4 paper
- Place it on the conveyor belt
- Take 15–20 photos from the camera at different angles/positions
- Run the calibration script → computes camera matrix + distortion coefficients
- Save to
config/camera_calibration.json - Pipeline applies
cv2.undistort()on every frame before detection
Math — what the camera matrix looks like:
Camera Matrix K:
┌ fx 0 cx ┐ fx, fy = focal lengths (pixels)
│ 0 fy cy │ cx, cy = principal point (image center)
└ 0 0 1 ┘
Distortion Coefficients:
[k1, k2, p1, p2, k3] (radial + tangential distortion)
Code: src/vision/calibration.py → calibrate_from_images() / calibrate_live() ✅
Script: python run_camera_calibration.py --dir path/to/checkerboard_images/ ✅
Currently the pipeline outputs positions in pixels (e.g. x=332, y=585). The robot needs positions in millimetres on the conveyor belt (e.g. X=87.5 mm from left edge, Y=210.3 mm from trigger line).
Procedure:
- Place 4 or more markers at known positions on the conveyor belt (e.g. corners of a rectangle with known mm dimensions)
- Record their pixel positions from the camera
- Compute the homography matrix H using
cv2.findHomography(pixel_pts, world_pts) - Save H to
config/homography.json - Pipeline applies the transform to every detection
Math — homography transform:
┌ Xw ┐ ┌ h11 h12 h13 ┐ ┌ xp ┐
│ Yw │ = H ⋅ │ h21 h22 h23 │ ⋅ │ yp │
└ 1 ┘ └ h31 h32 h33 ┘ └ 1 ┘
Where:
(xp, yp) = pixel coordinates from detection
(Xw, Yw) = real-world coordinates in mm on the conveyor
H = 3×3 homography matrix (computed once during calibration)
Example:
| What you have now | What you'll have after calibration |
|---|---|
x = 332 px, y = 585 px |
X = 87.5 mm, Y = 210.3 mm (from belt edge/trigger line) |
angle = 14.1° |
angle = 14.1° (already in degrees — no conversion needed) |
Code: src/vision/coordinate_transform.py → pixel_to_world() ✅
Script: python run_homography_calibration.py --interactive --source <video> --world-points "x1,y1 ..." ✅
The object is moving on the belt. Between the moment the camera detects it and the moment the robot reaches it, the object has moved. The robot must aim at the predicted future position.
Math:
X_pick = X_detected + V_belt × Δt
Where:
X_detected = position at time of detection (mm, from homography)
V_belt = conveyor belt speed (mm/s, from encoder or config)
Δt = detection-to-pick delay (seconds)
X_pick = where the robot should actually go
Example: Object detected at X = 87.5 mm. Belt moves at 200 mm/s. Robot takes 50 ms to reach. So: X_pick = 87.5 + 200 × 0.05 = 97.5 mm.
Code: src/vision/coordinate_transform.py → compensate_belt_motion() ✅
Config: Set belt.enabled: true, belt.speed_mm_s, and belt.detection_to_pick_delay_s in vision_config.yaml
Camera Frame
│
▼
cv2.undistort() ← Camera calibration (Step 1)
│
▼
preprocess() → mask
│
▼
detect_object() → (x_px, y_px, angle)
│
▼
pixel_to_world(x_px, y_px) ← Homography (Step 2)
│
▼
(X_mm, Y_mm, angle)
│
▼
compensate_belt_motion() ← Belt speed + delay (Step 3)
│
▼
(X_pick, Y_pick, angle) ← Final output to robot
│
▼
DetectionTracker ← Debounce + single-send (Step 4)
│ (sends once per part)
▼
RTDEsender.send_pose() ← RTDE to UR robot (Step 5)
| Task | Details | Status |
|---|---|---|
| Pi Global Shutter Camera capture | Implement a PiCameraSource class using picamera2 that matches the FrameSource interface. The Sony IMX296 global shutter sensor eliminates motion blur on the moving conveyor. Swap it into the pipeline via config. |
🔲 Needs hardware |
| Camera calibration code | Intrinsic calibration (lens distortion correction for the 6 mm CS-mount lens) using a checkerboard pattern + cv2.calibrateCamera(). Store results in config/camera_calibration.json. |
✅ Code done |
| Run camera calibration | Print checkerboard, take photos from Pi camera, run python run_camera_calibration.py. |
🔲 Needs hardware |
| Pi 5 optimisation | Ensure the pipeline runs efficiently on Pi 5's ARM CPU. Profile memory usage (8 GB available) and consider GPU acceleration via OpenCV's UMat if needed. | 🔲 Needs hardware |
| Task | Details | Status |
|---|---|---|
| Homography calibration tool | run_homography_calibration.py — interactive (click reference points) or direct (pass point pairs via CLI). |
✅ Done |
pixel_to_world() function |
Apply the homography to convert every detection from pixels to mm. | ✅ Done |
| Belt compensation | compensate_belt_motion() — predict pick position using belt speed and detection-to-pick delay. |
✅ Done |
| Pipeline integration | Pipeline auto-loads calibration + homography files, shows mm output, predicts pick position. | ✅ Done |
| Run homography calibration | Place reference markers on belt, run calibration script with camera. | 🔲 Needs hardware |
| Conveyor speed input | Either read from an encoder (via serial/GPIO) or set as a constant in config. | 🔲 Needs hardware |
| Task | Details | Status |
|---|---|---|
| RTDE protocol | Sends (X_m, Y_m, θ_rad, valid) to UR robot via RTDE input registers. |
✅ Done |
RTDEsender.py |
Connection, send, and cleanup. Auto-reconnects on disconnect. | ✅ Done |
| Detection tracker | State machine debounce — sends coordinates exactly once per physical part. | ✅ Done |
| Timing / sync | Measure the actual detection-to-pick delay and tune the belt compensation. | 🔲 Needs tuning |
| Task | Details |
|---|---|
| Multi-object detection | Extend detect_object() to return a list of results instead of just the largest. |
| Adaptive thresholding | Handle varying lighting conditions (e.g., cv2.adaptiveThreshold or HSV-based segmentation). |
| Unit tests | Add tests in tests/ for preprocessing, detection, and coordinate transforms. |
| Logging | Replace print() with Python logging module for configurable verbosity. |
| Error handling | Graceful recovery from dropped frames, serial disconnects, etc. |
| Task | Details |
|---|---|
| Calibration guide | Step-by-step instructions with photos for camera + homography calibration. |
| Deployment guide | How to run on the actual conveyor setup with the Pi 5. |
| Performance profiling | Benchmark end-to-end latency under real conditions. |
| Package | Version | Purpose |
|---|---|---|
numpy |
latest | Array operations |
opencv-python |
latest | Computer vision (capture, threshold, contours, display) |
pyyaml |
≥6.0 | Configuration file loading |
picamera2 |
— | Raspberry Pi camera library (commented out — enable when running on Pi 5) |
pyserial |
— | Robot serial communication (commented out — enable when hardware available) |
(To be determined)