RoadSense Pothole Detection FYP

RoadSense is a real-time multimodal pothole detection and geolocation system. It combines YOLO camera detections, ESP32 IMU/GPS telemetry, Random Forest classification, temporal LSTM classification, event severity estimation and a local review dashboard.

Install

.\.venv\Scripts\python.exe -m pip install -r requirements.txt

Hardware and model settings live in config.json.

Commands

Run the live detector:

.\.venv\Scripts\python.exe run.py detect

Tune fixed camera focus before a road run:

.\.venv\Scripts\python.exe run.py focus-tune

The same tuner also controls manual exposure. For bright daytime roads, use a more negative exposure value such as -7, -8, or -9 until the road texture is visible and the sky is not pure white.

Open the review dashboard:

.\.venv\Scripts\python.exe run.py dashboard

Then open http://127.0.0.1:5050.

Retrain both models and regenerate the comparison:

.\.venv\Scripts\python.exe run.py train-all

After collecting at least two runs with the upgraded raw telemetry logger, train the synchronized sensor LSTM:

.\.venv\Scripts\python.exe run.py train-sensor-lstm

Regenerate only the model comparison:

.\.venv\Scripts\python.exe run.py evaluate

Import a Car Scanner ELM OBD2 CSV export after a road run:

.\.venv\Scripts\python.exe run.py import-car-scanner "C:\path\to\car_scanner_export.csv" --run-id 20260627_134500

Run automated tests:

.\.venv\Scripts\python.exe run.py test

Live Outputs

• data/logs/event_log_*.csv: frame-aligned YOLO, sensor, ML, latency and severity data.
• data/telemetry/sensor_stream_*.csv: every raw IMU/GPS serial sample with PC time.
• data/obd/obd_stream_*.csv: optional OBD speed, RPM, throttle, engine load and coolant data.
• data/event_snapshots/: automatically saved visual, impact and ML events.
• models/ml/: trained Random Forest and LSTM artifacts.
• reports/model_comparison.json: shared held-out comparison used by the dashboard.

The logger automatically reconnects to the ESP32. Sensor age and freshness are logged, and stale IMU values are excluded from live decisions.

Camera autofocus and auto exposure are disabled by default in config.json using camera_disable_autofocus, camera_focus, camera_disable_auto_exposure, and camera_exposure. If the camera ignores software focus or exposure commands, run focus-tune and use the best visible values in config.json.

OBD is optional. If enable_obd is false, the detector still runs normally and logs empty OBD columns. For live OBD on the laptop, install the optional obd Python package, set enable_obd to true, and set obd_port to the adapter COM port. Many ELM327 adapters allow only one active connection, so if Car Scanner on the phone is connected during driving, export the Car Scanner CSV afterward and import it with run.py import-car-scanner.

Review Dashboard

The dashboard provides:

• Fast pothole, not-pothole and uncertain event review.
• Direct updates to data/labels/manual_event_labels.csv.
• GPS clustering to merge repeated detections near one road location.
• Severity, confidence and sensor-context inspection.
• Model comparison and road-run summaries.

Accuracy

Existing models were initially trained using weak labels derived from the original rules. Those scores are useful as a pipeline baseline but are not final real-world accuracy.

Use the dashboard to create manual labels, collect independent road runs, retrain, then evaluate on unseen roads. Follow docs/EXPERIMENT_PROTOCOL.md for the final study.

Manual label format:

log_file,start_frame,end_frame,label,notes
event_log_20260614_200311.csv,121,121,pothole,visually confirmed
event_log_20260614_200311.csv,339,339,normal,shadow false positive

Accepted training labels are pothole and normal. uncertain reviews are preserved but excluded from model ground truth.

Experimental Severity

Severity is reported as low, medium or high using visual confidence, bounding-box area, IMU delta and bump status. It is an experimental estimate and should only be claimed as validated after comparison with manually assigned severity or physical measurements.