RipSnapv2

RipSnapv2 is a React Native app for rip-current detection and capture upload. The current app supports local on-device TFLite inference. A planned remote streaming mode would allow heavier PyTorch .pt models to run on a server while users interact with a live camera/detection experience from inside the mobile app.

Current Local Detection Path

Phone camera -> local TFLite model -> native overlay -> native capture review -> Firebase upload

Local models live in:

ripsnap_models/

The app model registry lives in:

src/config/detection.ts

Remote Streaming Goal

Add an optional remote detection mode where users can stream camera frames or video to a server running heavier .pt models, receive detections in real time, and upload captures through the existing mobile app flow.

Embedded web app -> phone camera -> LiveKit/WebRTC -> server .pt model -> web overlay -> native upload

This remote mode would exist alongside the current local TFLite mode.

Planned Streaming Architecture

Embedded web app
  camera preview
  LiveKit client publishes camera video
  canvas overlay draws server detections
  sends capture events to React Native via postMessage

LiveKit
  handles WebRTC rooms, signaling, reconnects, and video transport
  carries camera video from the phone to the inference worker
  carries detection messages back to the web app

Python inference worker
  joins the LiveKit room
  reads video frames
  runs PyTorch .pt model
  returns normalized detection boxes

LiveKit is the preferred starting point because it gives us WebRTC without having to build raw signaling, room management, reconnect handling, and STUN/TURN plumbing ourselves.

If LiveKit ends up being too limiting for the research workflow, the later alternative would be a custom WebRTC stack. That should only be considered after we have benchmarked the LiveKit version and know what problem we are trying to solve.

Frontend Stack

Vite
React
TypeScript
HTML video element
Canvas overlay
LiveKit JavaScript client

Backend Stack

Python
LiveKit server / LiveKit Cloud
LiveKit Python SDK
PyTorch
OpenCV or Pillow

undecided on if we need a framework

Performance Target

The realistic goal is:

30 FPS camera preview
live detection updates as fast as the server/model can produce them

LiveKit should be able to carry a 30 FPS phone camera stream. The harder question is whether the .pt model and server pipeline can infer at 30 FPS. We should measure stream FPS, inference time, and end-to-end detection latency.

High-Level Architecture

RipSnap React Native app
  Remote Detection screen
    WebView loads remote web app
    native app provides auth/GPS/upload bridge

Embedded web app
  owns camera preview
  owns streaming connection
  draws live detection overlay
  sends capture events to native app

Remote inference server
  receives LiveKit/WebRTC video stream
  runs .pt model
  returns detections

Firebase
  stores uploaded media + metadata

Embedded Web UX

The WebView should feel like a native mode inside RipSnap.

Connection States

Disconnected
Connecting
Live
Reconnecting
Server unavailable
Poor network

Capture Flow

1. User taps Capture in embedded web app.
2. Web app freezes current frame and current detections.
3. Web app sends capture payload to React Native via postMessage.
4. Native takes over in existing review and upload flow

Responsibility Boundaries

Web App Owns

• Camera preview
• Remote streaming
• Detection overlay
• Remote model controls
• Latency display
• Capture frame extraction

Native App Owns

• App navigation
• Firebase auth
• GPS location
• Upload metadata
• Review form
• Offline/local TFLite mode
• Settings tab

Server Owns

• .pt model loading
• Preprocessing
• Inference
• Postprocessing
• Model version reporting
• Health checks

MVP Features

• Embedded WebView remote detection mode
• Camera preview in web app
• Live server detections
• Capture button for images or video
• Native review/upload handoff
• GPS metadata from native app
• Notes stored in Firestore and metadata JSON
• Remote model name/version in metadata
• Connection and latency indicators
• Threshold and max result controls
• Server can handle a small amount of users concurrently

Concerns

Camera in WebView

Mobile browser camera APIs work well in Safari/Chrome, but embedded WebView behavior needs testing on both iOS and Android.

Latency

Server inference time, upload bandwidth, and frame size determine UX quality. The prototype should measure round-trip latency from day one.

Server and Network Work

The mobile/web app is only part of the system. Remote detection also requires:

• GPU server setup for PyTorch inference
• LiveKit deployment or LiveKit Cloud configuration
• secure token/session creation
• model loading and versioning
• frame decoding/preprocessing
• inference queue/backpressure
• reconnect and timeout behavior
• basic monitoring for FPS, latency, and server errors

Cloud Build

First-time development build:

eas build --profile development --platform android

Or:

eas build --profile development --platform ios

Build Locally

yarn expo run:android

Or:

yarn expo run:ios

General Dev

yarn start

Notes

You need google-services.json and GoogleService-Info.plist in the root directory, plus the project .env file.