MuseGuard

Turn a spare phone into a home guardian for your baby, your pet or an older family member. It watches on the phone and only alerts you when it matters.

English · 繁體中文

Built for the Meta Global AI Developer Hackathon · Startup track · Build for Your World

---

Important

MuseGuard is an assistive alert. It does not replace adult supervision and is not a medical device.

Why

Most home cameras either stream video and leave all the watching to you, or fire alerts so often that families learn to ignore them. MuseGuard turns a spare phone into a guardian that watches continuously on the phone itself and only tells the family when something needs them. One household can have several guardians, each watching for one thing:

Care mode	What the guardian watches for	How it decides
Baby	Face covered, face-down sleep, no movement in the crib	On-device checks, then Meta's Muse Spark agent looks at the frames before a high-risk alert
Pet	A dog or cat stepping into a floor zone you draw, such as a doorway	Entirely on the phone, with no cloud calls
Older adult (pilot)	A possible fall in a room with one adult, with their agreement	On-device pose rules alert the family first; an optional AI review adds to the same check

Ollie, the owl, is on every guardian phone and tells the room what it's doing.

How it works

The diagram shows the Baby flow, which uses the full AI review. Pet runs only on the phone, and Older adult alerts from the phone first, with the AI review optional.

flowchart LR
    subgraph Monitor["Guardian phone (browser)"]
        C[Camera] --> S["On-device screening<br/>MediaPipe face + pose + motion"]
    end
    subgraph Server["Server (Node.js)"]
        A["Muse Spark 1.3 agent"] -- "tool: segment_objects" --> SAM["SAM 3.1"]
        A -- "tool: report_assessment" --> R{Risk level}
    end
    subgraph Parent["Family phones (browser / PWA)"]
        V[Live view]
        N[Push alert + AI reasoning]
    end
    C -- "WebRTC live video" --> V
    S -- "Trigger: 3 frames" --> A
    R -- "high" --> N

Loading

1. Screens on the device. The guardian phone runs MediaPipe detection in the browser, loading only the detectors its care mode needs. Video never leaves the phone during normal operation.
2. Asks AI for a second look. When something looks off, the phone sends 3 frames to the server:
   • face not visible for 10 s
   • likely face-down for 6 s
   • no movement for 90 s
3. Agent review. Muse Spark 1.3 reviews the frames and can call SAM 3.1 as a tool to segment the baby's face, blankets and cords. It then reports a risk level (none / low / high) and a category (face_covered, prone, entanglement, out_of_view).
4. Alerts only when it matters. A high assessment sends a Web Push notification and a live WebSocket update to every family member. Each alert carries the agent's reasoning as text; frames are not stored.

Features

• No extra hardware. Any phone with a browser becomes a guardian.
• One household, many guardians. Baby, Pet and Older adult guardians side by side, shared with family members as admins or viewers.
• Setup before guarding. A newly paired guardian shows Setting up and stays quiet until it's ready, then starts guarding by itself: Baby as soon as the phone is connected and linked to the household, Pet once its zone is drawn, Older adult once its floor area is drawn and the person in the room has agreed. The rest of the placement checklist is tips.
• Live view. Phone-to-phone WebRTC video for every family member who can view the guardian.
• Simple pairing. A 16-character code (XXXX-XXXX-XXXX-XXXX) or a share link.
• Explainable alerts. Each alert shows what SAM 3.1 found, a plain-language summary from the agent, and every step of the review.
• Activity history. High-risk alerts, events the AI cleared, and "needs check" alerts, filterable and grouped by day.
• English and 繁體中文. The whole app, the AI summaries and the push notifications follow the chosen language.
• Works without an API key. If MODEL_API_KEY is not set, on-device triggers send a "needs check" alert directly, so you can test the full flow.
• Detection Lab (/test.html). Test on-device screening with your own photos, processed in the browser only.
• Three care modes. Each guardian phone watches for one thing, chosen when it is paired and fixed after: Baby (the nursery checks above), Pet (a dog or cat stepping into a floor zone an admin draws, worked out on the phone with no cloud calls; routine entries go to Activity, with an optional "Notify on entry" push to family devices) and Older adult (pilot) (a visible descent from upright to staying low in a calibrated floor area; the family is alerted first and an optional AI review adds to the same check). Ollie stays on every guardian phone.
• Older adult privacy. These guardians belong to the household that enrolls them with a one-time credential; share codes never work for them. Nothing is captured until the person in the room agrees on the phone, cloud review is a separate choice, and Pause monitoring on the phone stops the camera, live view and uploads until someone there resumes.

Privacy & safety

• Video is analysed on the guardian phone. Frames are uploaded only when a trigger fires, and only 3 of them.
• Live video goes peer-to-peer over WebRTC and is not recorded by the server.
• Use the standard Meta Model API tier, not the Contributor tier: Contributor inputs may be used for training, which is not acceptable for images of babies or people at home.
• When the agent is unsure, it is instructed to rate the event high so that someone checks.

AI review limits

AI reviews are limited to 6 per room per hour, at least one minute apart; changing a share code does not reset them. Each recorded creator gets up to 10 per hour across their rooms. Rooms created before creator tracking, and rooms created without sign-in, share a separate pool of 10 per hour in total—we cannot safely infer who created an old room. These limits survive a restart and do not reset when an account is deleted. Site-wide limits remain 6 per minute, 30 per hour and 2 running at once. If a review is blocked, the on-device alert still arrives, marked as not reviewed by AI.

Tech stack

Layer	Tech
On-device vision	MediaPipe Tasks Vision (face detector, pose landmarker), self-hosted
AI review	Meta Model API: Muse Spark 1.3 (tool calling) + SAM 3.1 (segmentation)
Realtime	WebRTC (video), WebSocket (signalling and status)
Alerts	Web Push (VAPID) + service worker, installable PWA
Server	Node.js 22.13+, Express 5, ws, web-push, SQLite (node:sqlite)
Accounts	Firebase Authentication (Google sign-in)

Quick start

git clone https://github.com/Magic-Dala/museguard.git
cd museguard
npm install
cp .env.example .env   # add MODEL_API_KEY from https://dev.meta.ai
npm start              # http://localhost:3000

1. Open http://localhost:3000, sign in with Google (Firebase settings below; for local testing use the Auth emulator, see Data, backups and restore), create a household and tap Add → A spare phone or tablet. A QR code and pairing code appear.
2. In another tab (or on the spare phone), open http://localhost:3000/?guardian, scan the QR or type the code. The family tab moves on to the tutorial and the placement checklist (Pet and Older adult also draw their zone or floor area first). The guardian shows Setting up until it's ready (Baby: as soon as it connects), then starts guarding by itself.
3. On the guardian, open settings → Ask AI to check now, or cover the camera for 10 seconds, and the alert appears in the family app.

Light, dark and auto themes and English / 繁體中文 are in Account.

Testing with real phones

Camera access and push notifications require HTTPS (localhost is the exception). The quickest way is a tunnel:

npx cloudflared tunnel --url http://localhost:3000

Open the generated https://….trycloudflare.com URL on both phones.

Note

iPhone: push notifications need iOS 16.4+. In Safari, tap Share → Add to Home Screen, then open MuseGuard from the Home Screen before turning on alerts.

Deploying

Any host that supports WebSockets works (Render, Railway, Fly.io…). Set MODEL_API_KEY and fixed VAPID_* keys. Without fixed keys, a new key pair is generated on each deploy and family members have to turn alerts on again.

The public build runs at https://museguard.magicdala.com. It runs on the magicdala GCP VM (museguard-vm): every push to main rebuilds it through the self-hosted runner (.github/workflows/deploy.yml → deploy/docker-compose.yml). A manual run using this version of the Deploy workflow also deploys only from main; started on any other branch, the job is skipped. This guard is not a security boundary: an older or modified branch workflow can omit it. The repository owner must separately restrict the production runner to trusted workflows on main (and, if configured, a protected production environment); keep issue #18 open until those settings are verified. Cloudflare tunnel museguard-vm routes the hostname to the app container (see deploy/cloudflared/config.example.yml). Never publish the app port (3000) to the internet: the WebSocket connection limits trust the CF-Connecting-IP header from loopback and private (Docker) addresses, where cloudflared connects from, and Docker's port proxy would make outside clients look private. Secrets stay in /opt/museguard on the VM. Google sign-in only works on domains listed under Firebase Auth → Settings → Authorized domains.

Data, backups and restore

Everything the app saves lives in SQLite at $DATA_DIR/museguard.sqlite (/data/museguard.sqlite in Docker, on the museguard_data volume). It uses Node's built-in node:sqlite, so there is no database server and no extra dependency.

• Saved: rooms (code, name, hashed owner and camera keys), alert history as text (newest 50 per guardian), push subscriptions, accounts (Firebase UID, name, email, preferences), households, members and roles, the household's guardian list, and Ollie chat text (newest 200 messages per person, 30 per page).
• Never saved: video, alert snapshots or other images, audio, or raw model requests. Review frames go to the model and are dropped; the alert keeps the text summary.
• Accounts are keyed by the verified Firebase UID. Every account request verifies the Google ID token, and the server checks household membership and the admin / viewer role. Signing out ends the browser session only; nothing saved is deleted.
• Guardians in a household. A room joins a household only when an admin proves access: its owner or camera key (full control: rename, new code, unpair) or its share code (view only, dropped when the code changes). Removing a guardian from a household is separate from unpairing it.

Upgrading from rooms.json. On the first start with this version, the server imports rooms.json in one transaction, keeping rooms, owner and camera key hashes, push subscriptions and text alert history, and dropping image fields. No copy of the file is made. If the import fails, nothing is saved, rooms.json is unchanged, and the server refuses to start and logs why; fix or move the file and start again. Imported rooms belong to no household until an admin links them with the key or code.

Backup (safe while the app is running):

docker compose -p museguard -f deploy/docker-compose.yml exec -T app \
  node -e "const s = require('node:sqlite'); const f = '/data/backup-' + Date.now() + '.sqlite'; s.backup(new s.DatabaseSync('/data/museguard.sqlite'), f).then(() => console.log(f))"
docker cp museguard-app-1:/data/backup-<time>.sqlite ./   # the path printed above; then copy it off the VM to access-controlled storage

A Docker volume is not a backup: VM or disk loss still loses it. Keep copies off the VM and test a restore now and then.

Restore: stop the app, put the backup in place, start it again.

docker compose -p museguard -f deploy/docker-compose.yml stop app
docker run --rm -v museguard_data:/data -v "$PWD":/in alpine sh -c \
  'rm -f /data/museguard.sqlite-wal /data/museguard.sqlite-shm && cp /in/backup-<time>.sqlite /data/museguard.sqlite && chown 1000:1000 /data/museguard.sqlite'
docker compose -p museguard -f deploy/docker-compose.yml start app

Retiring old snapshots. Older versions saved one snapshot per AI-reviewed alert inside rooms.json, and earlier SQLite versions also made a raw copy, rooms.json.pre-sqlite-*.bak. On every start the server rewrites those files on the data volume as text only (rooms and alert text kept, images removed, so rooms.json can still be imported into a new, empty database). A file that isn't a valid rooms JSON file is left alone with a warning in the log; check it and delete it by hand. The server can't reach copies outside the volume: remove older volume, VM or off-site backups made before this change by hand, following your backup-retention period.

Local testing with the Firebase Auth emulator (no real Google account, no production data):

npx -y firebase-tools@13 emulators:start --only auth --project demo-museguard   # terminal 1
PORT=3100 DATA_DIR=/tmp/museguard-preview MODEL_API_KEY= FIREBASE_PROJECT_ID=demo-museguard \
  FIREBASE_API_KEY=demo-key FIREBASE_AUTH_EMULATOR_HOST=127.0.0.1:9099 npm start       # terminal 2

Open http://localhost:3100 and use Continue with Google: the emulator's own page lets you add test accounts. The server refuses FIREBASE_AUTH_EMULATOR_HOST unless the project id starts with demo-, because the emulator accepts unsigned tokens. npm test uses the same emulator mode with a small local fake lookup (tests/fixtures/app.js), so it never contacts Google.

Testing on a real phone. Phone cameras need HTTPS, so a phone reaches a local preview through a temporary tunnel (for example npx cloudflared tunnel --url http://localhost:3100). To sign in on the phone too, tunnel port 9099 as well and start the server with FIREBASE_AUTH_EMULATOR_URL set to that tunnel's address. In emulator mode the server accepts unsigned tokens, so anyone who has a tunnel link can act as any test user: use a throwaway DATA_DIR, leave MODEL_API_KEY empty, and stop the tunnels when you're done.

Configuration

Variable	Required	Description
PORT	no	HTTP port (default 3000)
MODEL_API_KEY	recommended	Meta Model API key. Without it, alerts come from on-device screening only.
META_API_BASE	no	API base URL (default https://api.meta.ai/v1)
META_CHAT_MODEL	no	Review model (default muse-spark-1.3)
META_SAM_MODEL	no	Segmentation model (default sam-3.1)
VAPID_PUBLIC_KEY / VAPID_PRIVATE_KEY / VAPID_SUBJECT	for production	Fixed Web Push keys. Auto-generated into data/vapid.json otherwise.
DATA_DIR	no	Where museguard.sqlite and vapid.json live (default ./data, /data in Docker)
FIREBASE_PROJECT_ID / FIREBASE_API_KEY / FIREBASE_APP_ID / FIREBASE_AUTH_DOMAIN	for the family app	Google sign-in (Firebase web app config). Without them the family app cannot sign in.
FIREBASE_AUTH_EMULATOR_HOST	local testing only	Firebase Auth emulator; only allowed with a demo- project id. Never set it on the VM.
FIREBASE_AUTH_EMULATOR_URL	local testing only	The emulator's address as a phone sees it, such as an HTTPS tunnel to port 9099, so a phone can sign in during local testing. Only used together with FIREBASE_AUTH_EMULATOR_HOST.
MUSEGUARD_DEMO_INPUTS	local demos only	1 lets a Pet or Older adult guardian opened with /?guardian&demo feed clearly labeled synthetic observations to its rules. Never set it on the VM.

Project structure

Path	Purpose
server.js	Express API, WebSocket signalling, alerts and push
lib/agent.js	Muse Spark 1.3 tool-calling loop and the SAM 3.1 tool
lib/push.js	Web Push (VAPID)
lib/db.js	SQLite schema, transactions and the one-time rooms.json import
lib/store.js	Rooms, push subscriptions and text alert history (cached in memory, written to SQLite)
lib/accounts.js	Accounts, households, roles, guardian links and Ollie chat, with permission checks
public/app.js	Family app: sign-in, household, home, add a guardian, live view, Ollie chat, activity, alerts, account
public/guardian.js	Guardian mode (/?guardian): QR scan, camera, WebRTC, screening, review triggers; portrait and landscape
public/session.js	Family app's connection to one guardian: presence, live video, status, alerts
public/account.js	Family app's account, household, guardians, preferences and chat, loaded from the server after sign-in
public/screening.js / checks.js	Shared MediaPipe screening logic and check wording; loads only the detectors a care mode needs
public/guardian-modes.js	Pet and Older adult guardian loops: detector sampling, zone overlay, Ollie's words, events and review uploads
public/zone.js / fall.js / geometry.js	The on-device rules (pet tracking and watch zone; possible-fall evidence) and zone geometry, shared by the phone, the tests and the replay tool
public/synthetic.js	Synthetic pose and pet observations for tests, the replay and the labeled local demo input
tools/replay/	Python offline replay and evaluation of the same rules on synthetic or separately consented landmark sequences (see its README)
public/ui.js / i18n.js	Icons, Ollie, theme (light / dark / auto), translations (EN / 繁體中文)
public/ollie.js / ollie-rig.css	Ollie as a layered puppet (head, eyes, lids, beak, wings) with states such as idle, listening, thinking, talking and night watch. See docs/ollie-rig/
public/ollie-lab.html	Ollie lab: every state at three sizes, light and dark, plus the greeting and an interrupted pose
public/legal.js	Terms of Service and Privacy Policy (EN / 繁體中文)
public/styles.css	Design system: tokens, components, light and dark themes, desktop layout
public/test.html	Detection Lab: photo-based screening test
public/sw.js	Service worker for push notifications
docs/	Landing page and brand assets

Known limitations & roadmap

• No TURN server yet (STUN only). Live video may fail when both phones are on mobile data or behind a strict NAT. Alerts are not affected.
• Adult-trained pose model. MediaPipe's pose model was trained on adults, so on-device thresholds are intentionally loose and the AI review filters the false positives.
• Single-host SQLite. One app process owns the database; several hosts would need a shared database.
• Ollie replies are samples. Chat text is saved, but the replies are still canned previews.
• Invites match a verified Google email. There is no invitation email yet; the invited person signs in with that address.
• Older adult is a pilot. The fall rules are checked only on synthetic sequences (tools/replay), not on real phones, real homes or real falls. It assumes one visible adult and a fixed camera, can miss a fall or notice one that didn't happen, and can't tell whether someone is hurt. Field validation with consented data is still open.
• Pet mode counts entries to one zone, dogs and cats only. Bowl visits, nighttime exits and "first seen low" checks are later work.

License

MIT © 2026 Magic Dala. Third-party code keeps its own license, for example .agents/skills/security-audit/ (MIT, Cloudflare).