An Android app that acts as an autonomous personal assistant — it sees your screen, thinks about what to do next, and controls your phone to complete multi-step tasks.
"Get the address from my last email and open it in Google Maps"
No APIs. No app integrations. Works on any app by reading the screen visually, just like a human would.
Docs: How it works · Setup guide · Architecture · Contributing
┌─────────────────────────────────────────────┐
│ PERCEIVE → THINK → ACT │
│ │
│ 1. Capture screenshot (MediaProjection) │
│ 2. Get UI element tree (Accessibility) │
│ 3. Send to Claude vision model │
│ 4. Claude decides: tap / swipe / type │
│ 5. Execute action on phone │
│ 6. Loop until task complete │
└─────────────────────────────────────────────┘
The vision LLM sees what's on screen and decides the next action. Your phone executes it. Repeats until done.
app/src/main/java/com/agentphone/
│
├── MainActivity.kt # UI — task input, voice input, log, permissions
│
├── ScreenCaptureService.kt # Captures screenshots via MediaProjection API
│ └── captureScreen() → Bitmap
│
├── AccessibilityActionService.kt # Executes actions on any app
│ ├── tap(x, y)
│ ├── swipe(x1, y1, x2, y2)
│ ├── typeText(text)
│ ├── pressBack() / pressHome()
│ └── getScreenTree() → String # dump UI hierarchy for LLM context
│
├── VisionLLMClient.kt # Claude API client
│ └── getNextAction(screenshot, task, history, stuckHint) → AgentAction
│
├── AgentLoop.kt # The core perceive→think→act loop
│ └── run(task) # suspending, runs until done/fail/stop
│
├── ScreenChangeDetector.kt # Pixel diff + perceptual hash (Phase 2)
│
└── SpeechInputHandler.kt # SpeechRecognizer wrapper (Phase 3)
- Android 10+ (API 29)
- An Anthropic API key
- Android Studio Hedgehog or later
git clone https://github.com/adev0x/android-agent
cd android-agent
# Open in Android Studio and run on device or emulator1. Screen Capture — tap the button in the app. Android shows its standard "Start recording?" dialog. Approve it.
2. Accessibility Service — tap the button, go to Settings → Accessibility → Downloaded apps → Agent Phone → Enable.
These are the only two permissions needed. No root required.
- Grant both permissions (Screen Capture + Accessibility)
- Enter your Anthropic API key
- Type a task — or tap the mic button to speak it
- Tap Run Agent
- Watch the log — the agent narrates each step
- For destructive actions (booking, sending), the app pauses and asks for confirmation
Open Gmail and tell me who sent the most recent email
Find the address in my last email from John and open it in Google Maps
Open Spotify and play something from my Liked Songs
Go to Settings and turn on Do Not Disturb
Open WhatsApp and send "On my way" to the last conversation
The LLM can output any of these actions:
| Action | Description |
|---|---|
tap(x, y) |
Tap at pixel coordinates |
swipe(x1,y1,x2,y2) |
Swipe gesture |
type(text) |
Type into focused field |
press(BACK|HOME) |
System buttons |
wait() |
Pause for screen to load |
confirm(message) |
Pause and ask user before proceeding |
done(result) |
Task complete |
fail(reason) |
Task could not be completed |
- Phase 1 — Core loop: screen capture + vision LLM + action execution
- Phase 2 — Autonomous loop improvements: change detection, error recovery, retry logic
- Phase 3 — Voice input: speak your task instead of typing
- Phase 4 — Task planner: break complex tasks into verified sub-steps
- Phase 5 — On-device LLM option: run Gemma locally, zero data leaves phone
| Approach | Works on | Requires |
|---|---|---|
| This app | Any app, any screen | Screen capture + accessibility |
| API-based (Gemini) | Partnered apps only | App-specific API access |
| Rooted device | Everything | Root |
The key insight: a vision model that can read any screen doesn't need special API access to any app. It works the same way a human looking at the screen would.
- Screenshots are sent to Anthropic's API to determine the next action
- No screenshots are stored persistently
- API key is stored in Android's SharedPreferences (local only)
- A future phase will support on-device vision models for fully offline operation
MIT