Phone Use SDK

Hardware-Level Phone Access for AI Agents

Build agents that can observe, tap, swipe, type, and run workflows on real mobile apps. Browser-use for real phones, without emulators or app APIs.

session = phone_use.connect("lumi-dock")
screen = session.observe_screen()
target = screen.find("Confirm appointment")

session.confirm_action("Tap confirmation button?")
session.tap(target)
session.type("Friday works. Sending calendar invite.")

memory.write({
  type: "todo",
  source: session.screenshot(),
  trace: session.action_log()
})

No emulator

Real phone, no API, no app-specific integration.

Lumi is for agent builders who need the messy, useful phone reality: visible screens, source traces, and human-in-the-loop control.

Vision module

Sees the real phone screen

4-axis mechanism

Positions for real taps and swipes

Touch actuator

Physical interaction with the screen

Phone dock

Stable alignment for common phones

AI head

Session state, voice, and local controls

SDK capabilities

Primitive actions, auditable workflows.

observe_screen()tap()swipe()type()screenshot()confirm_action()

Confirmation requiredSend this message?

Lumi asks before sensitive or irreversible actions.

09:41 observed Messages09:42 opened source thread09:42 created Todo draft09:43 waiting for approval

Human in the loop

Confirmation is part of the interface.

Agents should ask before risky actions, keep an action log, and expose screenshots as source evidence.

Turn Hidden Phone Promises Into Actionable Tasks

Phone chatPhysical actionTodo with source trace

Lumi sees screen and taps the message thread

Promised quote due Friday -> Todo, contact, reminder

A Relationship Inbox for the People Who Matter

Chats, calls, calendarPhysical actionPeople memory

Lumi checks visible context with approval

Waiting reply, next meeting, promised follow-up

Turn Phone Fragments Into an AI-Ready Knowledge Base

Screenshots, files, linksPhysical actionNotes, Calendar, Todo

Lumi groups fragments from the real phone

Tagged sources with editable memory cards

Developer kit

Join the Phone Use SDK Waitlist.

Mobile QA agents that operate real apps / Personal automation with human approval / PKM capture from screenshots and chats / Sales follow-up from phone-native context