Rust CLI giving AI agents structured JSON access to macOS accessibility trees, no screenshots, with 78-96% token reduction via progressive skeleton traversal.
Key Takeaways
agent-desktop exposes 53 commands covering observation, interaction, keyboard, mouse, clipboard, notifications, and window management against any app with an AX tree.
Progressive skeleton traversal (depth-3 overview + targeted --root @eN drill-down) achieves 78-96% token reduction on dense apps like Slack or VS Code.
Deterministic element refs (@e1, @e2) persist until the next snapshot; stale refs return machine-readable STALE_REF with a recovery hint.
A C-ABI cdylib (libagent_desktop_ffi) lets Python, Go, Node, Swift, and Ruby agents call in-process via dlopen instead of forking the CLI per command.
Currently macOS-only (13.0+, Rust 1.78+); Windows and Linux support listed as planned.