AgenticUI

AgenticUI

AgenticUI is a generator in the GerdsenAI toolkit: /gerdsenai:agentic-ui scaffolds a standalone web app (a deliverable, not a UI for this plugin) that lets anyone watch an AI agent work. It mirrors how /gerdsenai:init scaffolds projects and /gerdsenai:build makes PDFs.

The generated app is public-safe and BYO-model: it ships no credentials and no hardcoded model endpoints. The end user supplies their own model for the computer-use brain via environment variables.

The three phases (cumulative via --phase 1|2|3)

1. Terminal — a live in-browser terminal. xterm.js (frontend) ⇄ WebSocket ⇄ node-pty (backend PTY). The PTY runs a configurable command (default shell), so the app can be pointed at any agent.
2. + Pseudo-desktop — a Dockerized virtual display the agent renders into: Xvfb (headless X server) → x11vnc (exposes it as VNC) → websockify + noVNC (VNC in the browser). Shown in a Desktop tab.
3. + Computer-use — a control loop that screenshots the virtual desktop, asks a VLM where to act, and simulates input with pyautogui/xdotool. Provider-agnostic (see below). A --read-only variant observes only (screenshots, no input simulation).

BYO-model contract (never hardcode endpoints)

The computer-use agent reads its model config from the environment, in this order:

• VLM_BASE_URL + VLM_MODEL + VLM_API_KEY — any OpenAI-compatible vision endpoint (messages:[{type:"image_url", image_url:{url:"data:image/png;base64,…"}}]). Works with hosted or local servers (the user points it wherever they like).
• else ANTHROPIC_API_KEY — use Claude's computer-use tool loop.
• If neither is set, the agent runs in observe-only mode and says so. No localhost default ships.

Security model (bake into every generated app)

• Bind all services to 127.0.0.1 by default; never 0.0.0.0 on the host.
• Require a shared auth token on every WebSocket/HTTP control endpoint.
• Default-recommend --read-only (watch, don't drive) in docs; make enabling input simulation explicit.
• Run the desktop and computer-use pieces in containers (isolation); the agent only touches the virtual display, never the host input devices.
• Ship a SECURITY.md: computer-use is powerful/dual-use — run it only against environments you own or control, inside the provided container, never exposed to the network.
• Keys come from the environment / .env (git-ignored). .env.example holds placeholders only.

UI/UX quality bar (Apple-HIG-aware, accessible)

• Semantic layout; visible focus; keyboard operable; sufficient contrast (WCAG AA).
• Clear connection status and loading / error / empty states (a terminal that can't connect must say why, with a retry).
• Responsive: usable from a laptop down to a tablet width.
• Trust + AI transparency: label what's live, what the agent is doing, and when it's acting on the desktop. Never expose provider keys in client-side code.

How the scaffolder works

scripts/agentic-ui-build.py copies templates/agentic-ui/phase1..N/ into the target dir, rendering tokens (__APP_NAME__, __TERM_PORT__, __DESKTOP_PORT__, __AGENT_PORT__, __AUTH_TOKEN__, __READ_ONLY__). Later phases override earlier files (e.g., docker-compose.yml). It generates a random auth token into .env, refuses to clobber without --force, and emits the standard JSON result.

See references/architecture.md, references/ui-ux.md, references/security.md, references/models.md (computer-use capability tiers: Claude native vs reasoning/vision vs weak local), and references/design-ultraplan.md (design-driven planning via /gerdsenai:design-plan → /ultraplan → frontend-design) for detail.