# hoody-display Subskill ## Overview ### What This Service Does hoody-display provides fully embeddable, multiplayer desktop environments accessible via URL. It enables GUI application control through a virtual display server, allowing agents and users to interact with graphical applications remotely. **Core capabilities:** - Screenshot capture and retrieval (full-size and thumbnails) - Mouse and keyboard input simulation - Window management (focus, move, resize, close) - Clipboard read/write operations - Display geometry and information queries ### When to Use It - **GUI automation**: Interacting with desktop applications that require visual input - **Visual verification**: Capturing screenshots to confirm application state - **Multi-user collaboration**: Multiple agents or users sharing a single desktop environment - **Remote desktop access**: Accessing containerized GUI apps via URL - **Testing workflows**: Automated testing of graphical interfaces ### How It Fits Into Hoody Philosophy hoody-display embodies Hoody's "desktop environments that are fully embeddable and multiplayer" philosophy. Each container gets an isolated virtual display accessible via a standardized URL pattern: ``` https://{projectId}-{containerId}-display-{serviceId}.{node}.containers.hoody.icu ``` This enables zero-configuration access to GUI applications with automatic SSL, authentication, and multi-tenant isolation. --- ## Common Workflows ### Workflow 1: Health Check and Service Verification Always verify service availability before performing operations. ``` # Check display service health hoody display health -c ``` ``` { "status": "healthy", "service": "display", "version": "1.0.0", "uptime": 3600, "timestamp": "2025-01-15T10:30:00Z", "checks": { "display": "ok", "input": "ok", "screenshot": "ok" } } ``` ``` # Get display information and available screenshots hoody display info -c ``` ``` { "displayId": "abc123", "width": 1920, "height": 1080, "screenshots": [ { "timestamp": 1705312200, "width": 1920, "height": 1080 } ] } ``` ### Workflow 2: Screenshot Capture and Retrieval Capture visual state for verification or analysis. ``` # Capture a fresh screenshot hoody display capture -c ``` ``` # Get metadata without downloading image hoody display capture-metadata -c ``` ``` { "timestamp": 1705312200, "width": 1920, "height": 1080, "format": "png", "size": 245760 } ``` ``` # Retrieve the most recent screenshot hoody display latest -c ``` ``` # Get metadata for the latest screenshot hoody display latest-metadata -c ``` ``` # Retrieve a specific screenshot by timestamp hoody display by-timestamp 1705312200 -c ``` ``` # List all available screenshots hoody display list -c ``` ``` [ { "timestamp": 1705312200, "width": 1920, "height": 1080, "format": "png" }, { "timestamp": 1705312100, "width": 1920, "height": 1080, "format": "png" } ] ``` ### Workflow 3: Thumbnail Operations Use thumbnails for faster visual checks when full resolution isn't needed. ``` # Capture a new thumbnail (320x180 scaled) hoody display capture -c ``` ``` # Get the most recent thumbnail hoody display latest -c ``` ``` # Get thumbnail by specific timestamp hoody display by-timestamp 1705312200 -c ``` ### Workflow 4: Mouse Input Operations Control mouse interactions with the display. ``` # Get current cursor position hoody display location -c ``` ``` { "x": 960, "y": 540 } ``` ``` # Move cursor to absolute position hoody display move -c --x 500 --y 300 ``` ``` # Move cursor by relative offset hoody display move-relative -c --x 50 --y -25 ``` ``` # Click at current position hoody display click -c ``` ``` # Double-click at current position hoody display double-click -c ``` ``` # Press and hold mouse button hoody display down -c ``` ``` # Release mouse button hoody display up -c ``` ``` # Scroll in a direction hoody display scroll -c --direction down ``` ### Workflow 5: Keyboard Input Operations Type text and send key combinations. ``` # Type a string of text hoody display type -c --text "Hello, World!" ``` ``` # Press key combination (e.g., Ctrl+C) hoody display key -c --keys '["ctrl+c"]' ``` ``` # Press Enter key hoody display key -c --keys '["Return"]' ``` ``` # Hold a key down hoody display key-down -c --key Shift_L ``` ``` # Release a held key hoody display key-up -c --key Shift_L ``` ### Workflow 6: Clipboard Operations Read and write clipboard content. ``` # Read clipboard text hoody display get -c ``` ``` { "text": "Clipboard content here" } ``` ``` # Write text to clipboard hoody display set -c --text "New clipboard content" ``` ### Workflow 7: Window Management List and control windows on the display. ``` # List all windows hoody display list -c ``` ``` [ { "windowId": "0x0400001", "name": "Firefox", "geometry": { "x": 0, "y": 0, "width": 1920, "height": 1080 } }, { "windowId": "0x0400002", "name": "Terminal", "geometry": { "x": 100, "y": 100, "width": 800, "height": 600 } } ] ``` ``` # Get the active window hoody display active -c ``` ``` { "windowId": "0x0400001", "name": "Firefox" } ``` ``` # Get window properties hoody display properties 0x0400001 -c ``` ``` # Get window geometry hoody display geometry 0x0400001 -c ``` ``` { "x": 0, "y": 0, "width": 1920, "height": 1080 } ``` ``` # Get window title hoody display name 0x0400001 -c ``` ``` { "name": "Firefox" } ``` ``` # Focus a window hoody display focus -c --window-id 0x0400001 ``` ``` # Move a window hoody display move -c --window-id 0x0400001 --x 100 --y 100 ``` ``` # Resize a window hoody display resize -c --window-id 0x0400001 --width 800 --height 600 ``` ``` # Minimize a window hoody display minimize -c --window-id 0x0400001 ``` ``` # Close a window hoody display close -c --window-id 0x0400001 ``` ``` # Raise window to top hoody display raise -c --window-id 0x0400001 ``` ``` # Search for windows by pattern hoody display search -c --pattern "Firefox" ``` ### Workflow 8: Access Display Client Access the HTML5 web interface for direct visual interaction. ``` # Get the display client URL hoody display access -c ``` ``` { "url": "https://proj123-ctr456-display-svc789.node1.containers.hoody.icu" } ``` --- ## Advanced Operations ### Complex Multi-Step Workflow: Form Submission Automate filling out and submitting a form. ``` # Step 1: Capture screenshot to see current state hoody display capture -c # Step 2: Move to username field and click hoody display click-at -c --x 400 --y 250 # Step 3: Type username hoody display type -c --text "user@example.com" # Step 4: Press Tab to move to password field hoody display key -c --keys '["Tab"]' # Step 5: Type password hoody display type -c --text "securepassword123" # Step 6: Press Enter to submit hoody display key -c --keys '["Return"]' # Step 7: Wait for page load and capture result hoody display wait -c --duration 2000 hoody display capture -c ``` ### Complex Multi-Step Workflow: Drag and Drop ``` # Step 1: Move to start position hoody display move -c --x 200 --y 300 # Step 2: Press mouse button hoody display down -c # Step 3: Move to end position hoody display move -c --x 500 --y 400 # Step 4: Release mouse button hoody display up -c # Alternative: Use drag command for single operation hoody display drag -c --start-x 200 --start-y 300 --end-x 500 --end-y 400 ``` ### Complex Multi-Step Workflow: Text Selection ``` # Select text using click and shift-click hoody display select -c --start-x 100 --start-y 200 --end-x 400 --end-y 200 # Copy selected text hoody display key -c --keys '["ctrl+c"]' # Read clipboard to get selected text hoody display get -c ``` ### Batch Operations Execute multiple actions in sequence for efficiency. ``` # Execute a batch of actions hoody display batch -c --actions '[ {"type": "move", "x": 100, "y": 100}, {"type": "click"}, {"type": "type", "text": "Hello"}, {"type": "key", "keys": ["Return"]} ]' ``` ### Single Action with Screenshot Execute an action and capture the result in one call. ``` # Click and get screenshot of result hoody display act -c --action '{"type": "click", "x": 500, "y": 300}' --screenshot true ``` ### Wait with Screenshot Wait for a duration and capture the state. ``` # Wait 2 seconds and capture screenshot hoody display wait -c --duration 2000 --screenshot true ``` ### Emergency Input Reset Release all held inputs when things go wrong. ``` # Emergency release all inputs hoody display reset -c ``` ### Get Display Geometry ``` # Get display dimensions hoody display geometry -c ``` ``` { "width": 1920, "height": 1080 } ``` ### Error Recovery Patterns **Pattern 1: Screenshot verification loop** ``` # Capture initial state hoody display capture -c # Perform action hoody display click-at -c --x 500 --y 300 # Verify result hoody display capture -c # If unexpected state, reset and retry hoody display reset -c ``` **Pattern 2: Window focus recovery** ``` # If input isn't working, ensure correct window is focused hoody display active -c hoody display focus -c --window-id 0x0400001 ``` **Pattern 3: Clipboard state management** ``` # Save current clipboard before operations hoody display get -c > /tmp/clipboard_backup.txt # Perform operations that modify clipboard hoody display type -c --text "new content" hoody display key -c --keys '["ctrl+a"]' hoody display key -c --keys '["ctrl+c"]' # Restore original clipboard if needed hoody display set -c --text "$(cat /tmp/clipboard_backup.txt)" ``` ### Performance Considerations 1. **Use thumbnails for quick checks**: Thumbnails are 320x180 and faster to transfer 2. **Batch operations**: Use `batch` command for multiple sequential actions 3. **Minimize screenshots**: Only capture when visual verification is needed 4. **Use metadata endpoints**: When you only need dimensions/timestamp, use metadata variants 5. **Cache window IDs**: Window IDs are stable within a session; cache them to avoid repeated lookups --- ## Quick Reference ### Most Common Commands | Command | Description | |---------|-------------| | `hoody display health -c ` | Check service health | | `hoody display capture -c ` | Capture screenshot | | `hoody display latest -c ` | Get latest screenshot | | `hoody display click-at -c --x X --y Y` | Click at position | | `hoody display type -c --text "..."` | Type text | | `hoody display key -c --keys '["ctrl+c"]'` | Key combination | | `hoody display list -c ` | List windows | | `hoody display focus -c --window-id WID` | Focus window | | `hoody display get -c ` | Read clipboard | | `hoody display set -c --text "..."` | Write clipboard | | `hoody display reset -c ` | Emergency input reset | ### Essential Parameters | Parameter | Description | Example | |-----------|-------------|---------| | `-c ` | Target container (required) | `-c ctr_abc123` | | `-o json` | JSON output format | `-o json` | | `--x`, `--y` | Coordinates | `--x 500 --y 300` | | `--text` | Text content | `--text "Hello"` | | `--keys` | Key combinations (JSON array) | `--keys '["ctrl+c"]'` | | `--window-id` | Window identifier | `--window-id 0x0400001` | | `--direction` | Scroll direction | `--direction down` | | `--duration` | Wait duration in ms | `--duration 2000` | ### Typical Response Formats **Health check:** ``` { "status": "healthy", "service": "display", "version": "1.0.0", "uptime": 3600, "timestamp": "2025-01-15T10:30:00Z", "checks": { "display": "ok", "input": "ok", "screenshot": "ok" } } ``` **Screenshot metadata:** ``` { "timestamp": 1705312200, "width": 1920, "height": 1080, "format": "png", "size": 245760 } ``` **Window list:** ``` [ { "windowId": "0x0400001", "name": "Firefox", "geometry": { "x": 0, "y": 0, "width": 1920, "height": 1080 } } ] ``` **Mouse location:** ``` { "x": 960, "y": 540 } ``` **Clipboard content:** ``` { "text": "Clipboard content here" } ``` **Display geometry:** ``` { "width": 1920, "height": 1080 } ``` --- ## Notes - All commands require `-c ` or `HOODY_CONTAINER` environment variable - Use `HOODY_TOKEN` environment variable or `--token` flag for authentication - Window IDs are hexadecimal strings (e.g., `0x0400001`) - Coordinates are absolute pixel positions from top-left (0,0) - Key names follow X11 keysym conventions (e.g., `Shift_L`, `ctrl`, `Return`) - Screenshots are PNG format by default - Thumbnails are scaled to 320x180 pixels