# Daemons

**Page:** kit/daemons

[Download Raw Markdown](./kit/daemons.md)

---

**Think PM2, but for ANY program via HTTP.** Hoody Daemons is a universal process manager that works with Node.js, Python, Go, Rust, compiled binaries, shell scripts—literally any executable. No CLI needed, no language-specific tools, just a simple REST API to keep any program running forever.


**PM2**: Node.js process manager via CLI
**Hoody Daemons**: Universal process manager via HTTP API

- ✅ Works with ANY program (not just Node.js)
- ✅ HTTP API (no SSH/CLI required)
- ✅ Powered by supervisord (battle-tested since 2004)
- ✅ Auto-restart, logging, startup ordering
- ✅ Simple JSON configuration

Manage Python workers, Go servers, Rust daemons, and Node.js apps all through the same interface.


## What You Can Do

- 🚀 **Program Management** - Create, configure, and delete daemon programs
- ▶️ **Process Control** - Start, stop, enable, disable running processes
- 📊 **Status Monitoring** - Track process state, uptime, and PID
- 🔄 **Auto-Restart** - Configure automatic restart on crashes
- 🎯 **Priority Control** - Set startup order for dependencies
- 📝 **Logging** - Configure stdout/stderr log files
- 🔐 **User Isolation** - Run each process as specific system user
- ⚙️ **Environment** - Set custom environment variables per program

## API Endpoints Summary

All endpoints accessed relative to your Daemon Manager service URL:
```
https://PROJECT_ID-CONTAINER_ID-daemon-1.SERVER.containers.hoody.icu
```

**Program Management**:
- [`GET /api/v1/daemon/programs`](/api/daemon/management/) - List all programs
- [`GET /api/v1/daemon/programs/{id}`](/api/daemon/management/) - Get program details
- [`POST /api/v1/daemon/programs/add`](/api/daemon/management/) - Create new program
- [`POST /api/v1/daemon/programs/edit/{id}`](/api/daemon/management/) - Update program
- [`POST /api/v1/daemon/programs/remove/{id}`](/api/daemon/management/) - Delete program

**Process Control**:
- [`POST /api/v1/daemon/programs/{id}/enable`](/api/daemon/control/) - Enable program
- [`POST /api/v1/daemon/programs/{id}/disable`](/api/daemon/control/) - Disable program
- [`POST /api/v1/daemon/programs/{id}/start`](/api/daemon/control/) - Start process
- [`POST /api/v1/daemon/programs/{id}/stop`](/api/daemon/control/) - Stop process

**Monitoring**:
- [`GET /api/v1/daemon/status`](/api/daemon/monitoring/) - All process statuses
- [`GET /api/v1/daemon/status/{id}`](/api/daemon/monitoring/) - Single process status

**Logs**:
- [`GET /api/v1/daemon/programs/{id}/logs`](/api/daemon/monitoring/) - Get program stdout/stderr logs
- [`GET /api/v1/daemon/quick-start/{id}/logs`](/api/daemon/management/) - Get ephemeral program logs

**Quick Start (Ephemeral Programs)**:
- [`GET /api/v1/daemon/quick-start`](/api/daemon/management/) - List ephemeral programs
- [`POST /api/v1/daemon/quick-start`](/api/daemon/management/) - Launch ephemeral program
- [`GET /api/v1/daemon/quick-start/{id}/status`](/api/daemon/management/) - Get ephemeral program status
- [`POST /api/v1/daemon/quick-start/{id}/stop`](/api/daemon/management/) - Stop ephemeral program

## Process Lifecycle

**Enable → Start → Stop → Disable → Remove**


  
    ```bash
    # Create a daemon program
    hoody daemon programs create \
      --name "web-server" \
      --command "/usr/bin/node /app/server.js" \
      --user "www-data" \
      --autorestart true \
      --directory "/app"

    # Enable and start
    hoody daemon programs enable 1
    hoody daemon programs start 1

    # Check status
    hoody daemon programs status 1

    # Stop and remove
    hoody daemon programs stop 1
    hoody daemon programs disable 1
    hoody daemon programs delete 1
    ```
  
  
    ```typescript
    import { HoodyClient } from '@hoody-ai/hoody-sdk';

    const client = new HoodyClient({ baseURL: 'https://api.hoody.icu', token: process.env.HOODY_TOKEN });
    const containerClient = await client.withContainer({ id: CONTAINER_ID, project_id: PROJECT_ID, server: SERVER });

    // Create a daemon program
    const result = await containerClient.daemon.programs.add({
      name: 'web-server',
      command: '/usr/bin/node /app/server.js',
      user: 'www-data',
      autorestart: 'true',
      directory: '/app',
    });
    const programId = result.data.id;

    // Enable and start
    await containerClient.daemon.control.enable(programId);
    await containerClient.daemon.control.start(programId);

    // Check status
    const status = await containerClient.daemon.status.get(programId);

    // Stop and remove
    await containerClient.daemon.control.stop(programId);
    await containerClient.daemon.control.disable(programId);
    await containerClient.daemon.programs.remove(programId);
    ```
  
  
    ```bash
    # Create a daemon program
    curl -X POST "https://PROJECT_ID-CONTAINER_ID-daemon-1.SERVER.containers.hoody.icu/api/v1/daemon/programs/add" \
      -H "Content-Type: application/json" \
      -d '{
        "name": "web-server",
        "command": "/usr/bin/node /app/server.js",
        "user": "www-data",
        "autorestart": "true",
        "directory": "/app"
      }'

    # Enable and start
    curl -X POST ".../api/v1/daemon/programs/1/enable"
    curl -X POST ".../api/v1/daemon/programs/1/start"

    # Check status
    curl "https://PROJECT_ID-CONTAINER_ID-daemon-1.SERVER.containers.hoody.icu/api/v1/daemon/status/1"
    ```
  


**Creating and Running a Daemon**:

**1. Add program (explicitly disabled — default is `enabled: true`):**



**2. Enable program:**



**3. Start process:**



**4. Monitor status:**



**Stopping and Removing**:

**1. Stop process:**



**2. Disable program:**



**3. Remove configuration:**



## Enable vs Start

**Critical Distinction**:

**Enable** = Configuration change
- Makes program available to supervisord
- Doesn't start the process
- Required before starting

**Start** = Runtime action
- Launches the actual process
- Only works if program is enabled
- Creates running process with PID

**Example**:
```bash
# This sequence is REQUIRED:
POST /programs/{id}/enable   # Configuration: "program can run"
POST /programs/{id}/start    # Runtime: "start the process"

# This FAILS:
POST /programs/{id}/start    # Error: program not enabled
```

## Auto-Restart Policies

Configure how programs behave on crashes:

```json
{
  "name": "critical-service",
  "command": "/usr/bin/service",
  "user": "service",
  "autorestart": "true"  // Always restart on exit
}
```

**Options**:
- `"true"` - Always restart (recommended for services)
- `"false"` - Never restart (one-time tasks)
- `"unexpected"` - Restart only on crashes (not clean exits)

## Program Configuration

Full program example:

```json
{
  "name": "api-server",
  "description": "Main REST API server",
  "command": "/usr/bin/node /app/api/server.js",
  "user": "api",
  "enabled": true,
  "boot": true,
  "autorestart": "true",
  "directory": "/app/api",
  "priority": 10,
  "stdout_logfile": "/var/log/api/stdout.log",
  "stderr_logfile": "/var/log/api/stderr.log",
  "environment": {
    "NODE_ENV": "production",
    "PORT": "3000",
    "DB_HOST": "localhost"
  }
}
```

## Process States

Monitor process health via status endpoint:

- **RUNNING** ✅ - Process running normally with PID
- **STOPPED** ⏸️ - Process not running (expected)
- **STARTING** ⏳ - Currently launching (temporary)
- **STOPPING** ⏳ - Gracefully shutting down (temporary)
- **BACKOFF** ⚠️ - Failed to start, retrying
- **FATAL** 💀 - Failed to start after retries, manual intervention needed

**Status Response**:
```json
{
  "statusCode": 200,
  "message": "OK",
  "data": {
    "success": true,
    "status": {
      "id": 1,
      "status": "RUNNING",
      "pid": 12345,
      "uptime": "2:15:30"
    }
  }
}
```

## Startup Priority

Control boot order when multiple programs depend on each other:

```json
[
  {
    "name": "database",
    "priority": 1,  // Starts first
    "boot": true
  },
  {
    "name": "cache",
    "priority": 5,  // Starts second
    "boot": true
  },
  {
    "name": "web-server",
    "priority": 10, // Starts last
    "boot": true
  }
]
```

Lower priority number = starts earlier.

## Managing Hoody Kit Services


**The entire Hoody Kit is managed by this Daemon service.** Every Kit service (Terminal, Display, Files, SQLite, Browser, Exec, Workspaces, Code, cURL, Cron, Notifications, Pipe, Notes, Watch, Run, Tunnel, Proxy Logs, and Daemons itself) runs as a daemon program.

Want to disable hoody-terminal temporarily? Stop it via the Daemons API. Need to restart hoody-sqlite? Control it here. This means you have **complete control over which Hoody Kit services are running** in your container through this simple HTTP API.

List all programs to see every Kit service running as a daemon — for example `hoody-terminal`, `hoody-display`, `hoody-files`, `hoody-sqlite`, `hoody-browser`, `hoody-exec`, `hoody-workspaces`, `hoody-code`, `hoody-curl`, `hoody-cron`, `hoody-notifications`, `hoody-pipe`, `hoody-notes`, `hoody-watch`, `hoody-run`, `hoody-tunnel`, `hoody-proxy`, and `hoody-daemon` itself.


## Use Cases

### Web Servers & APIs
Run Node.js, Python, or Go servers as daemons with auto-restart, configure logging for debugging, set startup priority if dependencies exist, monitor with status endpoint.

### Background Workers
Queue processors (Bull, Sidekiq, Celery), scheduled task runners, data sync services, cleanup jobs.

### Custom Background Services
Run custom application services, workers, and scripts with proper startup order and auto-restart policies. **Do not** use Hoody Daemon to manage package-installed system services such as PostgreSQL, MySQL, Redis, MongoDB, nginx, or apache — use the system service manager (systemd, OpenRC) for those.

### Monitoring Agents
Prometheus exporters, log shippers, health check agents, metrics collectors.

### Development Servers
Hot-reload dev servers, file watchers, test runners, development proxies.

### Microservices
Independent service processes, inter-service communication, graceful shutdown coordination, centralized process management.

## Best Practices

### Initial Configuration
Always add programs with `enabled: false`, verify configuration before enabling, test manually before setting `boot: true`, enable auto-restart only after stability testing.

### Updating Running Programs
Stop the process first, disable program to prevent auto-restart, edit configuration, enable and start with new config, verify new configuration works before cleanup.

### Logging Strategy
Always configure stdout and stderr logs, use absolute paths for log files, implement log rotation to manage disk space, monitor logs for errors and warnings.

### User Permissions
Run each program as appropriate system user, never run services as root unless required, create dedicated users for each service type, use principle of least privilege.

### Dependency Management
Use priority field for startup ordering, lower priority (1-5) for core services, higher priority (10-20) for dependent services, test boot sequence thoroughly.

### Monitoring
Poll `/status` endpoint regularly, alert on BACKOFF or FATAL states, track uptime for reliability metrics, implement health checks via other services.

## Useful Questions

**Q: What's the difference between enable and start?**
Enable is configuration ("program can run"), start is runtime action ("launch the process"). You must enable before starting.

**Q: How do I make a program start on boot?**
Set `boot: true` in the configuration. The program will auto-start when the daemon service initializes.

**Q: Can I update a running program?**
You must stop and disable it first, then edit, then enable and start again. Changes don't apply to running processes.

**Q: What happens if a process crashes?**
Depends on `autorestart`: `"true"` restarts immediately, `"false"` stays stopped, `"unexpected"` restarts only on crashes.

**Q: How do I run multiple instances of the same program?**
Create separate program configurations with different names, unique ports/sockets in command, different data directories, distinct priority values.

**Q: Can I see process output?**
Configure `stdout_logfile` and `stderr_logfile`, access logs via file system or Files service, tail logs in real-time with Files API.

**Q: How do I check if a program is running?**
Call `GET /status/{id}` - look for `status: "RUNNING"` and presence of `pid` field.

## Troubleshooting

### Program Won't Start (BACKOFF/FATAL)
**Cause**: Command fails, port already in use, missing dependencies, wrong user permissions.
**Solution**: Check stderr log file, verify command works manually, ensure port is available, confirm user exists and has permissions, run command directly as that user to test.

### Can't Update Running Program
**Cause**: Changes don't apply to running processes.
**Solution**: Follow proper sequence - stop, disable, edit, enable, start. Supervisord only reloads config on enable/disable.

### Auto-Restart Not Working
**Cause**: `autorestart: "false"` or program disabled.
**Solution**: Set `autorestart: "true"`, ensure program is enabled, check it's not in FATAL state, review supervisord logs.

### Program Stops Immediately After Start
**Cause**: Command exits normally, missing keep-alive loop, process detaches.
**Solution**: Verify command stays running (not a one-shot task), add keep-alive loop if needed, check for immediate errors in logs.

### Priority Not Respected on Boot
**Cause**: Boot delay not set, priority too similar.
**Solution**: Add `delay_seconds` between priorities, ensure priority numbers differ by 5+, check all programs have `boot: true`.

### Status Shows Wrong State
**Cause**: Status cache or supervisord mismatch.
**Solution**: Refresh status endpoint, check supervisord directly, restart daemon manager if persistent, verify program actually running with `ps`.

## What's Next