GPU Energy Agent

Overview

The agent uses NVIDIA's NVML library (via pynvml) to sample each GPU every 5 seconds. Metrics are buffered locally and flushed to the AluminatiAi ingest API every 60 seconds. If an upload fails, metrics are persisted to a write-ahead log (data/wal/metrics.wal) and automatically replayed on the next successful connection.

Installation

Requires nvidia-smi to be working before you start. Run it to confirm your drivers are installed.

Option A — pip (recommended)

Install the agent directly from PyPI. No cloning required.

pip install aluminatiai

# Verify GPU detection
aluminatiai --help

Option B — install.sh (systemd auto-setup)

Installs as a systemd service, creates a dedicated system user, and writes your API key to /etc/aluminatai/agent.env.

pip install aluminatiai
curl -sSL https://aluminatiai.com/install.sh | bash

Dependencies:

CLI flags

Run with the aluminatiai command after install. Only ALUMINATAI_API_KEY is required.

# Minimal — reads API key from env
ALUMINATAI_API_KEY=alum_xxx aluminatiai

# 1-second sampling for high-accuracy energy profiling
aluminatiai --interval 1

# Run for 10 minutes, save CSV, no output
aluminatiai --duration 600 --output data/run.csv --quiet

Environment variables

Set these in your shell, a .env file in the repo root, or your systemd environment file. Only ALUMINATAI_API_KEY is required.

# .env file in repo root
ALUMINATAI_API_KEY=alum_YOUR_KEY_HERE
UPLOAD_INTERVAL=60
SAMPLE_INTERVAL=5.0
LOG_LEVEL=INFO

Running persistently

For production monitoring, run the agent as a background service so it survives SSH disconnects and reboots.

Option A — install.sh (easiest)

The installer handles everything: systemd unit, system user, environment file, and auto-start on boot.

pip install aluminatiai
curl -sSL https://aluminatiai.com/install.sh | bash
# Prompts for your API key, then:
# sudo systemctl status aluminatai-agent
# sudo journalctl -u aluminatai-agent -f

Option B — tmux (quick test)

tmux new-session -d -s aluminatai \
  "ALUMINATAI_API_KEY=alum_YOUR_KEY_HERE aluminatiai --quiet"

# Reattach to check logs
tmux attach -t aluminatai

Option C — systemd (manual setup)

Create /etc/systemd/system/aluminatai-agent.service:

[Unit]
Description=AluminatiAi GPU Energy Agent
After=network.target

[Service]
Type=simple
User=aluminatai
WorkingDirectory=/opt/aluminatai-agent
EnvironmentFile=/etc/aluminatai/agent.env
ExecStart=/usr/local/bin/aluminatiai
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

# Write your API key to the env file
sudo mkdir -p /etc/aluminatai
echo "ALUMINATAI_API_KEY=alum_YOUR_KEY_HERE" | sudo tee /etc/aluminatai/agent.env
sudo chmod 600 /etc/aluminatai/agent.env

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable aluminatai-agent
sudo systemctl start aluminatai-agent

# Check status
sudo systemctl status aluminatai-agent
sudo journalctl -u aluminatai-agent -f

Option D — Windows (PowerShell)

# Install then run
pip install aluminatiai
$env:ALUMINATAI_API_KEY = "alum_YOUR_KEY_HERE"
aluminatiai --quiet

# To run as a background job (Task Scheduler):
# Trigger: At startup
# Program: aluminatiai
# Arguments: --quiet
# Set ALUMINATAI_API_KEY in the task's environment variables

Reliability & WAL

The v0.2 agent is built for unattended production use. It handles network outages, API rate limits, and process restarts without losing metric data.

Write-ahead log (WAL)

When an upload fails, metrics are written to data/wal/metrics.wal as newline-delimited JSON. On the next startup the agent replays the WAL before resuming normal collection. The WAL is cleared only after all entries have been successfully uploaded.

Exponential backoff

Failed uploads are retried with exponential backoff: 1s → 2s → 4s → 8s → 16s, capped at 60s, with ±20% jitter. After UPLOAD_MAX_RETRIES (default 5) the batch is written to the WAL and the agent continues collecting new metrics.

Prometheus metrics

The agent exposes an HTTP metrics endpoint on localhost:9100/metrics (Prometheus format). Useful for integrating agent health into your existing monitoring stack. Set METRICS_PORT=0 to disable.

# Check agent health metrics
curl localhost:9100/metrics

Scheduler integration

The agent can enrich metrics with job attribution by polling your workload scheduler. When a job is detected on a GPU, the agent tags every metric from that GPU with job_id, team_id, model_tag, and scheduler_source.

# Manual attribution — wrap your training script
JOB_ID=llama3-finetune-v2 \
TEAM_ID=ml-infra \
MODEL_TAG=llama3-70b \
aluminatiai

Troubleshooting

# Ubuntu — install driver
sudo apt install nvidia-driver-535
sudo reboot

# Verify after reboot
nvidia-smi

# Check your groups
groups

# Add yourself to the video group
sudo usermod -a -G video $USER
# Log out and back in, then retry

pip install aluminatiai

# If using a venv, activate it first:
source venv/bin/activate && pip install aluminatiai

# Run with verbose logging
LOG_LEVEL=DEBUG aluminatiai

# Systemd logs
sudo journalctl -u aluminatai-agent -f

# Check WAL for buffered metrics
ls -lh data/wal/

# In PowerShell (not WSL):
pip install aluminatiai
$env:ALUMINATAI_API_KEY = "alum_YOUR_KEY_HERE"
aluminatiai