Monitoring

Prometheus metrics, packet logs, and live packet stream

Graywolf provides several monitoring interfaces for observing station activity and diagnosing issues.

The packet log viewer with live streaming, search filtering, and CSV export

Prometheus Metrics

Graywolf exposes a Prometheus-compatible /metrics endpoint on the HTTP port. Point your Prometheus scrape config at it to collect station telemetry.

scrape_configs:
  - job_name: graywolf
    static_configs:
      - targets: ['graywolf-host:8080']

Available Metrics

Packet Log API

The packet log maintains a ring buffer of the most recent packets (up to 1000 entries, 30-minute age limit). Query it via the REST API or view it in real time through the web UI.

Packet Entry Fields

Health Check

A lightweight health endpoint is available for load balancers and monitoring systems:

GET /api/health

{
  "status": "ok",
  "time": "2026-04-11T10:00:00Z",
  "started_at": "2026-04-11T08:30:00Z"
}

Status Endpoint

The status endpoint returns aggregated dashboard data in a single request — uptime, per-channel stats (RX/TX counts, DCD state, audio peak), and iGate connection status:

GET /api/status

pprof Debug Profiles

Graywolf can expose Go's net/http/pprof endpoints on a separate listener for diagnosing memory growth, goroutine leaks, and CPU hotspots. The listener is off by default; enable it by passing -pprof <addr>.

graywolf -pprof 127.0.0.1:6060

Common captures

# Heap snapshot (live allocation)
curl -o heap.prof http://127.0.0.1:6060/debug/pprof/heap

# Goroutine stack dump (human-readable)
curl http://127.0.0.1:6060/debug/pprof/goroutine?debug=2

# 30 s CPU profile
curl -o cpu.prof http://127.0.0.1:6060/debug/pprof/profile?seconds=30

# 5 s execution trace
curl -o trace.out http://127.0.0.1:6060/debug/pprof/trace?seconds=5

# Interactive web UI for any profile
go tool pprof -http=:8081 heap.prof

Memory leak workflow

1. Capture a baseline heap profile after warmup (curl -o base.heap ...).
2. Wait through a representative workload window (30+ minutes).
3. Capture a second profile (later.heap).
4. Diff with go tool pprof -http=:8081 -base base.heap later.heap and look at the inuse_space view. The diff highlights what grew between snapshots, which is what a leak looks like.

Note: the Rust modem child process is a separate PID and is not visible to Go pprof. Watch its memory through process_resident_memory_bytes on the Prometheus side or ps -o rss against the modem PID.

Logging

Graywolf outputs structured logs to stderr. By default, logs are at INFO level. Enable verbose logging with the -debug flag.

When running under systemd, logs are captured by the journal and queryable with:

journalctl -u graywolf -f