Monitoring

Prometheus metrics, packet logs, and live packet stream

Graywolf provides several monitoring interfaces for observing station activity and diagnosing issues.

The packet log viewer with live streaming, search filtering, and CSV export

Prometheus Metrics

Graywolf exposes a Prometheus-compatible /metrics endpoint on the HTTP port. Point your Prometheus scrape config at it to collect station telemetry.

scrape_configs:
  - job_name: graywolf
    static_configs:
      - targets: ['graywolf-host:8080']

Available Metrics

Understanding "Bad FCS"

Every AX.25 frame ends with a two-byte Frame Check Sequence (FCS) — a CRC‑16/CCITT checksum the sender computes over the packet. The receiver recomputes it and compares. The dashboard's Bad FCS counter (and the graywolf_rx_bad_fcs_total metric) counts frame-shaped signals that the modem demodulated all the way to a closing HDLC flag but whose checksum did not match — so the frame was rejected and dropped rather than passed on.

Only plausible candidates are counted: the demodulator must have found an opening and closing flag with at least a minimum-length packet of bits between them, and the frame must have failed both the direct check and every configured bit-fix retry. Pure noise, runts, and aborts are not counted. In other words, Bad FCS represents real HDLC-shaped energy on the channel that almost decoded but didn't.

What counts as a "high" value?

There is no fixed threshold, because a healthy value depends entirely on what your receiver hears. The two things that matter are the ratio of bad to good frames and the trend over time — not the raw number.

So a station seeing, for example, ~4,500 bad versus ~1,000 good frames is not automatically in trouble — on a busy APRS receiver that ratio is plausibly just collisions and weak DX. The question to answer is whether the good count is healthy for your location and whether the audio is clean. Use the steps below to decide.

Diagnosing poor FCS performance

1. Check the good-frame rate alongside it. If RX is still climbing at a normal pace for your area, you are mostly hearing collisions and marginal traffic — benign. If good frames have stalled while Bad FCS keeps rising, treat it as a local fault.
2. Look at the channel's audio level. On the dashboard, the per-channel level bar should peak comfortably without pinning at the top (clipping) or barely moving (too weak). Both extremes wreck the FCS rate.
3. Record a clip and decode it offline. Capture a short .wav of live traffic and run it back through the real demodulator:

   graywolf-modem --decode rx.wav

   Watch rx_bad_fcs versus rx_frames, and check level_dbfs_med (near 0 dBFS means clipping; very low means too weak) and twist_db_med (a large twist points at a de-emphasis / audio-response problem rather than overall level). See Audio Devices for the full field list.
4. Tune against the same clip. Because --decode is deterministic on a fixed file, decode the same recording repeatedly while adjusting receive gain; the setting that yields the most rx_frames with the fewest rx_bad_fcs and no clipping wins.
5. Hunt for interference. If the audio is clean but Bad FCS stays high with few decodes, suspect EMI — move USB cables, sound interfaces, and switching supplies away from the radio and antenna feed.

Packet Log API

The packet log maintains a ring buffer of the most recent packets (up to 1000 entries, 30-minute age limit). Query it via the REST API or view it in real time through the web UI.

Packet Entry Fields

Health Check

A lightweight health endpoint is available for load balancers and monitoring systems:

GET /api/health

{
  "status": "ok",
  "time": "2026-04-11T10:00:00Z",
  "started_at": "2026-04-11T08:30:00Z"
}

Status Endpoint

The status endpoint returns aggregated dashboard data in a single request — uptime, per-channel stats (RX/TX counts, DCD state, audio peak), and iGate connection status:

GET /api/status

pprof Debug Profiles

Graywolf can expose Go's net/http/pprof endpoints on a separate listener for diagnosing memory growth, goroutine leaks, and CPU hotspots. The listener is off by default; enable it by passing -pprof <addr>.

graywolf -pprof 127.0.0.1:6060

Common captures

# Heap snapshot (live allocation)
curl -o heap.prof http://127.0.0.1:6060/debug/pprof/heap

# Goroutine stack dump (human-readable)
curl http://127.0.0.1:6060/debug/pprof/goroutine?debug=2

# 30 s CPU profile
curl -o cpu.prof http://127.0.0.1:6060/debug/pprof/profile?seconds=30

# 5 s execution trace
curl -o trace.out http://127.0.0.1:6060/debug/pprof/trace?seconds=5

# Interactive web UI for any profile
go tool pprof -http=:8081 heap.prof

Memory leak workflow

1. Capture a baseline heap profile after warmup (curl -o base.heap ...).
2. Wait through a representative workload window (30+ minutes).
3. Capture a second profile (later.heap).
4. Diff with go tool pprof -http=:8081 -base base.heap later.heap and look at the inuse_space view. The diff highlights what grew between snapshots, which is what a leak looks like.

Note: the Rust modem child process is a separate PID and is not visible to Go pprof. Watch its memory through process_resident_memory_bytes on the Prometheus side or ps -o rss against the modem PID.

Logging

Graywolf outputs structured logs to stderr. By default, logs are at INFO level. Enable verbose logging with the -debug flag.

When running under systemd, logs are captured by the journal and queryable with:

journalctl -u graywolf -f