SysMon AI

Overview

Most "ML monitoring" projects are an Isolation Forest with a CSV loader bolted to the front. The interesting part of monitoring is not the model — it is everything around it: collecting telemetry without becoming the load you are measuring, persisting it durably, turning raw samples into features that mean something, deciding what counts as abnormal, and saying so in a way a human acts on.

SysMon is the whole pipeline. It runs on the machine it watches and never sends a metric anywhere.

Context

I wanted local observability on my own machines without routing host telemetry to a hosted service. That constraint — everything stays local — drives most of the design. There is no server to offload scoring to, so inference has to be cheap. There is no external store, so persistence has to be durable and bounded on disk.

My role

Sole author. Design, implementation, the synthetic evaluation harness, and the tests.

Constraints

• Continuous operation on a laptop, so overhead has to stay near the noise floor
• Bounded memory regardless of how long it runs
• Durable local persistence that survives an unclean shutdown
• A model whose sensitivity is a decision, not an accident
• A terminal interface dense enough to be useful and calm enough to be read

Architecture

Collection → bounded queue → SQLite (WAL) → window features → anomaly scoring → calibration → forecasting → alert engine → terminal UI.

psutil samples CPU, memory, disk and network. Samples go onto a bounded writer queue, which is the part that matters: when the writer falls behind, the queue drops samples and counts them, rather than growing until the monitor becomes the problem. Dropped-sample accounting is exposed, because a monitor that silently loses data is worse than no monitor.

Persistence is SQLite in WAL mode with synchronous=NORMAL, memory temp storage, a 64 MB cache and a 256 MB mmap. WAL lets the writer commit without blocking readers, which is what keeps the dashboard responsive while collection continues.

Data and features

Raw gauges are nearly useless for anomaly detection — 80% CPU means nothing without knowing whether it has been 80% for an hour or arrived in two seconds. The feature pipeline builds lags, short- and long-window rolling means and standard deviations, exponential moving averages, slopes, and I/O burstiness, so the model sees change rather than level.

Key decision: calibrate to a false-positive rate, not a score threshold

An Isolation Forest emits an unbounded anomaly score with no natural cut-off. Exposing that score, or picking a threshold by eye, produces a monitor whose sensitivity nobody understands.

Instead, the threshold is calibrated against a chronological slice of known-normal history to hit a target false-positive rate. That turns an opaque tuning knob into a statement an operator can reason about: how often am I willing to be interrupted for nothing? The chronological split matters — calibrating against a random slice would leak future behaviour into the threshold.

Key decision: keep persistence and inference local

No metric leaves the machine. This costs the ability to correlate across hosts, and it means the model only ever learns one machine's idea of normal. Both are acceptable trades for a personal-machine monitor, and they are the reason the project exists.

Reliability

Batched writes amortise transaction cost. The bounded queue provides backpressure. Dropped samples are counted and surfaced. Retention bounds the database.

Evaluation

Synthetic, and labelled as such everywhere it appears.

The generator produces baseline periodic CPU behaviour, gradually growing memory, log-normally distributed I/O and network activity, and injected isolated spikes. The documented setup is 100,000 training observations and 20,000 test observations with 5% injected anomalies.

Separately, a 24-hour run on a MacBook Pro M1 (8-core, 16 GB) reported 1.8% CPU overhead, 120 MB RSS, 6 ms p95 write latency, 180 ms dashboard refresh, and zero dropped samples.

Both sets of numbers describe exactly what they measured. The first is generated data. The second is one machine.

Design constraints

• The benchmark is synthetic by construction. The generator produces baseline periodic CPU behaviour, growing memory, log-normal I/O and injected spikes, which is what makes the evaluation reproducible on any machine.
• Forecast bands are residual-based, which keeps threshold forecasting cheap enough to run continuously in the collection loop.
• Calibration is only as representative as its window, which is why the calibration slice is chronological rather than random.
• Overhead figures are machine-specific — a MacBook Pro M1, 8-core, 16 GB, over 24 hours.

Where I would take this next

• Replay evaluation against recorded real traces, so the headline figure carries operational weight as well as reproducibility.
• Time-aware alert utility metrics — lead time weighted by whether the alert was actionable — rather than raw classification accuracy.
• Calibrated forecast uncertainty, upgrading the residual bands to true probabilistic intervals.
• Drift detection on the calibration window, so "normal" updates when the machine's workload genuinely changes.
• Verified overhead figures across more platforms, extending the single-machine profile into a range.