@open-matrix/system-observability (1.1.32)
Installation
@open-matrix:registry=npm install @open-matrix/system-observability@1.1.32"@open-matrix/system-observability": "1.1.32"About this package
System Observability
Hosted telemetry for minimal installations
Fresh homes do not need a local collector runtime. Their existing
system.observability/$ingest publisher crosses an exact, activation-gated NATS
service import into the shared services collector. The broker-delivered subject
selects the tenant; producer root, source, trace IDs and query filters cannot
select another tenant's records. The existing bounded sink and retention policy
also apply to hosted telemetry.
Query the hosted store explicitly with
matrix invoke system.observability.hosted.tracing trace.query '{"requestId":"…"}'
or the other existing trace.* operations. Check
matrix invoke system.observability.hosted '$status' '{}' first: an available
query surface with no persisted tenant records reports unknown / no-data,
not healthy. Loss counters are explicitly collector-wide; the latest producer
timestamp is not a collector receipt timestamp or proof of current freshness.
An unreachable responder is unavailable, not an empty result.
The original system.observability.tracing address remains local. This avoids
competing answers when a home also installs its own collector. No account gets
access to the services authority's unscoped store.
Telemetry ingest queues every frame, including terminal phases, instead of writing SQLite on the transport dispatch stack. The sink retains at most 1,000 pending frames and writes at most 50 per transaction, matching the existing telemetry publisher's queue and batch bounds. The first batch uses the existing 50 ms flush window; backlog continuations yield through timers between batches. Overflow drops the oldest pending frame. Failed writes drop their batch without retries, and the sink exposes local pendingFrames and droppedFrames counters. Disposal flushes the bounded remainder before closing storage and ignores later arrivals.
The derived SQLite store uses busy_timeout = 0: a competing writer causes immediate shedding instead of blocking the runtime's heartbeat thread. The standalone regression boots the real package with in-memory transport and real SQLite, sends 50 completed frames per second, and introduces a second SQLite writer. Before the change, a fixture lock wait shortened to 100 ms held $ingest for 100.948 ms; afterwards the same dispatch took 0.041 ms against the unchanged 50 ms budget. The queue regression measured a maximum 4.912 ms per 50-row batch with 1 KiB summaries and a 0.225 ms busy write. These are local measurements, not a diagnosis of the staging login failures. SQLite statements and filesystem I/O remain synchronous; batching bounds admitted rows and removes lock waiting, but cannot preempt a slow filesystem operation.
The package stores derived runtime telemetry in SQLite and performs retention on one service-owned worker connection, away from actor replies. The shipped policy uses a 25 ms work-admission budget, 128-row delete units, and 128-page vacuum slices. SQLite statements remain synchronous on the maintenance thread: the deadline is checked between operations, while the admitted operation, closing checkpoint, and final statistics read must finish. elapsedMs measures the complete maintenance pass and can exceed 25 ms. A refused writer lock retries at the existing continuation delay; actor replies never wait for that lock.
The worker prepares existing-store migrations before the reply connection opens, admits only one pass, reuses its connection through startup and continuation bursts, and retires after the next slice's delay plus measured startup cost. Disposal closes it and cancels further passes. A transactional SQLite counter replaces the connection-local row cache, so committed appends and pruning remain visible to both connections without repeating a full-table count. Both compiled entrypoints and schema.sql are recorded in the package's existing artifact metadata.
During an owned maintenance pass, the existing 1,000-frame queue holds incoming writes and resumes its 50-row batches afterwards. Optional legacy log capture uses the same queue. Disposal waits for that maintenance connection to close before flushing the remainder. Overflow and unrelated writer failures retain their existing loss accounting; they are never hidden behind a healthy collector status.
After building the package, run node tools/fleet/observability-retention-benchmark.mjs --max-query-ms 50 from the repository root. This disposable native SQLite/NATS fixture seeds 200,000 rows, issues queries from an independent worker client while continuously ingesting records, and reports latency, startup, CPU/RSS, retention and loss counters for 25 ms and 100 ms admission budgets. It requires nats-server on PATH and never attaches to an installed environment. --artifact <package-directory> selects a compiled baseline; --seed <database-directory> clones an existing fixture for matching before/after inputs.
The delete unit was measured on this machine against the production schema populated with 440,000 representative rows (377,745,408-byte database), 20 samples per size. A 2,000-row delete took 71.299 ms median, 94.510 ms p95, and 167.092 ms maximum, so it cannot fit the 25 ms admission budget. The shipped 128-row unit took 2.538 ms median, 6.163 ms p95, and 7.021 ms maximum; 256 rows already reached 39.927 ms. The shipped-configuration regression reads ObservabilityRoot.config.json, drives the real SQLite store through the adaptive ceiling, and asserts that every admitted delete uses the measured 128-row unit.
When backlog remains, the scheduler doubles the batch depth from one to two, four, and the configured ceiling of eight. That ceiling admits at most 1,024 deleted rows per deletion phase, subject to the shared deadline. With the shipped 128-row unit and ceiling, 20 representative whole passes deleted 512–768 rows each and took 41–60 ms, including checkpoint and statistics. The 1,000 ms continuation delay therefore yielded at least 1,000 ms after each measured burst (at most 5.7% observed maintenance duty cycle) while sustaining roughly 483–738 expired rows per second. A configured ceiling below the base depth is normalized to the base, and the scheduler resets to its configured base depth after age, size, and vacuum backlog converge.
Retention emits observability.retention.pruned when pressure starts or recovers: age, size, vacuum-backlog, or exhausted-budget state. Productive continuation passes with the same state do not add host-log records.
The continuous-ingest regression writes 30 expired synthetic rows per pass into a temporary SQLite store and verifies the bounded 1 → 2 → 4 → 8 adaptation, unchanged admission deadline, and convergence of both expired rows and vacuum backlog while arrivals continue. Run the package regressions with pnpm --filter @open-matrix/system-observability test.
Trace profiling without blocking the collector
The standard collector’s trace.timeProfile executes its bounded SQLite read and report on a separate read-only instance of the same retiring worker executor. One profile is admitted at a time; another receives TRACE_PROFILE_BUSY, and a stopped or failed owner returns TRACE_PROFILE_UNAVAILABLE without exposing storage errors. Profiling does not pause ingestion or compete for the writer lock. Disposal cancels outstanding profile work and does not publish a late hosted response.
Callers constructing TracingActor(store) directly keep the existing in-process query API. Collector factories supply the worker executor.
The broker-delivered tenant selects the worker's existing TenantRecordStore view. Caller options cannot choose a tenant, and the hosted 1,000-record cap remains; the collector's own profile keeps its 20,000-record bound. Gap attribution is computed only for the twenty gaps displayed, preserving report ordering and neighbor semantics without rescanning every record for every hidden gap.
Run node tools/fleet/observability-retention-benchmark.mjs --operation profile --max-query-ms 50 after the owner build. The same isolated native harness issues a profile while another NATS client continues ordinary queries and ingestion; it requires every emitted frame to persist. The retiring worker adds temporary memory and cold-start cost. The benchmark reports those costs rather than asserting a machine-wide memory or CPU saving.
Request rates and platform alerts
health.component derives each service's five-minute request and error counts
from matrix.request spans, grouped by their producing runtime mount. The SDK
request/reply dispatch pipeline is the one boundary every actor request crosses,
so it increments the count once and marks an error from the completed span status. The response includes requests per
minute and the error fraction alongside the existing latency and health data.
alerts.rules exposes the platform-owned defaults to operators:
- disk free below 10%, leaving recovery room before a full disk stops writes;
- three service restarts in 15 minutes, separating a restart loop from one recovery;
- service errors above 5% over five minutes after at least 20 requests, avoiding low-volume noise;
- no machine heartbeat for more than 45 seconds, matching the existing three-heartbeat offline horizon.
Every HiveDock evaluates its local machine and service samples every 15 seconds.
The home observability actor also evaluates enrolled-device last-seen values so
a HiveDock that has stopped reporting can raise the silent-machine rule. A
firing edge reconciles the stable alerts:<sha256> source into one current
needs-you item; repeated firing evaluations do not add items. The clearing edge
removes that current item and source marker, allowing a later breach to fire
again. Host Control marks its bounded runtime list as a complete service
snapshot, so alerts for services that disappear are cleared. Active edges and
canonical facts are both bounded to 1,024 target/rule entries per evaluator.
The wire declares separate exact capabilities for alerts.evaluate,
notification.alert.reconcile, and the notification owner's
alerts.notification.get callback. That callback is served by the dedicated
read-only system.observability.alert-source actor. Its exact inbox crosses
the tenant/services account wall; the Health inbox is never exported.
The resource trend used by Machines is the HiveDock's bounded one-hour history. The derived observability store has separate longer retention: seven days by default, six hours for spans, and a 6 GiB size ceiling pruned toward 5 GiB.
Dependencies
Dependencies
| ID | Version |
|---|---|
| @open-matrix/contracts | 1.1.32 |
| @open-matrix/core | 1.1.32 |
| @open-matrix/subjects | 1.1.32 |
| @open-matrix/system-sqlite | 1.1.32 |
Development Dependencies
| ID | Version |
|---|---|
| @matrix/test-utils | 1.1.32 |
| @open-matrix/nats-auth | 1.1.32 |
| @types/node | ^20.0.0 |
| esbuild | ^0.25.0 |
| nats | ^2.29.3 |
| tsx | ^4.22.4 |
| typescript | ^5.7.0 |