hivecast-sdk/packages/federation/DESIGN.md

# Design

This document captures the **full architecture** implemented in this repository.
It is written as a "soup-to-nuts" design spec so the implementation can be ported to other languages (e.g., the existing Matrix-3 TypeScript codebase) without losing intent.

## 1. Ubiquitous Language

### 1.1 Realm
A **realm** is the routing domain / tenant boundary (e.g., `matrix-3`, `flowpad-demo`, `flowpad-demo.tab-01`, `acme.com`).

- A realm may be a DNS name, an environment-qualified name, or a hierarchical dot-separated name.
- A realm **must be a single subject token** (i.e., it cannot contain `.` or `/`).

### 1.2 Path / toPath
A **path** (`toPath`, `fromPath`, `replyToPath`) is the internal semantic address of an atomic component method or event.

In the "Matrix-3" heritage this is represented as:

```
<mountSegments...>/<atom>

atom := <component.path>.<plane>.<message>
plane := accepts | emits
```

Examples:

- `project/daemon.project.accepts.scan`
- `root.ui.app.editor.accepts.setText`
- `root.push.emits.message`

### 1.3 Atomic component
An **atomic component** is the smallest independently addressable service endpoint, identified by an `atom` (component path + plane + message).

This matters because the federation layer must preserve atomicity: it routes to the exact atomic `accepts` handler, and forwards atomic `emits` events.

### 1.4 Mailbox
A **mailbox** is the realm's public ingress topic on the backbone transport.

- Legacy mailbox namespace: `public.ingress.<realm>`
- MX mailbox namespace: `mx/in/1/<realm>/<shard>`

The core property: **no wildcards required on the backbone**. Each realm subscribes to *its own* mailbox topic.

### 1.5 Backbone vs local bus
We separate two transports:

- **Backbone**: inter-realm federation transport (e.g., NATS cluster reachable from the public Internet).
- **Local bus**: intra-realm transport (component invocation within a single realm). This may allow wildcard subscriptions.

This is a deliberate "Ports & Adapters" (Hexagonal Architecture) decision:

- Federation code depends on an abstract `Transport` port.
- Concrete transports (in-memory, NATS, etc.) are adapters.

## 2. Protocol surface

### 2.1 MXURI/1 tolerant parsing
Supported forms:

1. **Legacy compact** (Matrix-3)

```
matrix://<realm>/<mount...>/<component.path>.<plane>.<message>
```

2. **Segmented** (new spec with explicit delimiter)

```
matrix://<realm>/<mount...>/-/<component/...>/<plane>/<message>
```

Both canonicalize to the same `(realm, mountSegments, componentPath, plane, messageName)`.

### 2.2 MXINGRESS/1
Ingress topics:

- `mx/in/1/<realm>/<shard>`

This implementation defaults `shard = "00"`.

### 2.3 Legacy ingress (Matrix-3)
Ingress topic:

- `public.ingress.<mxesc1(realm)>`

### 2.4 MXENV/1 envelope
The canonical envelope model is `MxEnvelopeV1`:

- `msgType`: `Cmd | Evt | Result | Sub | Unsub | Nack`
- nested routing objects `to`, `from`, `replyTo`
- `msgId`: stable dedup key
- `correlationId`: request/response correlation
- `principal` + `auth` placeholders

**TTL semantics**:

- `MxEnvelopeV1.ttl` is a *hop limit* for multi-hop forwarding (routing-level TTL).
- `CrossRealmEnvelopeLegacy.ttlMs` (when present) is an *expiry duration* in milliseconds relative to `timestamp`.
- This implementation preserves legacy `ttlMs` as-is and does **not** coerce it into `MxEnvelopeV1.ttl`.

### 2.5 Matrix-3 legacy envelope
Legacy profile supports:

- request: `CrossRealmEnvelopeLegacy` (flat fields)
- reply: `ReplyEnvelopeLegacy` (flat fields)
- optional legacy event: `EventEnvelopeLegacy` (extension point)

## 3. Layered architecture

### 3.1 Ports & Adapters

**Port**:

- `Transport` (`publish(topic, payload, meta?)` / `subscribe(filter, handler)`), where `meta` may carry transport-specific delivery options (e.g., QoS, retain).

**Adapters**:

- `InMemoryExactBroker`: exact-match topics only (simulates "no wildcards" backbone)
- `InMemoryFilterBroker`: NATS filter semantics (supports `*` and `>`)

The intent is pluggability via Strategy + Adapter:

- Swap in a real NATS adapter by implementing `Transport`.

### 3.2 Codec layer

- `mxesc1.ts`: deterministic percent-encoding for legacy mailbox compatibility
- `uri.ts`: tolerant MXURI parser
- `topics.ts`: mailbox/internal topic naming
- `envelope_codec.ts`: (de)serialization of both envelope profiles
- `wire_profile.ts`: profile registry (matrix3 vs mxenv1)

### 3.3 Federation edge

#### 3.3.1 `EdgeGateway`
The **EdgeGateway** is the "anti-corruption layer" between the backbone and the realm-local bus.

Responsibilities:

1. Subscribe to the realm mailbox topics (dual subscription for strangler migration).
2. Deserialize inbound payloads into a tagged envelope (matrix3 or mxenv1).
3. Apply inbound guard rails:
   - anti-open-relay (replyTo must target fromRealm)
   - optional replay cache (msgId dedup)
   - optional rate limiter
   - optional policy
4. Route inbound `Cmd/Sub/Unsub` to the **internal bus** by publishing an `InternalRequest`.
5. Subscribe to the per-request internal reply topic, then emit `Result` / legacy reply on the backbone.
   - If no internal reply arrives within `internalTimeoutMs`, EdgeGateway emits an error (`Nack` for MXENV/1, `ok:false` reply for legacy) with code `MX.EDGE.TIMEOUT`.
6. Route inbound `Evt` to the internal bus (fire-and-forget publish).

Key property:

- Backbones remain **wildcard-free**, but the local bus can be wildcard-capable.

#### 3.3.2 Internal request/reply protocol
EdgeGateway routes a backbone command/subscription into the local realm by publishing an **internal request message** to:

- `mx/1/<localRealm>/<toPath>`

Wire-level request shape:

```ts
interface InternalRequestWire {
  correlationId: string;
  replyTo: string;          // full internal topic (string)
  params: unknown;          // decoded envelope body

  // Optional local meta (enabled by default)
  _mx?: {
    fromRealm: string;
    incomingProfile: 'matrix3' | 'mxenv1';
    msgId?: string;
    principalKey?: string;
    traceId?: string;
  };

  // Optional: attach the inbound envelope for debugging/adapters
  // (enabled via `includeInboundEnvelope`)
  envelope?: unknown;
}
```

Components reply by publishing **anything** to the `replyTo` topic.

Strict reply shape (recommended):

```ts
interface InternalReply {
  correlationId?: string; // optional (replyTopic is unique per request)
  ok?: boolean;           // optional (defaults to ok=true)
  body?: unknown;         // optional (defaults to the whole JSON value)
  result?: unknown;       // optional convenience (treated as ok=true, body=result)
  error?: { code: string; message: string; details?: unknown };
}
```

Lenient decoding (supported for integration convenience):

- If payload matches `InternalReply`, it is used as-is.
- If payload is an object with `{ ok: boolean, body?, error? }` but lacks `correlationId`, the gateway supplies the correlationId.
- If payload is an object with `{ result: X }`, the gateway treats it as `ok=true` and `body=X`.
- If payload is any other JSON value/object, the gateway treats it as `ok=true` and `body=<that value>`.
- If payload is not valid JSON, the gateway treats it as `ok=true` and `body=<raw string>`.

This is intentionally similar to Request/Reply messaging patterns (Enterprise Integration Patterns), but keeps the internal contract soft so existing handler systems can adapt with minimal churn.


### 3.4 Runtime components

#### 3.4.1 `FederationFacade`
Implements subscription leases and event forwarding.

- Handles internal RPC endpoints:
  - `_federation.accepts.subscribe`
  - `_federation.accepts.refresh`
  - `_federation.accepts.unsubscribe`

- Maintains subscription state:

```ts
{ id, sourceTopic, sinkRealm, sinkPath, expiresAtMs }
```

- Subscribes to each source topic on the local bus.
- On event publish, wraps as `MxEnvelopeV1` with `msgType='Evt'` and publishes to sink realm mailbox.

This maps to the EIP patterns:

- "Message Channel" (NATS subjects)
- "Message Router" (forwarding)
- "Idempotent Receiver" (lease refresh)

#### 3.4.2 `SessionDirectory` + `SessionsFacade`
Implements a home-realm directory mapping `principalKey → [tabRealm...]`.

- Prevents spoofing: tabRealm is derived from the envelope's `fromRealm` (via internal meta), not from request body.
- Supports:
  - `sessions.accepts.register`
  - `sessions.accepts.heartbeat`
  - `sessions.accepts.lookup`

This enables unsolicited push:

1. Agent looks up principal tabs in home realm.
2. Agent sends `Evt` to each tab realm mailbox.
3. Tab realm EdgeGateway delivers event to local bus.

#### 3.4.3 `RouterBridge`
Implements a hierarchical routing overlay (MXROUTE-style).

- Each realm has a route inbox:
  - `mx/route/1/<realm>/<shard>`
- Routers form a tree (parent + child prefixes).
- Messages are forwarded upward until reaching a router whose subtree contains the destination, then forwarded down.
- TTL (`env.ttl`) is decremented at each hop.

This is a classic POSA-style "Broker" / "Forwarder" pattern combined with EIP "Routing Slip" semantics.

#### 3.4.4 `LocalRequestReplyClient`
Provides intra-realm request/reply on the internal bus (useful for tests and as a reference).

### 3.5 Security (MVP scaffolding)

Security in this repo is explicitly MVP-level but structurally correct:

- `principal.ts`: derives a stable `principalKey`
- `policy.ts`: policy port, plus a prefix-deny example policy
- `ratelimit.ts`: token bucket limiter
- `replay.ts`: replay cache

The *pattern intent* is the important part:

- **Chain of Responsibility** for inbound checks.
- All checks are concentrated in `EdgeGateway`.

## 4. Compatibility strategy

### 4.1 Strangler Fig dual subscription
Both client and server subscribe to:

- legacy mailbox (`public.ingress.<realm>`)
- mx mailbox (`mx/in/1/<realm>/00`)

Which encoding is used on publish is controlled by `WireProfile.encode`:

- `matrix3` profile publishes legacy by default
- `mxenv1` profile publishes MXENV/1 by default

### 4.2 Tolerant URI parsing
MXURI parsing accepts both legacy and segmented representations.

## 5. Test suite & coverage matrix

Tests are split into unit + integration.

### 5.1 Unit tests

- `tests/unit/mxesc1.test.ts`: encoding/decoding
- `tests/unit/topics.test.ts`: mailbox + internal topic derivations
- `tests/unit/uri.test.ts`: tolerant MXURI parsing
- `tests/unit/topic_filter.test.ts`: wildcard matching semantics

### 5.2 Integration tests

- `tests/integration/request_reply.test.ts`
  - local-to-local (internal bus)
  - public-to-realm (matrix3 legacy)
  - public-to-realm (mxenv1)
  - confirms backbone subscriptions contain no wildcards

- `tests/integration/subscription.test.ts`
  - subscribe / refresh / unsubscribe
  - event forwarding from source realm local bus → backbone → sink realm local bus
  - lease expiry behavior

- `tests/integration/directory.test.ts`
  - register + heartbeat + lookup
  - spoof prevention (tabRealm derived from fromRealm)
  - expiry cleanup
  - unsolicited push via Evt envelopes

- `tests/integration/discovery.test.ts`
  - direct preferred when available
  - fallback to backbone when direct publish fails
  - verifies the "either direct or public transport" requirement

- `tests/integration/router.test.ts`
  - hierarchical routing overlay end-to-end
  - TTL exhaustion drop
  - unknown route drop

- `tests/integration/security.test.ts`
  - anti-open-relay drop
  - msgId dedup via replay cache
  - rate limiting emits Nack
  - policy denial emits Nack

## 6. What is intentionally *not* implemented

This repo is a protocol-focused reference; a production deployment would add:

- Real NATS adapter (`Transport` implementation using a real client)
- MXDISC/1 discovery (`/.well-known/...`) and endpoint negotiation
- MXSEC/1 JWT/JWKS validation (cryptographic verification)
- Durable storage for directory and subscriptions
- Distributed replay cache

The abstractions are intentionally shaped so these can be plugged in without rewriting core logic.