Frame Processing Pipeline

The orchestrator's FrameProcessingPipeline (app/pipeline/frame_pipeline.py) processes each frame through 17 stages: from JPEG decode through person detection, privacy enforcement, spatial floor projection, ReID and pose inference, face identification, world tracking (which includes pre-association cross-camera dedup and Bayesian identity resolution), detection backfill, PH lifecycle management, posture classification, trajectory writing, keyframe sampling, identity revision, trail management, durable provenance persistence, governed ReID candidate creation, and event publishing. (A conditional fall-detection stage also runs between posture classification and trajectory writing when fall_detection.enabled; it is omitted from the count and diagram here because it is off by default.)

1. Frame fetch

JPEG frames are fetched from MinIO using the key published in the FrameReady proto message. Frames older than 30 s are dropped as stale replay backlog (they are XACK'd normally to keep the pending-entry list clean, but pipeline work is skipped).

The fetched image is compared against the FrameReady-reported dimensions. A mismatch (e.g., from EXIF rotation or thumbnail storage) is logged and the actual image shape is used for all downstream coordinate transforms.

2. Person detection

YOLO26L ONNX runs on Triton, returning normalized bounding boxes ([x1, y1, x2, y2] in [0, 1] space) with confidence scores. Detections below pipeline.detector_confidence (default 0.7) are discarded.

Post-decode IoU deduplication suppresses near-duplicate detections that survive the model's baked NMS. A greedy algorithm processes boxes in descending confidence order: a box is suppressed if its IoU with any already-kept box exceeds detection_iou_dedup_threshold (default: 0.55). This is belt-and-braces: the ONNX model already applies NMS, but double detections still occur at scene boundaries.

3. Privacy zone enforcement

Operator-drawn privacy polygons (configured at /admin/cts/privacy) are applied in two passes:

• Blur/mask: pixels inside privacy zones are obscured in the frame before any further processing. This affects crops, keyframes, and the published frame URL.
• Detection drop: detections whose foot-point (bbox bottom-center) falls in a drop zone are discarded before tracking. A Prometheus counter (privacy_detections_dropped_total) tracks the drop rate per camera.

4. Spatial projection

SpatialProjectionStage computes the calibrated floor-plane coordinate (FloorPoint) for every surviving detection using the per-camera 3x3 homography matrix fitted by the calibration tool. The result is carried on the detection's floor_point field with calibrated=True for cameras that have been calibrated, and calibrated=False with (0, 0) coordinates for uncalibrated cameras.

Floor projection runs before ReID and pose inference so that the cross-camera dedup pass (inside the world tracker) can use per-detection floor positions at the time of association.

5. ReID and pose inference

Per-detection person crops are extracted at native resolution and sent in parallel to two Triton models:

• SOLIDER-REID (768-dimensional L2-normalized embedding): used by the world tracker for cross-camera appearance similarity and by the Bayesian identity resolver for gallery k-NN search.
• RTMPose (17 COCO keypoints): used by the posture classifier and the motion energy tracker. Crops smaller than 16x32 pixels are skipped.

If the ReID embedder is unavailable, an empty embedding list is returned and the tracker falls back to geometry-only association.

6. Face identification

Face identification sends one crop per person to POST /api/v1/identify on the person-identification-service, which returns the faces it finds with an identity, a recognition state, and a raw similarity score.

Face crop. The face path cuts its own crop from the frame: the person box padded upward by 15% and sideways by 10% of its size (never downward), so a head at the top edge of the detector box keeps the context face detection needs. Privacy zones are already blurred or masked in the frame before any crop is taken, so padding cannot expose them. Crops are encoded at JPEG quality 95. ReID and pose keep using the tight person crop.

One face per person. A person crop can contain someone else's face when two people overlap. A face is attributed to the person only if its centre lies in the person's head region: the top 35% of the person box (plus the upward padding), widened by 10% of the box width on each side. If two faces fall in that region, the crop is ambiguous and produces no face evidence; an unknown identity is safer than a swapped one. Each resulting face anchor is attached to the observation of the detection it came from, never to another person seen by the same camera.

Setting (face_id.*)	Default	Description
cooldown_s	5.0	Minimum seconds between queries for the same tracked person on a camera
timeout_s	2.0	HTTP request timeout
min_raw_similarity	0.6	Recognized faces below this raw similarity are dropped
head_region_height_frac	0.35	Share of the person box, from the top, that can hold the face
head_region_side_margin_frac	0.10	Horizontal tolerance per side, as a share of box width
crop_pad_top_frac / crop_pad_side_frac	0.15 / 0.10	Face crop padding
crop_jpeg_quality	95	JPEG quality of face crops
enabled	true	Master switch
Per-camera overrides	camera_id to FaceIdCameraConfig	Enable/disable per camera, override the minimum similarity

Face identification can be disabled per camera (for example, top-down views where faces are never visible). Each FaceAnchor carries person_id, match_score (raw cosine similarity for recognized and candidate faces, the face detection score for an unrecognized marker), det_score, quality (the evidence-path weight: 1.0 for a face seen in this frame, lower for propagated or external evidence), face_quality (the person-ID service's image-quality score, null from an older service), calibrated_confidence (null unless calibration is ready), detection_id, and camera_id. The identity resolver weights face evidence 3x over ReID. Every face result is also recorded for measurement; see Measuring your cameras.

7. World tracking

WorldTrackingStage builds WorldObservation objects for each detection (using a synthetic virtual-tile floor point when a camera is uncalibrated), then calls WorldTracker.step. The tracker runs 10 numbered steps:

Steps 1-3: predict and associate. All open PHs are loaded and their Kalman states predicted forward to the current timestamp. dedup_observations() then collapses detections from different calibrated cameras that are within dedup_max_distance_m (default: 0.6 m) and not in identity conflict into single representative observations. The Hungarian algorithm solves the assignment over a cost matrix (geometric + appearance + height) with a Mahalanobis gate (gate_chi2 = 9.21, chi-squared 99%, 2 dof) and an identity-conflict hard gate.

Steps 4-5: update and spawn. Matched PHs receive a Kalman update, gallery-mean EMA (alpha = 1 / min(count+1, 100)), height EMA (alpha = 0.1), mean-quality EMA (0.1 * obs.quality + 0.9 * prev), and accumulate all contributing camera IDs. Unmatched observations spawn new PHs (gated by room polygon when configured), seeded with face anchor identity if present.

Steps 6-7: continuations and grace. For each spawn, recently closed PHs within inferred_handoff_max_s (600 s) and inferred_handoff_max_distance_m (5.0 m) emit a PHContinuationCandidate. Unmatched PHs past ph_close_grace_s (5.0 s) are closed; those within the grace window keep their predicted state.

Steps 8-10: persist, resolve, snapshot. All updated PHs are persisted. The Bayesian identity resolver runs on every PH that received an observation. WorldFrameSnapshot objects are built for PHs with observation_count >= min_observations_to_publish (3).

Pre-association cross-camera dedup

Before the Hungarian assignment runs, dedup_observations() (app/tracking/world/dedup.py) collapses detections from different cameras that correspond to the same physical person into a single representative observation.

The dedup gate passes a pair of observations when all three conditions hold:

1. The two detections are from different cameras.
2. Both have calibrated=True floor points within dedup_max_distance_m (default: 0.6 m) of each other.
3. When both carry committed face anchors, the face identities do not conflict (dedup_require_no_face_conflict: true).

Pairs that pass the gate are joined by a union-find algorithm; each resulting cluster elects one representative via _select_representative() (highest quality, ties broken deterministically). The cluster membership map propagates all source camera IDs back to the winning PH.

Config knob	Default	Description
dedup_enabled	true	Master switch for pre-association dedup
dedup_max_distance_m	0.6	Floor-plane distance gate in metres
dedup_require_no_face_conflict	true	Do not collapse observations with conflicting face identities

Uncalibrated cameras: dedup_observations() skips observations with calibrated=False. For uncalibrated home cameras, identity crosses cameras only through the shared ReID gallery, cross-GT face propagation, and PH continuation.

The integration proof lives in tests/integration/test_world_tracker_e2e.py::test_hallway_bathroom_one_person.

PersonHypothesis (PH) management and identity resolution

The WorldTracker maps confirmed per-camera tracks to PersonHypothesis (PH) entities. Each PH:

• carries a ph_id (UUID, the single physical-track identifier on the wire after R3).
• maintains a mean_quality field (exponential moving average of per-observation quality scores, added in migration 0002_quality).
• is the entity that receives Bayesian identity resolution.

Quality capture. Each observation's quality score is computed from the detection's confidence and the crop size, then used to update the PH's mean_quality. The same CropQuality scorer is used for observations as for gallery entries; there is no parallel scorer. The mean_quality field travels to CC via the IdentitySnapshot proto and then through the quality field on PersonLocationEnvelope.

Bayesian identity resolution. The IdentityResolver maintains a posterior probability distribution for each PH over {known_identities u UNKNOWN}. Three evidence sources are combined via pointwise multiplication, with missing sources treated as uniform (weight = 1.0):

posterior(identity) proportional to prior(identity) x face_likelihood(identity) x reid_likelihood(identity)

Temporal prior. Encodes continuity from the previous identity assignment. The previous MAP identity receives prior_weight = 0.6 of the probability mass. Identity persists for prior_maintenance_max_age_s (30 s) without new evidence; beyond that window, the prior alone cannot sustain a commit.

Face likelihood. The strongest recognized anchor contributes p_face = 0.5 + 0.5 * tanh(4 * (0.7 * match_score + 0.3 * q - 0.5)), capped at 0.99. This is an uncalibrated heuristic, not a probability fit to data. q is the anchor's quality; with resolver.face_quality_mode: enabled it is multiplied by the service's face quality, so a small, blurred, or steeply pitched face counts less. The default in settings.yaml is shadow: decisions use quality alone while a quality-weighted evaluation counts disagreements in cts_identity_shadow_mismatch_total{feature="face_quality"}. Face evidence receives a face_weight_multiplier of 3.0 over ReID evidence: ArcFace is more reliable than body appearance for disambiguating identities in multi-person households.

ReID likelihood. Gallery k-NN search (pgvector HNSW with StreamingDiskANN) retrieves similar embeddings and maps them to identities via a logistic curve.

Commit rule:

commit if: top_prob >= commit_prob (0.65)
          AND margin >= commit_margin (0.15)
          AND sensory_evidence_present (face or ReID, not prior alone)

A calibrated direct face match at or above arcface_authority_calibrated_confidence (0.80) grants direct-face authority; without a deployed calibration artifact this path stays closed.

Identity feedback loop. When a face anchor identifies a person on one camera, the resolved identity is stamped onto all gallery entries for that PH's detections, creating a self-reinforcing cycle that allows cross-camera identity carry even when a face is not visible on the second camera.

8. Detection backfill

DetectionBackfillStage enriches each detection in ctx.domain_detections with the ph_id assigned by the world tracker. This ensures downstream stages (posture, trajectory, publish) all read the same identity from the detection object rather than maintaining separate maps.

9. PH lifecycle (ClosePHStage)

ClosePHStage closes any PersonHypothesis that was active in the previous frame but no longer has an active detection in this frame. It writes the final trajectory and dwell segments for closed PHs so that dwell intervals are closed promptly rather than on the next detection.

10. Posture classification

PostureStage produces soft posture evidence for each detection rather than a single hard label. The 17 COCO keypoints drive geometric scorers (torso angle, knee bend, head-spine alignment, shin-drop geometry) into a PostureScores record with continuous evidence for lying, sitting, and standing_walking, plus a keypoint-confidence value. Soft scoring lets several weak cues accumulate instead of firing on one threshold. In a bedroom, a lying prior is added when the body is occluded.

Fusion and the final label happen in stage 11. GlobalPostureTracker keeps each camera's latest scores, fuses them quality-weighted across the person's active cameras, and applies hysteresis before committing a posture. Crops without pose output (smaller than 16x32 px or from cameras with pose disabled) contribute unknown. See Handling noisy readings for the full position and posture fusion model.

11. Trajectory writing

TrajectoryStage records two types of data in TimescaleDB hypertables:

• person_trajectories: one row per frame per tracked person, with identity, room, floor-projected coordinates (in mm), posture, motion energy, and identity confidence. Points are written regardless of identity status: UNKNOWN trajectories are recorded and relabeled if identity is later resolved.
• room_dwells: continuous intervals in a room with entry/exit times, cumulative dwell, still-seconds, min/max/mean motion energy, and dominant posture.

When homography is unavailable, uncalibrated (0, 0) floor points are written.

12. Keyframe sampling

KeyframeStage saves tagged JPEG keyframes to the tagged_keyframes table and publishes SceneSample protos to the scene.samples Redis stream. Two triggers:

• Periodic: one keyframe per PH per configurable sampling interval.
• Identity change: an immediate keyframe when a PH's identity is revised, tagged with tag_reason="identity_changed".

Keyframes carry annotations (ph_id, camera_id, identity_id, bounding box) used by the CC-side SceneSampleSubscriber for downstream scene analysis.

13. Identity revisions

When the committed identity changes (new assignment, reassignment, or demotion to UNKNOWN), an IdentityRevision proto is published to tracking.revisions. The revision carries the full posterior distribution as IdentityCandidate entries, the previous and new identity IDs, and evidence metadata. A rate limiter caps revisions at 3 per PH per minute.

When the identity committer is enabled, the PostgresIdentityRewriter performs retroactive cross-table rewriting: trajectory points, room dwells, and dementia signals for the affected PH are relabeled from the old identity to the new identity, covering the window from applies_from to applies_to.

14. Trail management

TrailsStage maintains an in-memory ring buffer of foot-point trail positions per PH (default: last 30 positions). These trails are published in the TrackingEvent proto and consumed by the CC live view to render person movement paths on the floor plan.

15. Provenance persistence

ProvenancePersistStage durably persists every identity decision that revises a PH's previous identity (revises_previous=True), unconditionally. It runs immediately before the per-camera publish throttle, and deliberately outside it: a decision that revises identity is audit data a caregiver-facing read model joins against, so it must never be dropped just because the frame that carried it was rate-limited for live-view publishing. The stage is a no-op when no identity_provenance_repo is configured.

16. ReID candidate creation

ReIDCandidateStage is the only pipeline path that writes to the ReID gallery. For each committed, face-matched detection it evaluates a governed eligibility gate (evaluate_candidate, app/tracking/identity/candidate_eligibility.py) and, when it passes, creates a pending_review candidate with crop and frame provenance, origin_tracklet_id, ph_id, and model/preprocessing versions. A face-derived candidate requires the direct recognized ArcFace identity to equal the committed identity with calibrated confidence at or above the authority threshold; without a deployed calibration artifact the stage produces no candidates rather than falling back to raw similarity. Like provenance persistence, this runs ahead of the publish throttle because candidate creation is audit-visible governance data. The per-(identity, orientation) cap on gallery growth counts pending_review and operator_verified rows together.

17. Event publishing

PublishStage assembles and publishes the final TrackingEvent proto to tracking.events. It carries per-frame detections enriched with ph_id, the posterior's top identity per PH (published even before formal commit, so the live view sees the current best guess immediately), pose keypoints, per-PH foot-point trails, and posterior evidence (top probability, top-2 probability, face anchor flag).

On the CC side, the TrackingEventSubscriber decodes this proto, builds a ph_id -> identity_id mapping from the IdentityRevision sub-messages, and assigns identity_id, display_name, and identity_confidence to each detection in the WebSocket broadcast.