Roadmap

Proposed features and integration pathways for future development. We welcome community input. If you're interested in working on any of these, open an issue to discuss your approach.

Home Assistant Webhook Triggers ✅

Status: Implemented

Rules can now be triggered via POST /api/v1/webhooks/{rule_id} with an X-Webhook-Secret header. Secrets are generated per-rule via POST /api/v1/webhooks/{rule_id}/generate-secret and validated with HMAC constant-time comparison. The JSON request body becomes pipeline_data["trigger_input"] for downstream steps.

See API Reference: Webhooks for details.

Enhanced Pipeline Triggers with Input Parameters ✅

Status: Implemented

TriggerContext now includes a webhook_payload field. Webhook payloads and manual trigger parameters are available in pipeline_data["trigger_input"] for downstream steps to reference.

Gemini Live Tool Calling ✅

Status: Implemented

The voice companion supports function calling via the Gemini Live session. Tool definitions are built from the MCP registry and a configurable subset is passed to GeminiLiveProvider at startup. When Gemini emits a FunctionCall part, the audio handler routes it to the matching MCP tool handler and sends a FunctionResponse back. The voice agent can now answer questions like "Where is grandma?" or "What happened today?" using real system data.

Configure which tools are available to the voice companion via mcp.gemini_tools in settings.yaml. See MCP Integration for the full tool list.

Knowledge Repository ✅

Status: Implemented

Caregivers can upload structured pieces of knowledge about the senior's life and convert them into two delivery artifacts: info cards (paraphrased text + images shown as PWA popup or e-ink) and quizzes (multiple-choice or open-ended, delivered via voice + tappable buttons). The senior can ask voice questions ("how many grandchildren do I have?") and the companion answers from the repository via RAG.

Key capabilities:

• Documents: upload source text + images → chunk + embed via embeddinggemma-300m on Triton → lifecycle (uploaded → chunked → approved → archived)
• Info cards: 5 layout types (text_only, single_hero, side_by_side, gallery_grid_2x2), per-surface image variants (PWA WebP, e-ink dithered PNG), voice delivery via Gemini Live
• Quizzes: multiple-choice + open-ended, voice + button dual input, LLM grader for open-ended answers, quiz session lifecycle with timeout, LLM-assisted generation
• Senior Q&A: voice questions → embed → pgvector cosine search → LLM synthesis → logged for caregiver review
• Voice instruction system: 3-layer composition (step override → resource column → yaml default → base) for per-delivery Gemini Live behavioral guardrails
• LLM-assisted content generation: paraphrase, quiz, and voice instruction suggestions with caregiver review gate

See Knowledge Repository for full documentation.

Cross-Camera Dedup (Hallway/Bathroom Case) ✅

Status: Implemented (May 2026)

When a hallway camera and an adjacent camera both observe the same senior at a bathroom door, the pre-association floor-point dedup pass resolves them to a single PersonHypothesis. This ensures the bathroom_dwell_anomaly signal starts from the correct time and that the admin live view shows one person rather than two phantom tracks. The integration proof is test_hallway_bathroom_one_person replaying hallway_bathroom_door.bin.

Dashboard Unification ✅

Status: Implemented (May 2026)

The separate CTS Dashboard and Presence Fusion dashboards have been collapsed into a single role-aware Tracking workspace (TrackingWorkspace.vue). All six panels (presence map, occupancy table, signal feed, trajectory heatmap, quality gauge, and floor plan) read from usePersonPresence as a single data layer. Role-aware panel visibility is controlled by permissions, not separate page code. The previous per-role dashboards and their redundant fetch paths have been removed.

Live Process Visualisation ✅

Status: Implemented (May 2026)

The Process Activity view (/activity/process) streams real-time pipeline execution events via GET /ws/pipeline. It shows active rule pipeline runs as live DAGs (via CcDagChart), recent ingest activity from reCamera sensors (via CcLiveActivityFeed), and a historical step timeline (via CcStatusTimeline). The useLivePipeline composable manages connection state with 3-second exponential backoff and never shows a stale DAG as live data.

CTS Robustness: Continuity, Multi-View ReID, and Zero-Calibration Cross-Camera ✅

Status: Implemented (June 2026)

Tracking now holds identity through turns, walks, and camera handoffs instead of fragmenting into many short-lived UNKNOWN tracks. PersonHypothesis revival reuses a recently-closed track instead of spawning a new one; sticky maintenance holds a committed identity through evidence gaps unless a recognized different-identity face contradicts it. Face evidence is three-valued (recognized, candidate, unrecognized) and pose-aware, so a resident at a bad angle corroborates their own identity rather than vanishing. Multi-view ReID represents body appearance per orientation and matches by the best view, so a person who turned around is still re-identified. Cross-camera linking works without homography: a learned camera-adjacency topology plus multi-view appearance carries identity across uncalibrated cameras, and overlapping opposite-perspective cameras are joined at the identity level with a co-presence link. CTS also emits low-rate semantic state-change events (presence, dwell) so the rule-trigger load is decoupled from camera frame rate. Each mechanism shipped behind a config flag with a frame-replay proof and a guardrail that a stranger never inherits a resident's identity.

Pipeline Templates / Presets

Status: Proposed

Problem: Creating pipelines from scratch is complex for new users. Each of the 14 step types has its own configuration, and choosing the right combination requires understanding the system.

Design: Ship JSON fixtures with preset pipeline definitions:

Template	Steps
Camera Alert	person_identification → llm_call (vision) → llm_call (reasoning) → notification
Periodic Check	llm_call (vision) → llm_call (reasoning) → condition → notification
Medication Reminder	activity_detection → wait → verification → notification
Door Monitor	person_identification → condition → notification → ha_action

A GET /rules/templates endpoint lists presets, and POST /rules/from-template creates a rule from one. A "Use Template" button in the rule creation dialog would allow one-click setup.

Impact: Dramatically lowers the barrier to entry for new users.

Activity Timeline

Status: Proposed

Problem: PersonActivity records exist but there is no timeline visualization. Caregivers must manually correlate activities, sightings, and alerts.

Design:

• Backend: GET /persons/{id}/timeline?date=YYYY-MM-DD endpoint merging activities, sightings, and alerts into a unified chronological view
• Frontend: Vuetify v-timeline component in the person detail drawer showing the day's events with icons, timestamps, and room information

Use cases:

• Caregiver reviews grandma's day at a glance
• Identify patterns (always in kitchen at noon, late medication on Tuesdays)
• Spot anomalies (no activity detected for an unusual period)

Impact: Provides the "daily story" that caregivers need to feel confident about their family member's wellbeing.

---

Contributing to the Roadmap

Have an idea that's not listed here? We'd love to hear it. The best way to propose a new feature is to open a GitHub issue with:

1. Problem statement: what limitation does this address?
2. Proposed design: how would you implement it?
3. Use cases: who benefits and how?
4. Impact: what does this unlock for the project?

We prioritize features that maintain the project's core values: privacy-first, on-premise, composable, and accessible to multigenerational households.