MCP Integration

Cognitive Companion treats the home as something an AI agent can safely operate. A built-in Model Context Protocol (MCP) server, built on the official MCP Python SDK, exposes 61 tools for AI agent integration: agents can discover system state, query sensor data, inspect enrollment, face calibration progress, and e-ink status, check person locations, review activity timelines and daily reports, explore semantic memory, read the unified signals feed, acknowledge behavioral signals with feedback, drive and analyze guided routine sessions, propose identity corrections, trigger rule executions, author new rules, and inspect plugin metadata. The same tools are shared with the Gemini Live voice companion for function calling during conversations.

This makes Cognitive Companion agent-native: household state has one governed, authenticated tool surface, and every agent, whether Claude Desktop, a custom orchestrator, or the internal voice companion, goes through it. There is no separate integration to audit per client, and tools stay in lockstep with the UI because both read the same service layer.

What is MCP?

The Model Context Protocol is an open standard for connecting AI models to external tools and data sources. It provides a structured way for AI agents to:

• Discover available tools and their parameters
• Execute tools with validated inputs
• Receive structured responses

Cognitive Companion's MCP server allows external AI agents (Claude, GPT, custom agents) to interact with the senior care system as part of their tool-calling workflows.

MCP and BFF parity guarantee

MCP tools and BFF router endpoints share one service layer (design rule D6). Any data exposed to the Vue UI through a router is exposed to MCP by reading the same service function, never a parallel query. Consequences:

• MCP tools contain no query logic of their own; they call service methods.
• Import-linter contracts enforce that mcp/ may not import a repository directly.
• When a service response envelope changes, both the router and the MCP tool reflect the change automatically.
• Smoke tests in backend/tests/mcp/ assert that every registered tool name resolves to a callable.

Architecture

The MCP server is implemented using the official mcp Python SDK's FastMCP class. Tools are defined as decorated async functions with type hints that auto-generate JSON schemas. The server is mounted as an ASGI sub-application on FastAPI at /mcp, serving the standard MCP protocol via streamable HTTP transport.

A GeminiToolAdapter reads the same tool definitions and converts them to Gemini FunctionDeclaration format, so the voice companion can call tools during conversations without duplicating implementations.

Available Tools

Tool	Description	Parameters
get_rooms	List all configured rooms	None
get_sensors	List sensors	room_name, sensor_type (optional filters)
get_room_occupancy	Current occupancy from presence sensors	room_name (optional)
get_recent_images	Recent camera images for a sensor	sensor_id, limit
get_light_level	Illuminance from a HA sensor	entity_id
get_signals_feed	Unified caregiver signals feed (CTS signals plus rule alerts)	source, severity_min, room_name, limit (optional)
get_event_logs	Rule execution event logs	rule_name, status, limit (optional)
get_rules	Configured automation rules	enabled_only (default true)
get_conversation_history	Recent conversation turns	session_id, limit (optional)
get_person_locations	Current location of all tracked members	None
get_enrolled_persons	Household members with face enrollment data	None
get_roster_health	Members with too few enrolled face photos, plus recurring visitors awaiting review	None
get_face_calibration_summary	Face calibration labeling progress per resident, walks, and visitor labels (read-only)	None
get_person_sightings	Camera sighting history for a person	person_id, limit
get_person_activities	Recent detected activities (eating, sleeping, etc.)	person_id, activity_type, minutes
get_workflow_executions	Recent pipeline workflow executions	rule_name, status, limit (optional)
get_rule_pipeline	Pipeline step definitions for a rule	rule_id
trigger_rule	Manually trigger a rule's pipeline execution	rule_id
get_eink_display_status	Active e-ink image state for one or all displays	sensor_id (optional)
get_local_datetime	Current local date and time for the household's timezone	None
get_weather	Current weather from Home Assistant	None
get_person_timeline	Chronological timeline of activities and sightings for a person	person_id, minutes (optional)
get_daily_report	End-of-day wellness report for one or all members	person_id, report_date (optional)
get_open_sessions	Currently open activity sessions (meals, bathroom, etc.)	person_id, activity_type (optional)
submit_user_response	Record user response to an interactive prompt step	execution_id, step_id, action
get_recent_scene_objects	Recent object presence in a room	room_id, since_minutes
get_scene_observations	Search scene observations with vector similarity	query_text, room_id, limit (optional)
get_person_movements	Movement transitions for a person between rooms	person_id, semantic, since_minutes (optional)
get_room_trend	Room-level trend state from object presence data	room_id, since_hours (optional)
search_similar_scenes	Vector search across scene embeddings	query_embedding, room_id, limit (optional)
get_tracking_status	Overall CTS tracking status and active PH count	(none)
get_person_location	Current location envelope for one person (with quality/staleness)	person_id
get_recent_dementia_signals	Recent dementia signals for a person with signal envelopes	person_id, limit (optional)
acknowledge_dementia_signal	Acknowledge a dementia signal, optionally with caregiver feedback	signal_id, feedback (optional)
get_gait_trend	Gait speed trend envelope for a resident	person_id, days (default 56)
get_heatmap	Aggregated floor-plan heatmap bins for a person over a time range	person_id, start_time, end_time, start_minute, end_minute (optional)
list_keyframe_frames	Keyframes grouped per physical source frame for identity review	person_id, camera_id, tag_reason, conflict_only, pending_review_only, limit, offset (all optional)
propose_identity_correction	Propose an observation-bounded identity correction segment for a PH	ph_id, observation_id, at (optional)
get_identity_correction_job	Status of an identity correction job	revision_id
get_active_guided_step	Current step of an active guided session	session_id
mark_guided_step_complete	Propose that the resident completed the current guided step	session_id, step_ord, note (optional)
repeat_guided_step	Rephrase and repeat the current guided step	session_id
report_step_blocked	Record that the resident is stuck on the current step	session_id, reason
request_caregiver_help	Escalate a guided session to the caregiver	session_id, reason (optional)
record_resident_preference	Record a stable resident preference (not a transient fact) to the knowledge repository	person_id, preference, context (optional)
get_guided_completion_summary	Guided-task completion outcomes	person_id, routine_id, since, until (optional)
get_guided_attempts_per_step	Retry pressure by guided routine step	person_id, routine_id, since, until (optional)
get_guided_time_to_complete	Guided routine completion durations	person_id, routine_id, since, until (optional)
get_guided_abandonment	Guided-task abandonment rate and reasons	person_id, routine_id, since, until (optional)
get_guided_escalation_breakdown	Guided-task escalation reasons	person_id, routine_id, since, until (optional)
get_guided_vision_agreement	Guided-task vision agreement quality	person_id, routine_id, since, until (optional)
get_guided_time_of_day	Guided outcomes bucketed by local hour	person_id, routine_id, since, until (optional)
get_guided_watch_summary	Guided-task watch runs and agreement	person_id, routine_id, since, until (optional)
get_guided_gate_cost_summary	Guided-task vision gate execution costs	person_id, routine_id, since, until (optional)
query_knowledge_base	Semantic search over the knowledge repository	query
get_current_quiz_question	Current question in an active quiz session	session_id
submit_quiz_answer	Record an answer to a quiz question	session_id, answer
complete_quiz_session	Close a quiz session	session_id
list_rules	List all rules with summary info	(none)
list_plugin_metadata	Metadata for all registered steps, filters, and channels	kind (optional: "step", "filter", "channel")
get_rule_bundle	Export a rule as a portable bundle	rule_id
import_rule_bundle	Validate or commit a rule bundle	bundle (RuleBundle dict), mode ("preview" or "commit")

Authentication

MCP tools require authentication via the API key. The key is configured in config/auth.yaml:

api_keys:
  - key: ${CC_MCP_API_KEY}
    name: mcp_agent
    permissions:
      - mcp_readonly

Pass the key in the X-API-Key header or an Authorization: Bearer <key> header. A key in the query string is rejected.

Every tool is listed to every authenticated agent, and each call is authorized against the token the tool requires:

Tools	Token	Granted to
All read tools	mcp.tools:read	mcp_readonly, admin
import_rule_bundle	rules:import	rules:write_via_mcp (the shipped agent key), admin
trigger_rule	rules:execute	admin
acknowledge_dementia_signal	cts.signals:acknowledge	admin
propose_identity_correction	cts.identity:propose	admin
Resident session tools (quiz answers, guided steps, preferences, submit_user_response)	resident_session:act	no API key: these are for the built-in voice companion only

So the mcp_readonly key can read anything the read tools expose but cannot trigger a rule, acknowledge a signal, or act for the resident. To let an agent key call a state-changing tool, add that tool's token to the key's role in config/auth.yaml. A denied call returns HTTP 403 with a JSON-RPC error and never runs the tool; a batch containing one denied call is rejected whole.

Endpoint

The MCP server is available at a single endpoint:

Method	Path	Description
POST	/mcp	MCP protocol endpoint (streamable HTTP, JSON-RPC)

Tool Discovery

curl -X POST \
  -H "X-API-Key: $CC_MCP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"tools/list","id":1}' \
  http://localhost:8000/mcp

Tool Execution

curl -X POST \
  -H "X-API-Key: $CC_MCP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"get_person_sightings","arguments":{"person_id":"grandma"}},"id":2}' \
  http://localhost:8000/mcp

Integration Patterns

With Claude Desktop

Configure Claude Desktop's MCP settings to point to your Cognitive Companion instance:

{
  "mcpServers": {
    "cognitive-companion": {
      "url": "http://your-cc-host:8000/mcp",
      "headers": {
        "X-API-Key": "your_mcp_key"
      }
    }
  }
}

With Custom Agents

Any agent framework that supports the MCP protocol can integrate with Cognitive Companion by pointing to the /mcp endpoint. The standard JSON-RPC interface handles tool discovery and execution.

Example Agent Workflow

An AI agent monitoring the household might:

1. Call get_person_locations to check where everyone is
2. Call get_enrolled_persons to see which household members are tracked
3. Call get_person_activities to check if lunch has been eaten
4. If lunch has not been detected, call trigger_rule on the lunch reminder rule
5. Call get_signals_feed to check for any unacknowledged signals

Voice Companion Integration

A configurable subset of MCP tools is also available to the Gemini Live voice companion via function calling. When the senior asks a question like "what's the weather?" or "where is everyone?", Gemini pauses audio generation, calls the appropriate tool, and incorporates the result into its spoken response. Tool results are never displayed raw to the user; they always flow through Gemini's natural language audio response.

The voice-enabled tool subset is configured in settings.yaml:

mcp:
  gemini_tools:
    - "get_rooms"
    - "get_room_occupancy"
    - "get_person_locations"
    - "get_signals_feed"
    - "get_weather"
    - "get_recent_scene_objects"
    - "get_room_trend"
    - "get_person_movements"
    - "get_local_datetime"
    - "get_person_activities"
    - "get_enrolled_persons"
    - "get_person_timeline"
    - "get_daily_report"
    - "get_open_sessions"
    - "submit_user_response"
    - "query_knowledge_base"
    - "submit_quiz_answer"
    - "get_current_quiz_question"
    - "complete_quiz_session"
    - "get_active_guided_step"
    - "mark_guided_step_complete"
    - "repeat_guided_step"
    - "report_step_blocked"
    - "request_caregiver_help"
    - "record_resident_preference"

The voice agent can declare and call exactly the tools in mcp.gemini_tools, checked again when it calls one, and an empty list gives it no tools. Destructive tools like trigger_rule are not in the shipped list. The guided-session tools are voice-enabled by design: they are how the realtime agent proposes step completion, repeats a step, or escalates to a caregiver during a guided routine. get_person_timeline, get_daily_report, and get_open_sessions are the typed daily-living ledger tools: the agent answers countable questions ("has she had her medication today") by calling these, never by aggregating semantic-memory captions.

Network Considerations

The MCP server runs on the same backend instance, so no additional deployment is needed. Since Cognitive Companion runs on-premise without a public endpoint, MCP clients must be on the same local network. For remote agent access, consider:

• VPN or tailnet for secure remote access
• A reverse proxy with authentication for controlled exposure

Adding New Tools

1. Add a @_register(token=...) decorated async function in backend/mcp/server.py. Type hints on parameters auto-generate JSON schemas. The token is required: TOOLS_READ for a read tool, or a new <resource>:<action> token (defined in config/auth.yaml) for a tool that changes state.
2. If the tool should be available in voice conversations, add it to mcp.gemini_tools in config/settings.yaml.

See the development guide for more details.