Owner/openspec/changes/add-full-duplex-agent-voice-assistant/specs/voice-pet-pipeline/spec.md

ADDED Requirements

Requirement: Full-duplex agent mode

The system SHALL support a planned full-duplex agent voice assistant mode that runs continuous microphone listening, streaming recognition, streaming response generation, streaming speech playback, interruption handling, memory retrieval, and safe tool routing as one coordinated pipeline.

Scenario: Full-duplex mode starts

• WHEN the future implementation starts with OWNER_ASSISTANT_MODE=full_duplex_agent
• THEN the system SHALL initialize continuous audio capture, audio playback, WebRTC audio processing, VAD, streaming STT, streaming TTS, conversation management, memory, and tool routing before accepting user interaction

Scenario: Turn-based mode remains available during migration

• WHEN the future implementation starts with the existing turn-based mode
• THEN the system SHALL preserve the existing wake-to-capture-to-STT-to-LLM-to-TTS behavior unless full-duplex mode is explicitly enabled

Scenario: Full-duplex provider is unavailable

• WHEN a required full-duplex provider cannot be initialized and no configured fallback is allowed
• THEN the system SHALL fail startup with a structured provider error rather than entering a fake full-duplex loop

Requirement: WebRTC audio processing foundation

The system SHALL define a WebRTC audio processing stage for full-duplex mode that applies acoustic echo cancellation, noise suppression, and automatic gain control to microphone capture audio using speaker playback audio as render reference.

Scenario: AEC uses playback reference

• WHEN TTS playback writes PCM frames to the speaker output path
• THEN the same render audio SHALL be provided to the WebRTC audio processing stage as echo cancellation reference

Scenario: Capture audio is processed

• WHEN microphone frames are captured during full-duplex mode
• THEN the frames SHALL pass through AEC, noise suppression, and automatic gain control before VAD, interruption detection, and streaming STT consume them

Scenario: Pure assistant echo is present

• WHEN the microphone receives only the assistant's own playback echo and no user speech
• THEN the processed audio SHALL NOT produce a valid user interruption or valid user transcript

Scenario: APM format mismatch occurs

• WHEN capture audio and render reference audio have incompatible sample rate, channel count, or frame size
• THEN the system SHALL either explicitly resample to the configured internal format or fail with AUDIO_APM_FORMAT_MISMATCH

Scenario: WebRTC APM fails at runtime

• WHEN the WebRTC audio processing stage raises a processing error
• THEN the pipeline SHALL emit a structured audio processing error and transition through recovery without corrupting conversation history

Requirement: Full-duplex audio ring buffers

The system SHALL use bounded capture and render reference ring buffers with timestamps so full-duplex capture, playback, AEC, VAD, STT, and interruption detection can run concurrently without unbounded memory growth.

Scenario: Capture frame is received

• WHEN the microphone callback receives a PCM frame
• THEN the frame SHALL be written to the capture ring buffer with a monotonic timestamp, frame id, sample rate, channel count, and duration

Scenario: Render frame is queued

• WHEN TTS or playback queues a PCM frame for speaker output
• THEN the frame SHALL be written to the render reference ring buffer with timing metadata usable by AEC

Scenario: Ring buffer overflows

• WHEN an audio ring buffer exceeds its configured capacity
• THEN the system SHALL drop the oldest safe frames, emit an audio_buffer_overrun event, and continue without blocking the audio callback indefinitely

Scenario: Reference drift is detected

• WHEN capture and render timestamps drift beyond the configured tolerance
• THEN the system SHALL emit an audio_reference_drift event and reduce confidence in interruption decisions until alignment recovers

Requirement: Full-duplex agent state machine

The system SHALL define a deterministic full-duplex state machine with idle, listening, thinking, speaking, interrupted, tool_running, and recovering states.

Scenario: User input reaches the LLM

• WHEN full-duplex listening produces a valid final user transcript
• THEN the state machine SHALL transition from listening to thinking

Scenario: First playable response is ready

• WHEN the LLM response stream and TTS pipeline produce the first playable audio chunk
• THEN the state machine SHALL transition from thinking to speaking

Scenario: User interrupts playback

• WHEN valid user speech is detected while the assistant is speaking
• THEN the state machine SHALL transition from speaking to interrupted

Scenario: Tool call starts

• WHEN the LLM requests an approved tool call
• THEN the state machine SHALL transition to tool_running until the tool result is returned, rejected, cancelled, or fails

Scenario: Recoverable error occurs

• WHEN a recoverable provider, memory, tool, STT, TTS, LLM, or playback error occurs
• THEN the state machine SHALL transition to recovering and then to a safe listening or idle state after cleanup

Requirement: Pipeline event bus for agent stages

The system SHALL expose a stable pipeline event bus for full-duplex audio, recognition, generation, playback, interruption, memory, tools, confirmation, and recovery events.

Scenario: Terminal reporter displays state

• WHEN the terminal displays full-duplex runtime status
• THEN it SHALL derive its output from pipeline events rather than directly reading provider internals

Scenario: GUI subscribes to the same events

• WHEN a future desktop pet GUI is implemented
• THEN it SHALL be able to subscribe to the same event bus used by the terminal reporter and tests

Scenario: Event contains diagnostics

• WHEN a stage emits an event
• THEN the event SHALL include the session id or turn id when available, stage name, event type, timestamp, and sanitized payload

Scenario: Sensitive payload is present

• WHEN an event involves credentials, raw audio, memory content, or tool output
• THEN the event payload SHALL omit secrets and SHALL truncate or summarize sensitive content according to security policy

Requirement: Low-latency interruption and cancellation

The full-duplex mode SHALL detect valid user speech during assistant playback and cancel the active response pipeline with a target interruption latency under 200 ms.

Scenario: User speaks while assistant is speaking

• WHEN the assistant is playing TTS audio and APM-processed microphone audio contains valid user speech
• THEN the interruption detector SHALL emit interrupt_detected, stop remaining playback, and cancel active LLM and TTS work

Scenario: Interruption latency is measured

• WHEN a test fixture injects user speech during speaking
• THEN the system SHALL record the latency from VAD speech start to playback stop and SHALL target P95 under 200 ms

Scenario: Echo does not interrupt

• WHEN microphone input during playback is explained by assistant render reference echo after AEC
• THEN the system SHALL NOT emit interrupt_detected

Scenario: Cancellation is idempotent

• WHEN interruption or recovery triggers cancellation more than once
• THEN LLM, TTS, playback, STT, and cancellable tool tasks SHALL handle repeated cancellation without unhandled exceptions

Scenario: Interrupted assistant text is stored

• WHEN assistant playback is interrupted before the full response is spoken
• THEN only assistant text that was fully spoken before interruption SHALL be eligible for short-term context or long-term memory

Requirement: Streaming STT and realtime transcript

The full-duplex mode SHALL use a streaming speech-to-text provider that emits partial, stable partial, and final transcripts while the user is speaking.

Scenario: Partial transcript is produced

• WHEN the user is speaking and streaming STT emits a changed partial transcript
• THEN the event bus SHALL emit a realtime transcript event for user-visible feedback

Scenario: Stable partial is produced

• WHEN a partial transcript remains stable according to the configured stability filter
• THEN the system SHALL mark it as stable partial while still withholding it from the LLM unless finalization rules allow submission

Scenario: Final transcript is produced

• WHEN the current user utterance ends
• THEN streaming STT SHALL emit a final transcript that becomes the user message candidate for conversation processing

Scenario: Empty transcript is produced

• WHEN final STT returns empty text, punctuation-only text, or text below confidence thresholds
• THEN the system SHALL discard the utterance, avoid invoking the LLM, and recover to listening

Scenario: Streaming STT provider changes

• WHEN the configured provider changes from faster_whisper to sensevoice or sherpa_onnx
• THEN the state machine and Conversation Manager SHALL keep the same transcript event contract

Requirement: Streaming response and TTS playback

The full-duplex mode SHALL stream LLM output into sentence segmentation, TTS synthesis, and playback without waiting for the full assistant reply to complete.

Scenario: First sentence is ready

• WHEN the LLM stream produces a complete sentence or configured safe speech segment
• THEN the sentence segmenter SHALL send sanitized text to the streaming TTS provider immediately

Scenario: TTS chunk is ready

• WHEN the TTS provider produces the first playable PCM chunk
• THEN playback SHALL begin without waiting for all remaining assistant text

Scenario: TTS text contains emoji

• WHEN LLM output contains emoji, sticker markers, Markdown image syntax, or other non-speech content
• THEN the text sent to TTS SHALL be sanitized before synthesis

Scenario: Playback is cancelled

• WHEN cancellation is requested during response playback
• THEN playback SHALL stop at the next configured audio chunk boundary and SHALL clear unplayed audio chunks

Scenario: TTS provider fails

• WHEN streaming TTS fails before or during playback
• THEN the system SHALL emit a structured TTS error and enter recovery without appending unspoken text to context

Requirement: Conversation manager for agent turns

The system SHALL define a Conversation Manager that coordinates short-term context, long-term memory retrieval, LLM streaming, tool calls, response playback, interruption, and recovery.

Scenario: User final transcript is accepted

• WHEN a valid final transcript is produced
• THEN the Conversation Manager SHALL create a user message candidate and gather short-term context and relevant long-term memory before invoking the LLM

Scenario: Assistant response completes

• WHEN the assistant response is fully spoken and no tool call or interruption is pending
• THEN the Conversation Manager SHALL commit the spoken assistant text to short-term context

Scenario: Response is interrupted

• WHEN user speech interrupts the assistant response
• THEN the Conversation Manager SHALL cancel the current response, preserve only committed spoken text, and process buffered user speech as the next input

Scenario: Conversation manager recovers

• WHEN a provider error occurs
• THEN the Conversation Manager SHALL emit a recoverable error event and return the agent to a safe listening or idle state when possible

Requirement: Long-term memory with FAISS and SQLite

The system SHALL define a long-term memory layer where SQLite stores text records and metadata while FAISS stores vector indexes for semantic retrieval.

Scenario: Memory is saved

• WHEN the user explicitly asks the assistant to remember a non-sensitive preference, fact, project detail, or task summary
• THEN the Memory Manager SHALL save a typed memory record with metadata in SQLite and an embedding vector in FAISS

Scenario: Memory is retrieved

• WHEN a later user input is semantically related to stored memories
• THEN the Memory Manager SHALL retrieve Top-K relevant memories and provide them to the Conversation Manager as separate memory context

Scenario: Runtime restarts

• WHEN the assistant process restarts and memory is enabled
• THEN saved non-sensitive memories SHALL remain available through SQLite and FAISS

Scenario: Memory is disabled

• WHEN OWNER_MEMORY_ENABLED=0
• THEN the system SHALL NOT read from or write to SQLite or FAISS memory storage

Scenario: Sensitive content is detected

• WHEN a memory candidate contains credentials, payment data, private identifiers, account secrets, or other sensitive content
• THEN the system SHALL NOT automatically save it and SHALL require explicit user confirmation or reject the save

Scenario: Memory index is inconsistent

• WHEN SQLite metadata and FAISS index records are missing, mismatched, or fail checksum validation
• THEN the Memory Manager SHALL report a health error and SHALL NOT return unverified memory records

Requirement: Tool Router and structured tool execution

The system SHALL define a Tool Router that validates structured tool calls, applies security policy, routes approved tools, enforces execution limits, and returns sanitized tool results to the LLM.

Scenario: Allowed memory search runs

• WHEN the LLM requests memory.search with valid arguments
• THEN the Tool Router SHALL execute the memory search within configured limits and return sanitized results

Scenario: Tool schema is invalid

• WHEN a tool call has an unknown name, missing required argument, invalid type, or unknown field
• THEN the Tool Router SHALL reject it with a structured validation error

Scenario: Tool exceeds call budget

• WHEN a turn exceeds the configured maximum number of tool calls or total tool runtime
• THEN the Tool Router SHALL reject further tool calls for that turn and return a budget error

Scenario: Tool output is too long

• WHEN a tool returns output larger than the configured limit
• THEN the Tool Router SHALL truncate the output, mark it as truncated, and avoid sending full oversized content to the LLM

Scenario: Tool execution fails

• WHEN an approved tool raises an error or times out
• THEN the Tool Router SHALL return a structured tool error and the Conversation Manager SHALL recover or ask the user for next steps

Requirement: Safe first-version tools

The first full-duplex Agent tool set SHALL prioritize low-risk tools and SHALL include only memory.search, memory.save, shell.readonly, openinterpreter.run, and browser.playwright as planned first-version adapters.

Scenario: Read-only shell command is allowed

• WHEN shell.readonly receives an allowlisted read-only command within allowed directories
• THEN the tool SHALL execute with timeout and output truncation

Scenario: Write shell command is rejected

• WHEN shell.readonly receives a command that writes, deletes, installs dependencies, changes permissions, uploads data, or mutates external state
• THEN the Tool Router SHALL reject the command

Scenario: Memory save is requested

• WHEN memory.save is requested for a non-sensitive user-approved memory
• THEN the tool SHALL save through the Memory Manager and return the new memory id

Scenario: High-risk action is requested

• WHEN any planned first-version tool request involves deletion, upload, payment, trading, account changes, permission changes, secret access, or arbitrary code execution
• THEN the Tool Router SHALL reject it or require explicit confirmation according to policy

Requirement: Open Interpreter external adapter

The system SHALL treat Open Interpreter as an optional external CLI or subprocess backend and SHALL NOT copy the local openinterpreter/ clone into the Owner source tree.

Scenario: Open Interpreter is disabled

• WHEN OWNER_OPENINTERPRETER_ENABLED=0
• THEN the Tool Router SHALL mark openinterpreter.run unavailable and SHALL NOT invoke any Open Interpreter command

Scenario: Open Interpreter path is missing

• WHEN OWNER_OPENINTERPRETER_ENABLED=1 but the configured command cannot be found
• THEN the adapter SHALL return OPENINTERPRETER_UNAVAILABLE without affecting other tools

Scenario: Low-risk Open Interpreter task is approved

• WHEN a low-risk Open Interpreter task passes directory, timeout, and argument validation
• THEN the adapter SHALL run it as an external process with bounded runtime and sanitized output

Scenario: High-risk Open Interpreter task is requested

• WHEN an Open Interpreter task would write files, delete data, change accounts, install dependencies, upload data, or execute broad arbitrary code
• THEN the Tool Router SHALL require explicit confirmation or reject the task in first-version policy

Requirement: Browser automation boundary

The system SHALL define browser.playwright as a planned browser automation adapter with explicit safety boundaries.

Scenario: Safe browser read task is requested

• WHEN the user asks for a low-risk browser read or extraction task and Playwright is enabled
• THEN the adapter SHALL run in an approved browser context with timeout, navigation limits, and sanitized output

Scenario: Sensitive browser action is requested

• WHEN a browser task involves login, payment, purchase, form submission, account changes, private messages, uploads, or irreversible actions
• THEN the adapter SHALL require explicit confirmation or reject the task according to policy

Scenario: Playwright is unavailable

• WHEN Playwright or its browser runtime is not installed
• THEN browser.playwright SHALL report a structured unavailable error and SHALL NOT affect voice conversation

Requirement: Computer control reservation

The system SHALL reserve a future ComputerControlProvider boundary for GUI control while excluding GUI click, keyboard, screen, and Accessibility automation from the first full-duplex Agent implementation.

Scenario: GUI control is requested in first version

• WHEN a tool call requests direct GUI click, keyboard input, screen control, or Accessibility automation
• THEN the first-version Tool Router SHALL reject it as unsupported

Scenario: Future provider is designed

• WHEN a future change implements computer control
• THEN it SHALL use public capabilities such as macOS Accessibility, Playwright, or trycua-style providers and SHALL define separate confirmation and audit requirements

Scenario: Codex Computer Use is referenced

• WHEN design work references Codex Computer Use behavior
• THEN the system SHALL use it only as a safety policy reference and SHALL NOT copy private or bundled implementation details

Requirement: Agent security and privacy

The full-duplex Agent architecture SHALL protect credentials, raw audio, memory data, and tool execution boundaries by default.

Scenario: Credentials are configured

• WHEN LLM, embedding, tool, or external adapter credentials are needed
• THEN they SHALL be read from local uncommitted configuration and SHALL NOT be hard-coded or logged

Scenario: Raw audio is processed

• WHEN microphone or render audio is processed for APM, VAD, STT, or interruption
• THEN raw audio and derived voice features SHALL remain process-local unless the user explicitly enables a documented recording feature

Scenario: Long-term memory stores text

• WHEN memory is saved
• THEN the memory record SHALL include sensitivity metadata and SHALL NOT include raw audio or credentials

Scenario: Tool audit is recorded

• WHEN a tool executes
• THEN the audit record SHALL include tool name, risk level, confirmation status, duration, status, and sanitized summary without secrets

Scenario: User disables tools

• WHEN OWNER_TOOL_ROUTER_ENABLED=0
• THEN no tool calls SHALL execute and LLM tool requests SHALL be rejected or ignored with a structured message

Requirement: Full-duplex performance targets

The full-duplex Agent architecture SHALL define measurable latency and reliability targets for audio processing, interruption, streaming STT, LLM-to-TTS, and tool routing.

Scenario: Audio frame is processed

• WHEN WebRTC APM receives a normal 20 ms frame
• THEN processing SHALL target completion before the next frame interval in normal local runtime conditions

Scenario: User begins speaking during playback

• WHEN valid user speech begins while assistant speech is playing
• THEN playback stop SHALL target P95 latency under 200 ms from speech start

Scenario: User starts speaking in listening mode

• WHEN the user begins speaking while the system is listening
• THEN streaming STT SHALL target first stable user-visible transcript within 800 ms P95

Scenario: LLM produces first sentence

• WHEN the LLM stream yields a complete sentence
• THEN TTS SHALL target first playable audio chunk within 1000 ms P95

Scenario: Tool call runs

• WHEN an approved low-risk tool runs
• THEN it SHALL obey configured timeout, output limit, and maximum calls per turn

Requirement: Full-duplex testability

The full-duplex Agent architecture SHALL be testable with fake audio devices, fake WebRTC APM reference audio, fake VAD, fake streaming STT/TTS, fake memory stores, fake tools, and deterministic virtual time.

Scenario: Fake APM suppresses echo

• WHEN a test injects playback reference audio and matching microphone echo
• THEN the fake or real APM test SHALL verify that echo does not trigger user transcript or interruption

Scenario: Speaking interruption is tested

• WHEN a test injects user speech during assistant playback
• THEN the state sequence SHALL include speaking -> interrupted -> listening and active LLM/TTS/playback tasks SHALL be cancelled

Scenario: Streaming STT and TTS are tested

• WHEN fake providers emit partial transcripts, final transcript, LLM deltas, sentence segments, and TTS chunks
• THEN tests SHALL verify event order, context writes, and playback order

Scenario: Memory persists across restart

• WHEN a memory is saved and a new runtime instance starts with memory enabled
• THEN the memory SHALL be retrievable from SQLite and FAISS

Scenario: Tools are tested

• WHEN fake tool calls cover allowed, rejected, confirmation-required, timeout, and oversized output paths
• THEN Tool Router tests SHALL verify decisions, audit events, and sanitized tool results

Scenario: OpenSpec validation runs

• WHEN this planning change is complete
• THEN openspec validate add-full-duplex-agent-voice-assistant --strict and openspec validate --all --strict SHALL pass