mkbk
380 lines
23 KiB
Markdown
380 lines
23 KiB
Markdown
## ADDED Requirements
|
|
|
|
### Requirement: Full-duplex agent mode
|
|
The system SHALL support a planned full-duplex agent voice assistant mode that runs continuous microphone listening, streaming recognition, streaming response generation, streaming speech playback, interruption handling, memory retrieval, and safe tool routing as one coordinated pipeline.
|
|
|
|
#### Scenario: Full-duplex mode starts
|
|
- **WHEN** the future implementation starts with `OWNER_ASSISTANT_MODE=full_duplex_agent`
|
|
- **THEN** the system SHALL initialize continuous audio capture, audio playback, WebRTC audio processing, VAD, streaming STT, streaming TTS, conversation management, memory, and tool routing before accepting user interaction
|
|
|
|
#### Scenario: Turn-based mode remains available during migration
|
|
- **WHEN** the future implementation starts with the existing turn-based mode
|
|
- **THEN** the system SHALL preserve the existing wake-to-capture-to-STT-to-LLM-to-TTS behavior unless full-duplex mode is explicitly enabled
|
|
|
|
#### Scenario: Full-duplex provider is unavailable
|
|
- **WHEN** a required full-duplex provider cannot be initialized and no configured fallback is allowed
|
|
- **THEN** the system SHALL fail startup with a structured provider error rather than entering a fake full-duplex loop
|
|
|
|
### Requirement: WebRTC audio processing foundation
|
|
The system SHALL define a WebRTC audio processing stage for full-duplex mode that applies acoustic echo cancellation, noise suppression, and automatic gain control to microphone capture audio using speaker playback audio as render reference.
|
|
|
|
#### Scenario: AEC uses playback reference
|
|
- **WHEN** TTS playback writes PCM frames to the speaker output path
|
|
- **THEN** the same render audio SHALL be provided to the WebRTC audio processing stage as echo cancellation reference
|
|
|
|
#### Scenario: Capture audio is processed
|
|
- **WHEN** microphone frames are captured during full-duplex mode
|
|
- **THEN** the frames SHALL pass through AEC, noise suppression, and automatic gain control before VAD, interruption detection, and streaming STT consume them
|
|
|
|
#### Scenario: Pure assistant echo is present
|
|
- **WHEN** the microphone receives only the assistant's own playback echo and no user speech
|
|
- **THEN** the processed audio SHALL NOT produce a valid user interruption or valid user transcript
|
|
|
|
#### Scenario: APM format mismatch occurs
|
|
- **WHEN** capture audio and render reference audio have incompatible sample rate, channel count, or frame size
|
|
- **THEN** the system SHALL either explicitly resample to the configured internal format or fail with `AUDIO_APM_FORMAT_MISMATCH`
|
|
|
|
#### Scenario: WebRTC APM fails at runtime
|
|
- **WHEN** the WebRTC audio processing stage raises a processing error
|
|
- **THEN** the pipeline SHALL emit a structured audio processing error and transition through recovery without corrupting conversation history
|
|
|
|
### Requirement: Full-duplex audio ring buffers
|
|
The system SHALL use bounded capture and render reference ring buffers with timestamps so full-duplex capture, playback, AEC, VAD, STT, and interruption detection can run concurrently without unbounded memory growth.
|
|
|
|
#### Scenario: Capture frame is received
|
|
- **WHEN** the microphone callback receives a PCM frame
|
|
- **THEN** the frame SHALL be written to the capture ring buffer with a monotonic timestamp, frame id, sample rate, channel count, and duration
|
|
|
|
#### Scenario: Render frame is queued
|
|
- **WHEN** TTS or playback queues a PCM frame for speaker output
|
|
- **THEN** the frame SHALL be written to the render reference ring buffer with timing metadata usable by AEC
|
|
|
|
#### Scenario: Ring buffer overflows
|
|
- **WHEN** an audio ring buffer exceeds its configured capacity
|
|
- **THEN** the system SHALL drop the oldest safe frames, emit an `audio_buffer_overrun` event, and continue without blocking the audio callback indefinitely
|
|
|
|
#### Scenario: Reference drift is detected
|
|
- **WHEN** capture and render timestamps drift beyond the configured tolerance
|
|
- **THEN** the system SHALL emit an `audio_reference_drift` event and reduce confidence in interruption decisions until alignment recovers
|
|
|
|
### Requirement: Full-duplex agent state machine
|
|
The system SHALL define a deterministic full-duplex state machine with `idle`, `listening`, `thinking`, `speaking`, `interrupted`, `tool_running`, and `recovering` states.
|
|
|
|
#### Scenario: User input reaches the LLM
|
|
- **WHEN** full-duplex listening produces a valid final user transcript
|
|
- **THEN** the state machine SHALL transition from `listening` to `thinking`
|
|
|
|
#### Scenario: First playable response is ready
|
|
- **WHEN** the LLM response stream and TTS pipeline produce the first playable audio chunk
|
|
- **THEN** the state machine SHALL transition from `thinking` to `speaking`
|
|
|
|
#### Scenario: User interrupts playback
|
|
- **WHEN** valid user speech is detected while the assistant is speaking
|
|
- **THEN** the state machine SHALL transition from `speaking` to `interrupted`
|
|
|
|
#### Scenario: Tool call starts
|
|
- **WHEN** the LLM requests an approved tool call
|
|
- **THEN** the state machine SHALL transition to `tool_running` until the tool result is returned, rejected, cancelled, or fails
|
|
|
|
#### Scenario: Recoverable error occurs
|
|
- **WHEN** a recoverable provider, memory, tool, STT, TTS, LLM, or playback error occurs
|
|
- **THEN** the state machine SHALL transition to `recovering` and then to a safe `listening` or `idle` state after cleanup
|
|
|
|
### Requirement: Pipeline event bus for agent stages
|
|
The system SHALL expose a stable pipeline event bus for full-duplex audio, recognition, generation, playback, interruption, memory, tools, confirmation, and recovery events.
|
|
|
|
#### Scenario: Terminal reporter displays state
|
|
- **WHEN** the terminal displays full-duplex runtime status
|
|
- **THEN** it SHALL derive its output from pipeline events rather than directly reading provider internals
|
|
|
|
#### Scenario: GUI subscribes to the same events
|
|
- **WHEN** a future desktop pet GUI is implemented
|
|
- **THEN** it SHALL be able to subscribe to the same event bus used by the terminal reporter and tests
|
|
|
|
#### Scenario: Event contains diagnostics
|
|
- **WHEN** a stage emits an event
|
|
- **THEN** the event SHALL include the session id or turn id when available, stage name, event type, timestamp, and sanitized payload
|
|
|
|
#### Scenario: Sensitive payload is present
|
|
- **WHEN** an event involves credentials, raw audio, memory content, or tool output
|
|
- **THEN** the event payload SHALL omit secrets and SHALL truncate or summarize sensitive content according to security policy
|
|
|
|
### Requirement: Low-latency interruption and cancellation
|
|
The full-duplex mode SHALL detect valid user speech during assistant playback and cancel the active response pipeline with a target interruption latency under 200 ms.
|
|
|
|
#### Scenario: User speaks while assistant is speaking
|
|
- **WHEN** the assistant is playing TTS audio and APM-processed microphone audio contains valid user speech
|
|
- **THEN** the interruption detector SHALL emit `interrupt_detected`, stop remaining playback, and cancel active LLM and TTS work
|
|
|
|
#### Scenario: Interruption latency is measured
|
|
- **WHEN** a test fixture injects user speech during `speaking`
|
|
- **THEN** the system SHALL record the latency from VAD speech start to playback stop and SHALL target P95 under 200 ms
|
|
|
|
#### Scenario: Echo does not interrupt
|
|
- **WHEN** microphone input during playback is explained by assistant render reference echo after AEC
|
|
- **THEN** the system SHALL NOT emit `interrupt_detected`
|
|
|
|
#### Scenario: Cancellation is idempotent
|
|
- **WHEN** interruption or recovery triggers cancellation more than once
|
|
- **THEN** LLM, TTS, playback, STT, and cancellable tool tasks SHALL handle repeated cancellation without unhandled exceptions
|
|
|
|
#### Scenario: Interrupted assistant text is stored
|
|
- **WHEN** assistant playback is interrupted before the full response is spoken
|
|
- **THEN** only assistant text that was fully spoken before interruption SHALL be eligible for short-term context or long-term memory
|
|
|
|
### Requirement: Streaming STT and realtime transcript
|
|
The full-duplex mode SHALL use a streaming speech-to-text provider that emits partial, stable partial, and final transcripts while the user is speaking.
|
|
|
|
#### Scenario: Partial transcript is produced
|
|
- **WHEN** the user is speaking and streaming STT emits a changed partial transcript
|
|
- **THEN** the event bus SHALL emit a realtime transcript event for user-visible feedback
|
|
|
|
#### Scenario: Stable partial is produced
|
|
- **WHEN** a partial transcript remains stable according to the configured stability filter
|
|
- **THEN** the system SHALL mark it as stable partial while still withholding it from the LLM unless finalization rules allow submission
|
|
|
|
#### Scenario: Final transcript is produced
|
|
- **WHEN** the current user utterance ends
|
|
- **THEN** streaming STT SHALL emit a final transcript that becomes the user message candidate for conversation processing
|
|
|
|
#### Scenario: Empty transcript is produced
|
|
- **WHEN** final STT returns empty text, punctuation-only text, or text below confidence thresholds
|
|
- **THEN** the system SHALL discard the utterance, avoid invoking the LLM, and recover to listening
|
|
|
|
#### Scenario: Streaming STT provider changes
|
|
- **WHEN** the configured provider changes from `faster_whisper` to `sensevoice` or `sherpa_onnx`
|
|
- **THEN** the state machine and Conversation Manager SHALL keep the same transcript event contract
|
|
|
|
### Requirement: Streaming response and TTS playback
|
|
The full-duplex mode SHALL stream LLM output into sentence segmentation, TTS synthesis, and playback without waiting for the full assistant reply to complete.
|
|
|
|
#### Scenario: First sentence is ready
|
|
- **WHEN** the LLM stream produces a complete sentence or configured safe speech segment
|
|
- **THEN** the sentence segmenter SHALL send sanitized text to the streaming TTS provider immediately
|
|
|
|
#### Scenario: TTS chunk is ready
|
|
- **WHEN** the TTS provider produces the first playable PCM chunk
|
|
- **THEN** playback SHALL begin without waiting for all remaining assistant text
|
|
|
|
#### Scenario: TTS text contains emoji
|
|
- **WHEN** LLM output contains emoji, sticker markers, Markdown image syntax, or other non-speech content
|
|
- **THEN** the text sent to TTS SHALL be sanitized before synthesis
|
|
|
|
#### Scenario: Playback is cancelled
|
|
- **WHEN** cancellation is requested during response playback
|
|
- **THEN** playback SHALL stop at the next configured audio chunk boundary and SHALL clear unplayed audio chunks
|
|
|
|
#### Scenario: TTS provider fails
|
|
- **WHEN** streaming TTS fails before or during playback
|
|
- **THEN** the system SHALL emit a structured TTS error and enter recovery without appending unspoken text to context
|
|
|
|
### Requirement: Conversation manager for agent turns
|
|
The system SHALL define a Conversation Manager that coordinates short-term context, long-term memory retrieval, LLM streaming, tool calls, response playback, interruption, and recovery.
|
|
|
|
#### Scenario: User final transcript is accepted
|
|
- **WHEN** a valid final transcript is produced
|
|
- **THEN** the Conversation Manager SHALL create a user message candidate and gather short-term context and relevant long-term memory before invoking the LLM
|
|
|
|
#### Scenario: Assistant response completes
|
|
- **WHEN** the assistant response is fully spoken and no tool call or interruption is pending
|
|
- **THEN** the Conversation Manager SHALL commit the spoken assistant text to short-term context
|
|
|
|
#### Scenario: Response is interrupted
|
|
- **WHEN** user speech interrupts the assistant response
|
|
- **THEN** the Conversation Manager SHALL cancel the current response, preserve only committed spoken text, and process buffered user speech as the next input
|
|
|
|
#### Scenario: Conversation manager recovers
|
|
- **WHEN** a provider error occurs
|
|
- **THEN** the Conversation Manager SHALL emit a recoverable error event and return the agent to a safe listening or idle state when possible
|
|
|
|
### Requirement: Long-term memory with FAISS and SQLite
|
|
The system SHALL define a long-term memory layer where SQLite stores text records and metadata while FAISS stores vector indexes for semantic retrieval.
|
|
|
|
#### Scenario: Memory is saved
|
|
- **WHEN** the user explicitly asks the assistant to remember a non-sensitive preference, fact, project detail, or task summary
|
|
- **THEN** the Memory Manager SHALL save a typed memory record with metadata in SQLite and an embedding vector in FAISS
|
|
|
|
#### Scenario: Memory is retrieved
|
|
- **WHEN** a later user input is semantically related to stored memories
|
|
- **THEN** the Memory Manager SHALL retrieve Top-K relevant memories and provide them to the Conversation Manager as separate memory context
|
|
|
|
#### Scenario: Runtime restarts
|
|
- **WHEN** the assistant process restarts and memory is enabled
|
|
- **THEN** saved non-sensitive memories SHALL remain available through SQLite and FAISS
|
|
|
|
#### Scenario: Memory is disabled
|
|
- **WHEN** `OWNER_MEMORY_ENABLED=0`
|
|
- **THEN** the system SHALL NOT read from or write to SQLite or FAISS memory storage
|
|
|
|
#### Scenario: Sensitive content is detected
|
|
- **WHEN** a memory candidate contains credentials, payment data, private identifiers, account secrets, or other sensitive content
|
|
- **THEN** the system SHALL NOT automatically save it and SHALL require explicit user confirmation or reject the save
|
|
|
|
#### Scenario: Memory index is inconsistent
|
|
- **WHEN** SQLite metadata and FAISS index records are missing, mismatched, or fail checksum validation
|
|
- **THEN** the Memory Manager SHALL report a health error and SHALL NOT return unverified memory records
|
|
|
|
### Requirement: Tool Router and structured tool execution
|
|
The system SHALL define a Tool Router that validates structured tool calls, applies security policy, routes approved tools, enforces execution limits, and returns sanitized tool results to the LLM.
|
|
|
|
#### Scenario: Allowed memory search runs
|
|
- **WHEN** the LLM requests `memory.search` with valid arguments
|
|
- **THEN** the Tool Router SHALL execute the memory search within configured limits and return sanitized results
|
|
|
|
#### Scenario: Tool schema is invalid
|
|
- **WHEN** a tool call has an unknown name, missing required argument, invalid type, or unknown field
|
|
- **THEN** the Tool Router SHALL reject it with a structured validation error
|
|
|
|
#### Scenario: Tool exceeds call budget
|
|
- **WHEN** a turn exceeds the configured maximum number of tool calls or total tool runtime
|
|
- **THEN** the Tool Router SHALL reject further tool calls for that turn and return a budget error
|
|
|
|
#### Scenario: Tool output is too long
|
|
- **WHEN** a tool returns output larger than the configured limit
|
|
- **THEN** the Tool Router SHALL truncate the output, mark it as truncated, and avoid sending full oversized content to the LLM
|
|
|
|
#### Scenario: Tool execution fails
|
|
- **WHEN** an approved tool raises an error or times out
|
|
- **THEN** the Tool Router SHALL return a structured tool error and the Conversation Manager SHALL recover or ask the user for next steps
|
|
|
|
### Requirement: Safe first-version tools
|
|
The first full-duplex Agent tool set SHALL prioritize low-risk tools and SHALL include only `memory.search`, `memory.save`, `shell.readonly`, `openinterpreter.run`, and `browser.playwright` as planned first-version adapters.
|
|
|
|
#### Scenario: Read-only shell command is allowed
|
|
- **WHEN** `shell.readonly` receives an allowlisted read-only command within allowed directories
|
|
- **THEN** the tool SHALL execute with timeout and output truncation
|
|
|
|
#### Scenario: Write shell command is rejected
|
|
- **WHEN** `shell.readonly` receives a command that writes, deletes, installs dependencies, changes permissions, uploads data, or mutates external state
|
|
- **THEN** the Tool Router SHALL reject the command
|
|
|
|
#### Scenario: Memory save is requested
|
|
- **WHEN** `memory.save` is requested for a non-sensitive user-approved memory
|
|
- **THEN** the tool SHALL save through the Memory Manager and return the new memory id
|
|
|
|
#### Scenario: High-risk action is requested
|
|
- **WHEN** any planned first-version tool request involves deletion, upload, payment, trading, account changes, permission changes, secret access, or arbitrary code execution
|
|
- **THEN** the Tool Router SHALL reject it or require explicit confirmation according to policy
|
|
|
|
### Requirement: Open Interpreter external adapter
|
|
The system SHALL treat Open Interpreter as an optional external CLI or subprocess backend and SHALL NOT copy the local `openinterpreter/` clone into the Owner source tree.
|
|
|
|
#### Scenario: Open Interpreter is disabled
|
|
- **WHEN** `OWNER_OPENINTERPRETER_ENABLED=0`
|
|
- **THEN** the Tool Router SHALL mark `openinterpreter.run` unavailable and SHALL NOT invoke any Open Interpreter command
|
|
|
|
#### Scenario: Open Interpreter path is missing
|
|
- **WHEN** `OWNER_OPENINTERPRETER_ENABLED=1` but the configured command cannot be found
|
|
- **THEN** the adapter SHALL return `OPENINTERPRETER_UNAVAILABLE` without affecting other tools
|
|
|
|
#### Scenario: Low-risk Open Interpreter task is approved
|
|
- **WHEN** a low-risk Open Interpreter task passes directory, timeout, and argument validation
|
|
- **THEN** the adapter SHALL run it as an external process with bounded runtime and sanitized output
|
|
|
|
#### Scenario: High-risk Open Interpreter task is requested
|
|
- **WHEN** an Open Interpreter task would write files, delete data, change accounts, install dependencies, upload data, or execute broad arbitrary code
|
|
- **THEN** the Tool Router SHALL require explicit confirmation or reject the task in first-version policy
|
|
|
|
### Requirement: Browser automation boundary
|
|
The system SHALL define `browser.playwright` as a planned browser automation adapter with explicit safety boundaries.
|
|
|
|
#### Scenario: Safe browser read task is requested
|
|
- **WHEN** the user asks for a low-risk browser read or extraction task and Playwright is enabled
|
|
- **THEN** the adapter SHALL run in an approved browser context with timeout, navigation limits, and sanitized output
|
|
|
|
#### Scenario: Sensitive browser action is requested
|
|
- **WHEN** a browser task involves login, payment, purchase, form submission, account changes, private messages, uploads, or irreversible actions
|
|
- **THEN** the adapter SHALL require explicit confirmation or reject the task according to policy
|
|
|
|
#### Scenario: Playwright is unavailable
|
|
- **WHEN** Playwright or its browser runtime is not installed
|
|
- **THEN** `browser.playwright` SHALL report a structured unavailable error and SHALL NOT affect voice conversation
|
|
|
|
### Requirement: Computer control reservation
|
|
The system SHALL reserve a future `ComputerControlProvider` boundary for GUI control while excluding GUI click, keyboard, screen, and Accessibility automation from the first full-duplex Agent implementation.
|
|
|
|
#### Scenario: GUI control is requested in first version
|
|
- **WHEN** a tool call requests direct GUI click, keyboard input, screen control, or Accessibility automation
|
|
- **THEN** the first-version Tool Router SHALL reject it as unsupported
|
|
|
|
#### Scenario: Future provider is designed
|
|
- **WHEN** a future change implements computer control
|
|
- **THEN** it SHALL use public capabilities such as macOS Accessibility, Playwright, or trycua-style providers and SHALL define separate confirmation and audit requirements
|
|
|
|
#### Scenario: Codex Computer Use is referenced
|
|
- **WHEN** design work references Codex Computer Use behavior
|
|
- **THEN** the system SHALL use it only as a safety policy reference and SHALL NOT copy private or bundled implementation details
|
|
|
|
### Requirement: Agent security and privacy
|
|
The full-duplex Agent architecture SHALL protect credentials, raw audio, memory data, and tool execution boundaries by default.
|
|
|
|
#### Scenario: Credentials are configured
|
|
- **WHEN** LLM, embedding, tool, or external adapter credentials are needed
|
|
- **THEN** they SHALL be read from local uncommitted configuration and SHALL NOT be hard-coded or logged
|
|
|
|
#### Scenario: Raw audio is processed
|
|
- **WHEN** microphone or render audio is processed for APM, VAD, STT, or interruption
|
|
- **THEN** raw audio and derived voice features SHALL remain process-local unless the user explicitly enables a documented recording feature
|
|
|
|
#### Scenario: Long-term memory stores text
|
|
- **WHEN** memory is saved
|
|
- **THEN** the memory record SHALL include sensitivity metadata and SHALL NOT include raw audio or credentials
|
|
|
|
#### Scenario: Tool audit is recorded
|
|
- **WHEN** a tool executes
|
|
- **THEN** the audit record SHALL include tool name, risk level, confirmation status, duration, status, and sanitized summary without secrets
|
|
|
|
#### Scenario: User disables tools
|
|
- **WHEN** `OWNER_TOOL_ROUTER_ENABLED=0`
|
|
- **THEN** no tool calls SHALL execute and LLM tool requests SHALL be rejected or ignored with a structured message
|
|
|
|
### Requirement: Full-duplex performance targets
|
|
The full-duplex Agent architecture SHALL define measurable latency and reliability targets for audio processing, interruption, streaming STT, LLM-to-TTS, and tool routing.
|
|
|
|
#### Scenario: Audio frame is processed
|
|
- **WHEN** WebRTC APM receives a normal 20 ms frame
|
|
- **THEN** processing SHALL target completion before the next frame interval in normal local runtime conditions
|
|
|
|
#### Scenario: User begins speaking during playback
|
|
- **WHEN** valid user speech begins while assistant speech is playing
|
|
- **THEN** playback stop SHALL target P95 latency under 200 ms from speech start
|
|
|
|
#### Scenario: User starts speaking in listening mode
|
|
- **WHEN** the user begins speaking while the system is listening
|
|
- **THEN** streaming STT SHALL target first stable user-visible transcript within 800 ms P95
|
|
|
|
#### Scenario: LLM produces first sentence
|
|
- **WHEN** the LLM stream yields a complete sentence
|
|
- **THEN** TTS SHALL target first playable audio chunk within 1000 ms P95
|
|
|
|
#### Scenario: Tool call runs
|
|
- **WHEN** an approved low-risk tool runs
|
|
- **THEN** it SHALL obey configured timeout, output limit, and maximum calls per turn
|
|
|
|
### Requirement: Full-duplex testability
|
|
The full-duplex Agent architecture SHALL be testable with fake audio devices, fake WebRTC APM reference audio, fake VAD, fake streaming STT/TTS, fake memory stores, fake tools, and deterministic virtual time.
|
|
|
|
#### Scenario: Fake APM suppresses echo
|
|
- **WHEN** a test injects playback reference audio and matching microphone echo
|
|
- **THEN** the fake or real APM test SHALL verify that echo does not trigger user transcript or interruption
|
|
|
|
#### Scenario: Speaking interruption is tested
|
|
- **WHEN** a test injects user speech during assistant playback
|
|
- **THEN** the state sequence SHALL include `speaking -> interrupted -> listening` and active LLM/TTS/playback tasks SHALL be cancelled
|
|
|
|
#### Scenario: Streaming STT and TTS are tested
|
|
- **WHEN** fake providers emit partial transcripts, final transcript, LLM deltas, sentence segments, and TTS chunks
|
|
- **THEN** tests SHALL verify event order, context writes, and playback order
|
|
|
|
#### Scenario: Memory persists across restart
|
|
- **WHEN** a memory is saved and a new runtime instance starts with memory enabled
|
|
- **THEN** the memory SHALL be retrievable from SQLite and FAISS
|
|
|
|
#### Scenario: Tools are tested
|
|
- **WHEN** fake tool calls cover allowed, rejected, confirmation-required, timeout, and oversized output paths
|
|
- **THEN** Tool Router tests SHALL verify decisions, audit events, and sanitized tool results
|
|
|
|
#### Scenario: OpenSpec validation runs
|
|
- **WHEN** this planning change is complete
|
|
- **THEN** `openspec validate add-full-duplex-agent-voice-assistant --strict` and `openspec validate --all --strict` SHALL pass
|