Owner/openspec/specs/voice-pet-pipeline/spec.md

# voice-pet-pipeline Specification

## Purpose
TBD - created by archiving change add-voice-pet-pipeline. Update Purpose after archive.
## Requirements
### Requirement: Python desktop pet runtime
The system SHALL be specified as a local Python desktop pet application that owns the voice pipeline, desktop pet state, local audio input, and local audio output in the first version.

#### Scenario: App starts in local desktop mode
- **WHEN** the user starts the future application on macOS
- **THEN** the system SHALL initialize as a local desktop pet process rather than a remote service or browser-only tool

#### Scenario: Implementation follows OpenSpec artifacts
- **WHEN** this OpenSpec change is implemented
- **THEN** the repository SHALL contain Python source code, automated tests, validation commands, and project-local pet assets aligned with the planning artifacts

### Requirement: Local microphone and speaker transport
The system SHALL use the local microphone as the first-version input Transport and the local system speaker as the first-version output Transport, and `run-live` SHALL exercise those real local devices by default.

#### Scenario: Microphone input is available
- **WHEN** the configured microphone is available and permitted
- **THEN** the live Transport SHALL provide streaming audio frames to wake detection, VAD, and STT stages

#### Scenario: Speaker output is available
- **WHEN** TTS returns a valid local audio segment or audio file and the configured speaker is available
- **THEN** the live Transport or macOS playback command SHALL play the audio through the local speaker

#### Scenario: Audio device is unavailable
- **WHEN** the microphone or speaker is missing, denied, unsupported, or inaccessible through `sounddevice`
- **THEN** the system SHALL expose a recoverable Transport error with a stable error code and SHALL NOT silently fall back to fixture audio in live mode

### Requirement: Wake word detection
The system SHALL listen locally for the Chinese wake word “小杰小杰” before accepting user speech for a conversation turn.

#### Scenario: Wake word is detected
- **WHEN** the user says “小杰小杰” and the wakeword provider returns confidence above the configured threshold
- **THEN** the pipeline SHALL transition from wake listening to speech detection

#### Scenario: Wake word is not detected
- **WHEN** background speech or noise does not match “小杰小杰”
- **THEN** the pipeline SHALL remain in wake listening and SHALL NOT invoke STT, LLM, or TTS

#### Scenario: Wake model fails
- **WHEN** the wakeword provider cannot load or process audio
- **THEN** the system SHALL report a wakeword error and SHALL NOT crash the desktop pet process

### Requirement: VAD speech endpoint detection
After wakeword detection, the system SHALL use VAD to identify when the user starts and stops speaking.

#### Scenario: User begins speaking
- **WHEN** VAD detects continuous speech above the configured start threshold
- **THEN** the pipeline SHALL begin recording the current utterance

#### Scenario: User stops speaking
- **WHEN** VAD detects continuous silence above the configured end threshold
- **THEN** the pipeline SHALL close the current audio segment and send it to STT

#### Scenario: User says nothing after wakeword
- **WHEN** no speech is detected before the configured no-speech timeout
- **THEN** the pipeline SHALL return to wake listening without invoking STT or LLM

#### Scenario: Recording exceeds maximum duration
- **WHEN** speech continues beyond the configured maximum utterance duration
- **THEN** the pipeline SHALL end the segment, mark the end reason, and continue to STT with the captured audio

### Requirement: Local STT transcription
The system SHALL transcribe captured user utterances through the configured STT provider; cloud mode SHALL use NewAPI-compatible ASR and local mode SHALL use project-local `sherpa-onnx` model assets under `models/`.

#### Scenario: STT succeeds
- **WHEN** configured STT returns non-empty text for a captured live audio segment
- **THEN** the pipeline SHALL add the trimmed text as a user message to the current process conversation context

#### Scenario: STT returns empty text
- **WHEN** STT returns empty text, punctuation-only text, or an invalid transcript
- **THEN** the live runtime SHALL skip LLM invocation and return to standby with a recoverable status

#### Scenario: STT provider fails
- **WHEN** the STT provider raises an error, cloud ASR fails, or local model loading fails
- **THEN** the system SHALL emit an STT or model error code and SHALL recover to a state where future wake attempts are possible if startup can continue safely

### Requirement: Conversation context management
The system SHALL maintain an in-memory conversation context for the active desktop pet session, and live repeated voice runtime SHALL define that session as the current `run-live` process only.

#### Scenario: User transcript is accepted
- **WHEN** a valid STT transcript is produced during a live turn
- **THEN** the system SHALL append it to the current process context as a user message before invoking the LLM

#### Scenario: Assistant reply completes
- **WHEN** the LLM and TTS stages complete a non-empty reply during a live turn
- **THEN** the system SHALL append the assistant text to the current process context

#### Scenario: Context exceeds configured budget
- **WHEN** the context exceeds the configured message or character budget
- **THEN** the system SHALL preserve the system prompt and most recent conversation turns while removing older ordinary messages

#### Scenario: Runtime exits
- **WHEN** the `run-live` process exits
- **THEN** the system SHALL discard the context and SHALL NOT persist it across process restarts

### Requirement: Cloud LLM streaming reply
The system SHALL use a cloud LLM provider for reply generation, with an OpenAI/NewAPI-compatible chat completions endpoint as the first-version runtime target.

#### Scenario: LLM stream begins
- **WHEN** the LLM provider receives valid context messages and configuration
- **THEN** it SHALL stream reply deltas rather than waiting for the full reply before producing output

#### Scenario: LLM model is configured
- **WHEN** the system starts
- **THEN** the LLM base URL, API key, and model name SHALL be read from `.env` or local uncommitted configuration and SHALL NOT be hard-coded in pipeline logic

#### Scenario: LLM request fails
- **WHEN** the LLM provider times out, is rate limited, lacks credentials, or encounters a network error
- **THEN** the pipeline SHALL emit a structured LLM error and transition through error recovery to wake listening

### Requirement: Local TTS synthesis and playback
The system SHALL synthesize assistant replies through the configured TTS provider and SHALL play synthesized speech through the local system output path; cloud mode SHALL use NewAPI-compatible TTS and local mode SHALL use a local playback-capable TTS provider.

#### Scenario: Reply text is ready for speech
- **WHEN** the LLM produces a non-empty reply for a live turn
- **THEN** the TTS provider SHALL synthesize that text into locally playable speech

#### Scenario: Playback starts
- **WHEN** the first synthesized audio output is available
- **THEN** the output Transport or macOS playback command SHALL begin playback through the local speaker

#### Scenario: TTS fails
- **WHEN** TTS returns empty audio, fails to create an audio file, or playback command fails
- **THEN** the live runtime SHALL emit a structured TTS or playback error and SHALL recover without terminating default loop mode

### Requirement: Pipeline state machine
The system SHALL expose deterministic pipeline states for `idle`, `wake_listening`, `speech_detecting`, `recording`, `transcribing`, `thinking`, `speaking`, `interrupted`, and `error_recovering`.

#### Scenario: Normal conversation turn
- **WHEN** wakeword, VAD, STT, LLM, TTS, and playback all succeed
- **THEN** the state sequence SHALL progress through wake listening, speech detection, recording, transcribing, thinking, speaking, and back to wake listening

#### Scenario: Recoverable provider error
- **WHEN** any provider fails during a conversation turn
- **THEN** the state machine SHALL enter error recovery and then return to wake listening after cleanup

#### Scenario: UI subscribes to state
- **WHEN** the pipeline state changes
- **THEN** the desktop pet UI SHALL be able to update its visible state without directly invoking audio or model providers

### Requirement: Desktop pet visual states
The system SHALL define desktop pet visual states for idle, listening, recording, thinking, speaking, and error feedback.

#### Scenario: Wakeword is detected
- **WHEN** the pipeline enters speech detection or recording
- **THEN** the desktop pet SHALL show a listening or recording visual state

#### Scenario: LLM is generating
- **WHEN** the pipeline enters thinking
- **THEN** the desktop pet SHALL show a thinking visual state

#### Scenario: TTS is playing
- **WHEN** the pipeline enters speaking
- **THEN** the desktop pet SHALL show a speaking visual state

#### Scenario: Error occurs
- **WHEN** the pipeline enters error recovery
- **THEN** the desktop pet SHALL show a concise visible error state and then return to idle or wake listening when recovered

### Requirement: Generated pet asset specification
The system SHALL specify that production desktop pet images are generated with the image generation skill when usable, or with a reproducible project-local fallback when image generation output is unavailable or fails validation, and saved inside the project as transparent PNG assets.

#### Scenario: Image generation output is usable
- **WHEN** pet images are generated with `imagegen` and pass role and transparency validation
- **THEN** the images SHALL be processed for transparency, validated, and saved to a project asset directory

#### Scenario: Image generation output is unavailable or invalid
- **WHEN** image generation output is unavailable, inaccessible as a project file, or fails role/transparency validation
- **THEN** the system SHALL use a reproducible project-local fallback to create transparent PNG pet state assets and SHALL validate them before use

#### Scenario: Planning phase is executed
- **WHEN** this OpenSpec change is implemented
- **THEN** generated or generated-derived pet assets SHALL be saved inside the project and SHALL NOT be referenced from a temporary generation directory

### Requirement: Audio feedback suppression
The system SHALL prevent the desktop pet's own TTS playback from being treated as a new user wakeword or speech input.

#### Scenario: TTS playback is active
- **WHEN** the output Transport is playing synthesized assistant speech
- **THEN** wakeword detection and VAD input processing SHALL be suppressed or ignored for that playback window by default

#### Scenario: User speaks during playback
- **WHEN** the user speaks while TTS playback is active in the first version
- **THEN** the system SHALL ignore that speech unless a future interruption feature is explicitly specified

### Requirement: Structured errors and logging
The system SHALL represent stage failures with structured error codes and log state transitions, latency metrics, and provider names.

#### Scenario: Stage transition occurs
- **WHEN** the pipeline changes state
- **THEN** the system SHALL log the turn ID, old state, new state, timestamp, and triggering event

#### Scenario: Provider call completes
- **WHEN** a provider call succeeds or fails
- **THEN** the system SHALL log provider name, stage, duration, and error code if present

#### Scenario: Sensitive data is present
- **WHEN** logs are written
- **THEN** the system SHALL NOT log API keys, authorization headers, or raw credentials

### Requirement: Performance targets
The system SHALL define measurable first-version latency and reliability targets for wakeword response, endpoint detection, LLM first output, and speech playback.

#### Scenario: Wakeword latency is measured
- **WHEN** wakeword detection succeeds in a normal local environment
- **THEN** the system SHALL target visible listening feedback within 800 ms

#### Scenario: User stops speaking
- **WHEN** the user stops speaking after a normal utterance
- **THEN** VAD endpoint detection SHALL target transition to transcription within 900 ms

#### Scenario: LLM begins responding
- **WHEN** a valid transcript has been sent to the LLM provider
- **THEN** the system SHALL target first playable reply text within 3.5 seconds

#### Scenario: Normal reply is spoken
- **WHEN** a normal Chinese question under 10 seconds is processed
- **THEN** the system SHALL target the start of spoken playback within 5 seconds P95 after the user stops speaking

### Requirement: Security and privacy
The system SHALL protect credentials and minimize audio persistence.

#### Scenario: API key is required
- **WHEN** the LLM provider needs an OpenAI API key
- **THEN** the key SHALL be loaded from `.env` or local uncommitted configuration and SHALL NOT be committed

#### Scenario: Audio is processed
- **WHEN** user speech is captured for STT
- **THEN** raw audio SHALL NOT be persisted by default

#### Scenario: Cloud LLM is invoked
- **WHEN** the system sends data to the cloud LLM
- **THEN** it SHALL send transcript text and necessary conversation context, not raw microphone audio

### Requirement: Testability
The system SHALL be designed so each stage can be tested with mock providers, file-based audio fixtures, and fake live runtime components without requiring real devices in automated tests.

#### Scenario: Repeated runtime is unit tested
- **WHEN** fake live providers produce two deterministic turns
- **THEN** tests SHALL verify two STT calls, two LLM calls, two TTS calls, two playback calls, and final return to standby

#### Scenario: Temporary context is unit tested
- **WHEN** fake live providers run two turns in one runtime instance
- **THEN** tests SHALL verify the second LLM request includes the first turn user and assistant messages

#### Scenario: Process-local context is unit tested
- **WHEN** a second runtime instance is created after a first instance has conversation history
- **THEN** tests SHALL verify the second instance starts with no user/assistant history

#### Scenario: OpenSpec planning validation runs
- **WHEN** this change is complete before implementation
- **THEN** `openspec validate complete-live-repeat-voice-runtime --strict` and `openspec validate --all --strict` SHALL pass

### Requirement: Module-level git commit gates
The system implementation process SHALL require an immediate Git commit after each major module or milestone is completed, where a major module means a top-level task group from `tasks.md` or a phase milestone from `proposal.md`.

#### Scenario: Major module is completed
- **WHEN** an implementer completes a top-level task group or milestone phase
- **THEN** the implementer SHALL run the applicable build, tests, lint/type checks, and OpenSpec validation before committing

#### Scenario: Build validation fails before commit
- **WHEN** the applicable build, tests, lint/type checks, or OpenSpec validation fail
- **THEN** the implementer SHALL NOT create the module commit until the failure is fixed and validation passes

#### Scenario: Module commit is created
- **WHEN** validation passes for the completed module
- **THEN** the implementer SHALL immediately create a Git commit using the Chinese message format “[模块名]：完成[具体功能描述]，包含[关键变更]”

#### Scenario: Intermediate work remains after module completion
- **WHEN** module-related changes remain unstaged or uncommitted after the module is completed
- **THEN** the implementer SHALL include those changes in the module commit or explicitly separate unrelated external changes before continuing to the next module

### Requirement: Live repeat voice runtime
The system SHALL provide a `run-live` command that performs real repeated voice conversation with local microphone input, configured speech recognition and speech synthesis providers, cloud LLM reply generation, local speaker playback, and automatic return to standby.

#### Scenario: Live runtime starts in standby
- **WHEN** the user runs `PYTHONPATH=src python3.11 -m owner_voice_pet run-live`
- **THEN** the system SHALL load `.env`, validate required live dependencies, initialize local audio/model providers, and enter a standby listening loop

#### Scenario: Live runtime completes two turns
- **WHEN** the user wakes the system with “小杰小杰”, asks a question, hears the reply, then wakes it again and asks another question
- **THEN** the system SHALL complete wake, recording, STT, LLM, TTS, playback for both turns and SHALL return to standby after each turn

#### Scenario: Once mode completes one turn
- **WHEN** the user runs `PYTHONPATH=src python3.11 -m owner_voice_pet run-live --once`
- **THEN** the system SHALL run at most one wake-to-playback turn and exit after the turn completes or fails with a documented live error

### Requirement: Temporary in-process conversation history
The live runtime SHALL maintain conversation history only in memory for the current process and SHALL include retained user/assistant history in later LLM requests during that same process.

#### Scenario: Second turn uses first turn history
- **WHEN** the first live turn appends a user transcript and an assistant reply
- **AND** a second live turn sends an LLM request
- **THEN** the second request SHALL include the first turn user message and assistant message unless configured context limits require truncation

#### Scenario: New runtime starts empty
- **WHEN** a new `run-live` process or new live runtime instance starts
- **THEN** it SHALL NOT load user/assistant history from a previous process or previous runtime instance

#### Scenario: Process exits
- **WHEN** the live runtime exits for any reason
- **THEN** conversation history SHALL be discarded and SHALL NOT be written to disk, database, logs, or model files

### Requirement: Local speech model management
The system SHALL provide project-local speech model preparation and diagnostics for live VAD/STT operation, storing downloaded model artifacts under `models/` without committing them to Git.

#### Scenario: Models are downloaded
- **WHEN** the user runs `python3.11 scripts/download_speech_models.py --dir models`
- **THEN** the script SHALL create or update a project-local model directory with the files required by the configured VAD/STT providers

#### Scenario: Model check succeeds
- **WHEN** required dependencies and model files are available
- **THEN** `PYTHONPATH=src python3.11 -m owner_voice_pet model-check` SHALL exit successfully and report the model directory and checked providers

#### Scenario: Model check fails
- **WHEN** `sherpa-onnx` is unavailable, a model file is missing, or a model cannot be loaded
- **THEN** `model-check` SHALL fail with a structured model error and SHALL NOT start live microphone listening

### Requirement: Configurable speech provider mode
The system SHALL read `OWNER_SPEECH_PROVIDER` from `.env` to choose between cloud speech providers and local speech providers for first-version live runtime.

#### Scenario: Cloud speech provider is selected
- **WHEN** `OWNER_SPEECH_PROVIDER=cloud`
- **THEN** live ASR SHALL use the configured cloud model from `OWNER_ASR_MODEL`, live TTS SHALL use the configured cloud model from `OWNER_TTS_MODEL`, and the implementation SHALL default those models to `mimo-v2.5-asr` and `mimo-v2.5-tts`

#### Scenario: Local speech provider is selected
- **WHEN** `OWNER_SPEECH_PROVIDER=local`
- **THEN** live ASR/VAD SHALL use project-local speech model assets and local TTS SHALL use a local playback-capable provider

#### Scenario: Speech provider is invalid
- **WHEN** `OWNER_SPEECH_PROVIDER` is neither `cloud` nor `local`
- **THEN** startup validation SHALL fail with a structured configuration error

### Requirement: Live audio device readiness
The system SHALL provide live audio device diagnostics and SHALL use `sounddevice` for first-version real microphone and speaker access.

#### Scenario: Device check succeeds
- **WHEN** `sounddevice` can be imported and at least one input and one output device are available
- **THEN** `PYTHONPATH=src python3.11 -m owner_voice_pet device-check` SHALL exit successfully and report usable audio devices

#### Scenario: Device check fails
- **WHEN** `sounddevice` is missing, device query fails, microphone permission is denied, or no usable input/output device exists
- **THEN** `device-check` SHALL fail with a structured audio device error

#### Scenario: Live runtime uses real devices
- **WHEN** `run-live` starts successfully
- **THEN** it SHALL use the local microphone as its default input and the local speaker or macOS playback command as its default output rather than fixture audio

### Requirement: Live terminal state reporting
The live runtime SHALL emit concise Chinese terminal status messages for observable runtime states.

#### Scenario: Normal turn status
- **WHEN** a live turn succeeds
- **THEN** terminal output SHALL include states equivalent to standby, wake hit, recording, transcribing, thinking, speaking, and returning to standby

#### Scenario: Recoverable error status
- **WHEN** a live turn encounters empty STT, LLM failure, TTS failure, or playback failure
- **THEN** terminal output SHALL include the failing stage and a stable error code or recoverable explanation

### Requirement: Live runtime error recovery
The live runtime SHALL recover from turn-level failures and continue listening in default loop mode.

#### Scenario: LLM fails during default loop
- **WHEN** the LLM provider times out, is rate limited, or returns a network error during a live turn
- **THEN** the system SHALL report `LIVE_LLM_FAILED` or an equivalent structured error and SHALL return to standby without terminating the process

#### Scenario: TTS or playback fails during default loop
- **WHEN** local TTS or audio playback fails during a live turn
- **THEN** the system SHALL report the failure and SHALL return to standby without terminating the process

#### Scenario: Startup dependency fails
- **WHEN** `.env`, required models, or audio devices are unavailable at startup
- **THEN** the system SHALL exit with a documented non-zero status instead of entering a fake live loop