Owner/openspec/changes/separate-wake-and-realtime-transcript/specs/voice-pet-pipeline/spec.md

ADDED Requirements

Requirement: Live assistant pipeline events

The live runtime SHALL execute as a stage-based voice assistant pipeline with stable events for every externally observable stage.

Scenario: Successful live turn emits ordered events

• WHEN a live turn completes successfully
• THEN the event stream SHALL include wake listening, wake detected, acknowledgement, capture started, speech started, speech ended, STT started, final transcript, LLM started, TTS started, playback finished, and standby resumed in that order

Scenario: Terminal reporting consumes events

• WHEN the terminal reporter prints user-facing live status
• THEN it SHALL derive the status from pipeline events rather than duplicating stage logic

Requirement: Primary speaker endpointing

The live runtime SHALL support a per-turn primary speaker endpoint mode that ends the user utterance when the speaker who started the utterance is no longer present.

Scenario: Primary speaker disappears

• WHEN the user starts speaking after wake and their temporary per-turn voice profile is established
• THEN capture SHALL end after the configured primary speaker absence duration even if background noise or non-primary speech remains

Scenario: Temporary profile is insufficient

• WHEN the live runtime cannot establish a reliable per-turn voice profile because speech is too short or too quiet
• THEN capture SHALL fall back to configured VAD silence endpointing and SHALL NOT hang indefinitely

Scenario: No persistent voiceprint

• WHEN the process exits
• THEN no primary speaker profile SHALL be stored or reused across runs

Requirement: Low latency capture and first utterance preservation

The live runtime SHALL preserve the start of the user's formal utterance after wake acknowledgement and SHALL end capture promptly after the primary speaker stops.

Scenario: Acknowledgement finishes before the user speaks

• WHEN acknowledgement playback completes and the runtime enters capture
• THEN the runtime SHALL NOT perform an additional timed microphone read-and-discard window before listening for the user's formal question

Scenario: Queued microphone frames are available

• WHEN the live microphone transport has multiple frames queued
• THEN the transport SHALL return the available batch to the pipeline so wake and capture stages do not process stale audio one frame at a time

Scenario: Primary speaker profile becomes ready quickly

• WHEN the user starts the formal utterance after wake
• THEN the primary speaker endpoint SHALL use an independent profile readiness threshold no greater than the configured OWNER_SPEAKER_PROFILE_MIN_MS rather than waiting for generic VAD minimum duration

Scenario: User stops after a short first question

• WHEN the primary speaker profile is ready and the primary speaker disappears for the configured absence duration
• THEN capture SHALL close the utterance and proceed to STT without waiting for a second repeated question or generic VAD minimum duration

Requirement: Realtime transcript terminal output

The live runtime SHALL display recognized user utterance text in the terminal both during recording when partial transcript is available and after final STT succeeds before the LLM request is sent.

Scenario: User utterance partial is available during recording

• WHEN the user is speaking after wake and the local realtime STT session produces a changed partial transcript
• THEN the terminal output SHALL include a realtime transcript message containing the partial text before final STT completes

Scenario: Partial transcript is displayed

• WHEN a partial transcript is emitted
• THEN it SHALL be treated as user-visible feedback only and SHALL NOT be appended to the conversation context or sent to the LLM

Scenario: User utterance is transcribed

• WHEN a live turn captures a user utterance and STT returns non-empty text
• THEN the terminal output SHALL include a transcript message containing the recognized text before the thinking/LLM status is emitted

Scenario: Transcript is empty

• WHEN STT returns empty text, punctuation-only text, or an invalid transcript
• THEN the runtime SHALL NOT emit a misleading transcript as valid user input and SHALL recover to standby without invoking the LLM

Requirement: Local voice chain and noise filtering

The live runtime SHALL run wake, VAD, STT, realtime transcript, TTS, and capture-stage noise filtering locally by default, while continuing to send only the final user text and session history to the configured cloud LLM.

Scenario: Default speech provider is local

• WHEN the user starts run-live without overriding OWNER_SPEECH_PROVIDER
• THEN the runtime SHALL use local sherpa-onnx STT and local macOS TTS
• AND it SHALL NOT call cloud ASR or cloud TTS providers during the turn

Scenario: Capture audio is denoised before downstream stages

• WHEN OWNER_NOISE_FILTER_ENABLED=1 and the user speaks after wake acknowledgement
• THEN the capture stage SHALL pass microphone frames through the configured local denoiser before VAD analysis, realtime partial STT, and final STT segment assembly
• AND all three downstream consumers SHALL observe the same denoised frame content for a given captured frame

Scenario: Wake remains raw by default

• WHEN OWNER_WAKE_DENOISE_ENABLED is unset or 0
• THEN wake word detection SHALL receive raw microphone frames rather than denoised frames

Scenario: Denoiser model is missing

• WHEN models/denoise/gtcrn_simple.onnx is absent
• THEN model-check SHALL fail with a structured missing model error
• AND run-live SHALL NOT enter live microphone listening as if noise filtering were available

Scenario: Denoiser runtime fails

• WHEN the local denoiser raises an error while processing a formal user utterance frame
• THEN the current turn SHALL emit a stage error and recover to standby
• AND the runtime SHALL NOT invoke final STT, LLM, or TTS for that failed turn

Scenario: Local ASR uses the 2025 CTC model by default

• WHEN the default manifest is written by download_speech_models.py
• THEN the STT provider SHALL be declared as sherpa-onnx-streaming-zipformer2-ctc
• AND required STT files SHALL be tokens.txt and model.int8.onnx, not transducer encoder/decoder/joiner files

Scenario: Realtime partial filters noise

• WHEN local streaming STT emits a single meaningful character or a short transient regression
• THEN the terminal reporter SHALL NOT display that text as 实时转写
• AND final STT SHALL remain responsible for the text sent to the LLM

Scenario: Audio privacy is preserved

• WHEN local noise filtering and local STT run during capture
• THEN raw PCM, denoised PCM, VAD features, partial transcript metadata, and temporary speaker profile data SHALL remain process-local and SHALL NOT be sent to the cloud LLM

Requirement: Simulated microphone live acceptance

The system SHALL provide a deterministic simulated microphone acceptance command that exercises the current live VoiceAssistantPipeline without requiring a human to speak into a physical microphone.

Scenario: Simulated live command completes repeated turns

• WHEN the developer runs .venv/bin/python -m owner_voice_pet simulate-live --turns 2
• THEN the command SHALL feed simulated microphone audio frames into the live pipeline and complete two wake-to-playback turns
• AND the JSON output SHALL report two completed turns, zero failed turns, two final transcripts, two LLM calls, at least two playback completions, and standby resumed after each turn

Scenario: Simulated input includes noisy realtime partials

• WHEN simulated formal question frames contain short noisy partial text such as 家 or 家确
• THEN realtime transcript output SHALL filter those short noise partials and SHALL only report stable question text

Scenario: Simulated input includes background speech after the user stops

• WHEN simulated background speech remains after the primary speaker frames end
• THEN primary speaker endpointing SHALL close the formal utterance and SHALL NOT merge later simulated turns into the current question

Scenario: Simulated command can write and replay fixtures

• WHEN the developer passes --write-fixture path.jsonl
• THEN the command SHALL write the generated simulated microphone frames as JSONL
• AND a later run using --fixture path.jsonl SHALL replay the same frames and produce equivalent successful live acceptance results

Scenario: Simulated microphone frames are exhausted unexpectedly

• WHEN the simulated microphone transport runs out of frames before the expected turn completes
• THEN the command SHALL fail with a structured error instead of waiting indefinitely

Requirement: Real provider live fixture acceptance

The system SHALL support scripted full-chain acceptance that drives the live pipeline with generated microphone PCM while using real local speech providers, the configured cloud LLM, and local TTS/playback capability.

Scenario: Real provider fixture completes one turn

• WHEN generated wake and formal question PCM frames are passed through the live pipeline
• THEN the pipeline SHALL use the real local KWS, local VAD, local denoiser, local STT, configured cloud LLM, local TTS, and playback provider path to complete a wake-to-standby turn

Scenario: Real provider fixture completes repeated turns with temporary context

• WHEN two generated wake/question turns run in one process and the first turn gives a user fact
• THEN the second LLM request SHALL include the first turn's user and assistant history from process-local memory
• AND no history SHALL be read from disk or persisted after the process exits

Scenario: Zero playback drain still flushes queued echo

• WHEN ACK or TTS playback completes and OWNER_POST_PLAYBACK_DRAIN_MS=0
• THEN the runtime SHALL call flush_input() once to clear already queued playback echo
• AND it SHALL NOT perform an additional timed microphone read-and-discard loop

Scenario: No acknowledgement playback is configured

• WHEN OWNER_WAKE_ACK_TEXT is empty and no wake acknowledgement audio is played
• THEN the runtime SHALL NOT run post-playback drain before capture starts

Scenario: Fixture transport preserves future frames

• WHEN a memory or fixture transport represents pre-generated future microphone frames
• THEN tests SHALL be able to configure flush to record the cleanup call without deleting those future frames

Requirement: Wake acknowledgement before recording

The live runtime SHALL provide an audible local acknowledgement after local wake detection and before it starts recording the user's formal question.

Scenario: Wake is detected

• WHEN the local wakeword provider detects “小杰小杰”
• THEN the runtime SHALL play a short acknowledgement such as “我在” before entering the user utterance recording state

Scenario: Acknowledgement playback finishes

• WHEN the acknowledgement playback completes
• THEN the runtime SHALL clear buffered microphone input captured during acknowledgement playback before starting VAD recording for the user's question and SHALL NOT discard additional post-acknowledgement user speech by default

Requirement: Fast user utterance endpointing

The live runtime SHALL use project-local VAD capability with an energy-threshold fallback by default for user utterance endpoint detection and SHALL expose configurable silence timing so the recording stops promptly after the user stops speaking.

Scenario: User stops speaking after wake

• WHEN VAD observes the configured continuous silence duration after a started utterance
• THEN the runtime SHALL close the utterance segment and proceed to STT without waiting for the maximum recording duration

Scenario: Local VAD model is available

• WHEN OWNER_VAD_PROVIDER=local
• THEN the runtime SHALL use the project-local sherpa-onnx VAD model rather than a raw energy threshold for live user utterance endpointing

Scenario: Hybrid VAD is configured

• WHEN OWNER_VAD_PROVIDER=hybrid
• THEN the runtime SHALL load both the project-local VAD model and an energy-threshold fallback and SHALL treat a frame as speech when either detector reports speech

Scenario: Wake acknowledgement prompt ordering

• WHEN local wake detection succeeds and OWNER_WAKE_ACK_TEXT is non-empty
• THEN the terminal SHALL emit wake hit, acknowledgement, question prompt, and recording states in that order so the user is cued to speak only after acknowledgement playback

MODIFIED Requirements

Requirement: Wake word detection

The system SHALL listen locally for the Chinese wake word “小杰小杰” with a dedicated local wake word provider before accepting user speech for a conversation turn, and the default live runtime implementation SHALL use a project-local KWS model rather than cloud ASR or the formal user STT provider for wake detection.

Scenario: Wake word is detected

• WHEN the user says “小杰小杰” and the local wakeword provider returns confidence above the configured threshold
• THEN the pipeline SHALL transition from wake listening to speech detection without invoking the configured STT provider for the wake audio

Scenario: Wake word is not detected

• WHEN background speech or noise does not match “小杰小杰”
• THEN the pipeline SHALL remain in wake listening and SHALL NOT invoke STT, LLM, or TTS

Scenario: Wake model fails

• WHEN the wakeword provider cannot load or process audio
• THEN the system SHALL report a wakeword error and SHALL NOT crash the desktop pet process

Scenario: Wake audio is isolated from user utterance

• WHEN the local wakeword provider detects “小杰小杰”
• THEN the runtime SHALL reset the user utterance recorder and SHALL NOT add wake audio or wake transcript text to the LLM conversation context

Requirement: Local speech model management

The system SHALL provide project-local speech model preparation and diagnostics for live wake/VAD/STT operation, storing downloaded model artifacts under models/ without committing them to Git.

Scenario: Models are downloaded

• WHEN the user runs python3.11 scripts/download_speech_models.py --dir models
• THEN the script SHALL create or update a project-local model directory with the files required by the configured wake, VAD, STT, and denoiser providers

Scenario: Model check succeeds

• WHEN required wake, VAD, STT, denoiser dependencies and model files are available
• THEN PYTHONPATH=src python3.11 -m owner_voice_pet model-check SHALL exit successfully and report the model directory and checked providers

Scenario: Model check fails

• WHEN sherpa-onnx is unavailable, a wake model file is missing, a VAD/STT/denoiser model file is missing, or a model cannot be loaded
• THEN model-check SHALL fail with a structured model error and SHALL NOT start live microphone listening

Requirement: Live repeat voice runtime

The system SHALL provide a run-live command that performs real repeated voice conversation with local microphone input, local model wake detection, configured speech recognition and speech synthesis providers, cloud LLM reply generation, local speaker playback, and automatic return to standby.

Scenario: Live runtime starts in standby

• WHEN the user runs PYTHONPATH=src python3.11 -m owner_voice_pet run-live
• THEN the system SHALL load .env, validate required live dependencies, initialize local wake/audio/model providers, and enter a standby listening loop

Scenario: Live runtime completes two turns

• WHEN the user wakes the system with “小杰小杰”, asks a question, hears the reply, then wakes it again and asks another question
• THEN the system SHALL complete local wake, audible acknowledgement, recording, STT, transcript display, LLM, TTS, playback for both turns and SHALL return to standby after each turn

Scenario: Once mode completes one turn

• WHEN the user runs PYTHONPATH=src python3.11 -m owner_voice_pet run-live --once
• THEN the system SHALL run at most one local-wake-to-playback turn and exit after the turn completes or fails with a documented live error

Requirement: Live terminal state reporting

The live runtime SHALL emit concise Chinese terminal status messages for observable runtime states, including explicit user transcript output.

Scenario: Normal turn status

• WHEN a live turn succeeds
• THEN terminal output SHALL include states equivalent to standby, wake hit, recording, transcribing, transcript result, thinking, speaking, and returning to standby

Scenario: Recoverable error status

• WHEN a live turn encounters empty STT, LLM failure, TTS failure, or playback failure
• THEN terminal output SHALL include the failing stage and a stable error code or recoverable explanation

Requirement: Testability

The system SHALL be designed so each stage can be tested with mock providers, file-based audio fixtures, and fake live runtime components without requiring real devices in automated tests.

Scenario: Wake and STT separation is unit tested

• WHEN fake live providers run a turn with a wake frame and a user utterance
• THEN tests SHALL verify wake detection happens through the wake provider and the formal STT provider is called only for the user utterance

Scenario: Repeated runtime is unit tested

• WHEN fake live providers produce two deterministic turns
• THEN tests SHALL verify two wake detections, two STT calls, two LLM calls, two TTS calls, two playback calls, transcript output, and final return to standby

Scenario: Wake keyword does not pollute LLM input

• WHEN a wake frame contains “小杰小杰” and the later user utterance transcript is “第一问”
• THEN the current user message sent to the LLM SHALL be exactly “第一问” rather than a concatenation with the wake keyword