Owner/openspec/changes/complete-live-repeat-voice-runtime/design.md

补齐真实可重复对话的实时语音运行版技术设计

Context

add-voice-pet-pipeline 已归档并生成 voice-pet-pipeline 主规范。当前仓库已有 Python 包、CLI acceptance、.env 配置读取、Provider 协议、mock 测试、桌宠资产和 OpenSpec 主规范，但真实运行能力仍不完整：没有 run-live 常驻命令，没有真实麦克风输入和扬声器输出，没有本地模型下载/检查，没有证明同一运行进程内可以连续多轮携带历史。

本设计聚焦第一版“无 GUI 的真实实时语音版”。GUI 桌宠窗口、长期记忆、播放中打断、跨设备 Transport 和云端 STT/TTS 均不在本变更范围内。

Goals

1. 提供 owner_voice_pet run-live，默认常驻重复语音对话。
2. 提供 --once，便于单轮真实验收和自动化测试。
3. 使用 .env 作为默认配置来源，避免要求用户导出 shell 环境变量。
4. 使用 sounddevice 真实读取本机麦克风。
5. 使用 sherpa-onnx 本地 VAD/STT 模型，模型存放在项目 models/。
6. 使用 macOS say/afplay 先保证本地 TTS 真实播出。
7. 在同一个 run-live 进程内保留最近 user/assistant 历史，进程退出即丢弃。
8. 新增自动化测试证明两轮 runtime、临时上下文、进程隔离和错误恢复。

Non-Goals

1. 不实现 GUI 桌宠窗口实时联动。
2. 不实现长期记忆、数据库或跨进程会话恢复。
3. 不实现播放中用户打断。
4. 不把模型文件提交到 Git。
5. 不把 acceptance fixture 当作 run-live 默认输入。
6. 不要求自动化测试依赖真实麦克风、扬声器或真实模型。

Architecture

CLI
  -> AppConfig.from_dotenv(".env")
  -> LiveVoiceRuntimeFactory
    -> SoundDeviceAudioTransport
    -> WakeDetector("小杰小杰")
    -> SherpaOnnxVadProvider
    -> SherpaOnnxSttProvider
    -> OpenAI/NewAPI LlmProvider
    -> MacOsSayTtsProvider
    -> ConversationContext(empty per process)
    -> TerminalRuntimeReporter

LiveVoiceRuntime.run()
  -> standby loop
  -> wait_wake()
  -> record_utterance_with_vad()
  -> transcribe()
  -> build messages with temporary history
  -> generate reply
  -> synthesize/play
  -> append assistant
  -> recover to standby

Runtime Lifecycle

1. Parse CLI args.
2. Load .env and merge only explicitly supported defaults.
3. Validate LLM config.
4. Validate models when live mode requires local VAD/STT.
5. Validate audio devices.
6. Create one ConversationContext for the process.
7. Enter loop:
   1. report 待机;
   2. wait for wake word;
   3. report 唤醒命中;
   4. record utterance with VAD;
   5. report 转写中;
   6. transcribe;
   7. append user to context;
   8. report 思考中;
   9. call LLM with system + retained history + current user;
   10. report 播放中;
   11. synthesize and play;
   12. append assistant to context;
   13. report 恢复待机;
   14. break only if --once.
8. On Ctrl-C, close audio streams and exit without persisting context.

Interfaces

LiveVoiceRuntime

class LiveVoiceRuntime:
    def run(self, *, once: bool = False) -> RuntimeSummary: ...
    def run_turn(self, turn_id: int) -> TurnResult: ...
    def shutdown(self) -> None: ...

run() owns the loop. run_turn() owns one wake -> record -> STT -> LLM -> TTS/play -> standby cycle. shutdown() closes transport streams and removes temporary TTS files.

RuntimeReporter

class RuntimeReporter(Protocol):
    def status(self, state: str, message: str, *, turn_id: int | None = None) -> None: ...
    def error(self, stage: str, code: str, message: str, *, turn_id: int | None = None) -> None: ...

The default implementation prints short Chinese status lines to stdout/stderr and redacts credentials.

SpeechModelManifest

@dataclass(frozen=True)
class SpeechModelManifest:
    models_dir: Path
    vad_model_path: Path
    stt_model_dir: Path
    sample_rate: int = 16000

The manifest is used by download_speech_models.py, model-check, and live provider construction.

SoundDeviceAudioTransport

class SoundDeviceAudioTransport:
    def open_input(self, sample_rate: int, channels: int, device_id: int | None = None) -> None: ...
    def read_chunk(self, timeout_s: float) -> AudioFrame: ...
    def record_until_endpoint(self, vad: VadProvider, limits: RecordingLimits) -> AudioSegment: ...
    def play_pcm(self, segment: AudioSegment) -> PlaybackResult: ...
    def close(self) -> None: ...

The implementation keeps sounddevice imports optional so unit tests and non-audio commands still import without audio dependencies.

MacOsSayTtsProvider

class MacOsSayTtsProvider:
    def synthesize_to_file(self, text: str) -> Path: ...
    def synthesize(self, text: str) -> AudioSegment: ...

First version may return a file-backed audio segment or call say -o <tmpfile> --data-format=LEF32@16000. Playback may use afplay.

State Machine

initializing
  -> standby
  -> wake_listening
  -> wake_hit
  -> recording
  -> transcribing
  -> thinking
  -> speaking
  -> recovering_to_standby
  -> standby

recoverable turn error
  -> error
  -> recovering_to_standby
  -> standby

fatal startup error
  -> startup_failed
  -> exit

Fatal startup errors include invalid .env, missing required model directory, no audio input device, and no usable live audio dependency. Recoverable turn errors include no speech after wake, empty STT text, LLM timeout, TTS failure, and playback failure.

Temporary Context Design

Each LiveVoiceRuntime receives a newly constructed ConversationContext. This object:

1. starts with only system prompt behavior;
2. appends user text after valid STT;
3. appends assistant text after non-empty LLM reply;
4. builds messages for every LLM request using retained history;
5. truncates by context_max_messages and context_max_chars;
6. is never serialized;
7. is never stored in a module-level global;
8. is lost when the process exits.

Test requirements:

1. A fake two-turn runtime must prove the second LLM call includes first turn user/assistant.
2. A new runtime instance must start with no prior messages.
3. Context truncation must still preserve current user input.

Wake/VAD/STT Design

First implementation can use one of two local strategies:

1. Preferred: sherpa-onnx VAD for endpointing plus sherpa-onnx STT for short wake windows and user utterances.
2. Future replacement: dedicated local KWS provider for “小杰小杰”.

The implementation must not send raw audio to the cloud for wake, VAD, or STT. If a dedicated wake model is unavailable, wake detection may transcribe short local segments and search for the normalized wake word.

Model Download Design

scripts/download_speech_models.py --dir models creates:

models/
  manifest.json
  vad/
    ...
  stt/
    ...

manifest.json records model names, source URLs, expected key files, and sample rate. The script must be idempotent: if files exist and pass checks, it should skip or report already present.

Error Handling

All live errors must convert to structured codes:

1. LIVE_CONFIG_INVALID
2. LIVE_MODEL_UNAVAILABLE
3. LIVE_AUDIO_DEVICE_UNAVAILABLE
4. LIVE_WAKE_TIMEOUT
5. LIVE_NO_SPEECH
6. LIVE_STT_EMPTY
7. LIVE_LLM_FAILED
8. LIVE_TTS_FAILED
9. LIVE_PLAYBACK_FAILED
10. LIVE_INTERRUPTED

Startup errors return non-zero CLI exit codes. Turn-level errors in default loop log and return to standby. In --once, turn-level errors return non-zero except no-speech or empty transcript may return a documented recoverable code.

Testing Strategy

1. Unit tests use fake audio/provider objects and do not require live audio devices.
2. test_live_runtime_repeats_two_turns verifies two STT, two LLM, two TTS, two playback events and final standby.
3. test_live_runtime_temporary_context verifies second LLM messages include first user/assistant.
4. test_live_runtime_process_local_context verifies a new runtime has empty history.
5. test_model_check verifies missing/valid manifest behavior.
6. test_device_check mocks sounddevice device query.
7. Existing tests for acceptance, assets, security, config, transport, wake/VAD/STT remain passing.

Migration

No persistent data migration is needed. Existing .env remains valid. New optional .env keys receive defaults. Existing acceptance command remains available.

Open Questions

1. Which exact sherpa-onnx Chinese STT model gives the best latency/accuracy tradeoff on this Mac.
2. Whether future wake detection should use a dedicated KWS model instead of short-window STT.
3. Whether TTS should later move from macOS say to sherpa-onnx TTS for consistent voice and offline packaging.