Owner/openspec/changes/archive/2026-06-17-complete-live-repeat-voice-runtime/design.md

补齐真实可重复对话的实时语音运行版技术设计

Context

add-voice-pet-pipeline 已归档并生成 voice-pet-pipeline 主规范。当前仓库已有 Python 包、CLI acceptance、.env 配置读取、Provider 协议、mock 测试、桌宠资产和 OpenSpec 主规范，但真实运行能力仍不完整：没有 run-live 常驻命令，没有真实麦克风输入和扬声器输出，没有本地模型下载/检查，没有证明同一运行进程内可以连续多轮携带历史。

本设计聚焦第一版“无 GUI 的真实实时语音版”。GUI 桌宠窗口、长期记忆、播放中打断和跨设备 Transport 不在本变更范围内；ASR/TTS 由 .env 的 OWNER_SPEECH_PROVIDER=cloud|local 控制，默认先走云端以保证真实可用。

Goals

1. 提供 owner_voice_pet run-live，默认常驻重复语音对话。
2. 提供 --once，便于单轮真实验收和自动化测试。
3. 使用 .env 作为默认配置来源，避免要求用户导出 shell 环境变量。
4. 使用 sounddevice 真实读取本机麦克风。
5. 使用 OWNER_SPEECH_PROVIDER=cloud|local 明确选择语音模型路径。
6. 默认 cloud 模式使用 NewAPI 的 mimo-v2.5-asr 和 mimo-v2.5-tts，local 模式保留 sherpa-onnx 和本地 TTS 路径。
7. 在同一个 run-live 进程内保留最近 user/assistant 历史，进程退出即丢弃。
8. 新增自动化测试证明两轮 runtime、临时上下文、进程隔离和错误恢复。

Non-Goals

1. 不实现 GUI 桌宠窗口实时联动。
2. 不实现长期记忆、数据库或跨进程会话恢复。
3. 不实现播放中用户打断。
4. 不把模型文件提交到 Git。
5. 不把 acceptance fixture 当作 run-live 默认输入。
6. 不要求自动化测试依赖真实麦克风、扬声器或真实模型。

Architecture

CLI
  -> AppConfig.from_dotenv(".env")
  -> LiveVoiceRuntimeFactory
    -> SoundDeviceAudioTransport
    -> WakeDetector("小杰小杰")
    -> Energy/Sherpa VAD provider
    -> CloudAsrSttProvider or SherpaOnnxSttProvider
    -> OpenAI/NewAPI LlmProvider
    -> CloudTtsProvider or MacOsSayTtsProvider
    -> ConversationContext(empty per process)
    -> TerminalRuntimeReporter

LiveVoiceRuntime.run()
  -> standby loop
  -> wait_wake()
  -> record_utterance_with_vad()
  -> transcribe()
  -> build messages with temporary history
  -> generate reply
  -> synthesize/play
  -> append assistant
  -> recover to standby

Runtime Lifecycle

1. Parse CLI args.
2. Load .env and merge only explicitly supported defaults.
3. Validate LLM config.
4. Validate speech provider mode; cloud mode validates credentials and model names, local mode validates project-local VAD/STT models.
5. Validate audio devices.
6. Create one ConversationContext for the process.
7. Enter loop:
   1. report 待机;
   2. wait for wake word;
   3. report 唤醒命中;
   4. record utterance with VAD;
   5. report 转写中;
   6. transcribe;
   7. append user to context;
   8. report 思考中;
   9. call LLM with system + retained history + current user;
   10. report 播放中;
   11. synthesize and play;
   12. append assistant to context;
   13. report 恢复待机;
   14. break only if --once.
8. On Ctrl-C, close audio streams and exit without persisting context.

Interfaces

LiveVoiceRuntime

class LiveVoiceRuntime:
    def run(self, *, once: bool = False) -> RuntimeSummary: ...
    def run_turn(self, turn_id: int) -> TurnResult: ...
    def shutdown(self) -> None: ...

run() owns the loop. run_turn() owns one wake -> record -> STT -> LLM -> TTS/play -> standby cycle. shutdown() closes transport streams and removes temporary TTS files.

RuntimeReporter

class RuntimeReporter(Protocol):
    def status(self, state: str, message: str, *, turn_id: int | None = None) -> None: ...
    def error(self, stage: str, code: str, message: str, *, turn_id: int | None = None) -> None: ...

The default implementation prints short Chinese status lines to stdout/stderr and redacts credentials.

SpeechModelManifest

@dataclass(frozen=True)
class SpeechModelManifest:
    models_dir: Path
    vad_model_path: Path
    stt_model_dir: Path
    sample_rate: int = 16000

The manifest is used by download_speech_models.py, model-check, and live provider construction.

SoundDeviceAudioTransport

class SoundDeviceAudioTransport:
    def open_input(self, sample_rate: int, channels: int, device_id: int | None = None) -> None: ...
    def read_chunk(self, timeout_s: float) -> AudioFrame: ...
    def record_until_endpoint(self, vad: VadProvider, limits: RecordingLimits) -> AudioSegment: ...
    def play_pcm(self, segment: AudioSegment) -> PlaybackResult: ...
    def close(self) -> None: ...

The implementation keeps sounddevice imports optional so unit tests and non-audio commands still import without audio dependencies.

MacOsSayTtsProvider

class MacOsSayTtsProvider:
    def synthesize_to_file(self, text: str) -> Path: ...
    def synthesize(self, text: str) -> AudioSegment: ...

First version may return a file-backed audio segment or call say -o <tmpfile> --data-format=LEF32@16000. Playback may use afplay.

State Machine

initializing
  -> standby
  -> wake_listening
  -> wake_hit
  -> recording
  -> transcribing
  -> thinking
  -> speaking
  -> recovering_to_standby
  -> standby

recoverable turn error
  -> error
  -> recovering_to_standby
  -> standby

fatal startup error
  -> startup_failed
  -> exit

Fatal startup errors include invalid .env, missing required model directory, no audio input device, and no usable live audio dependency. Recoverable turn errors include no speech after wake, empty STT text, LLM timeout, TTS failure, and playback failure.

Temporary Context Design

Each LiveVoiceRuntime receives a newly constructed ConversationContext. This object:

1. starts with only system prompt behavior;
2. appends user text after valid STT;
3. appends assistant text after non-empty LLM reply;
4. builds messages for every LLM request using retained history;
5. truncates by context_max_messages and context_max_chars;
6. is never serialized;
7. is never stored in a module-level global;
8. is lost when the process exits.

Test requirements:

1. A fake two-turn runtime must prove the second LLM call includes first turn user/assistant.
2. A new runtime instance must start with no prior messages.
3. Context truncation must still preserve current user input.

Wake/VAD/STT Design

First implementation can use one of two strategies:

1. Default cloud mode: local VAD cuts speech segments, mimo-v2.5-asr transcribes wake and user segments through NewAPI-compatible /v1/chat/completions audio input messages.
2. Local mode: sherpa-onnx VAD/STT uses project-local model assets under models/.
3. Future replacement: dedicated local KWS provider for “小杰小杰”.

Cloud mode sends only VAD-cut speech segments to ASR, not the continuous microphone stream. Local mode does not send wake, VAD, or STT audio to the cloud. If a dedicated wake model is unavailable, wake detection may transcribe short speech segments and search for the normalized wake word.

Model Download Design

scripts/download_speech_models.py --dir models creates:

models/
  manifest.json
  vad/
    ...
  stt/
    ...

manifest.json records model names, source URLs, expected key files, and sample rate. The script must be idempotent: if files exist and pass checks, it should skip or report already present.

Error Handling

All live errors must convert to structured codes:

1. LIVE_CONFIG_INVALID
2. LIVE_MODEL_UNAVAILABLE
3. LIVE_AUDIO_DEVICE_UNAVAILABLE
4. LIVE_WAKE_TIMEOUT
5. LIVE_NO_SPEECH
6. LIVE_STT_EMPTY
7. LIVE_LLM_FAILED
8. LIVE_TTS_FAILED
9. LIVE_PLAYBACK_FAILED
10. LIVE_INTERRUPTED

Startup errors return non-zero CLI exit codes. Turn-level errors in default loop log and return to standby. In --once, turn-level errors return non-zero except no-speech or empty transcript may return a documented recoverable code.

Testing Strategy

1. Unit tests use fake audio/provider objects and do not require live audio devices.
2. test_live_runtime_repeats_two_turns verifies two STT, two LLM, two TTS, two playback events and final standby.
3. test_live_runtime_temporary_context verifies second LLM messages include first user/assistant.
4. test_live_runtime_process_local_context verifies a new runtime has empty history.
5. test_model_check verifies missing/valid manifest behavior.
6. test_device_check mocks sounddevice device query.
7. Existing tests for acceptance, assets, security, config, transport, wake/VAD/STT remain passing.

Migration

No persistent data migration is needed. Existing .env remains valid. New optional .env keys receive defaults. Existing acceptance command remains available.

Open Questions

1. Which exact sherpa-onnx Chinese STT model gives the best latency/accuracy tradeoff on this Mac.
2. Whether future wake detection should use a dedicated KWS model instead of short-window STT.
3. Whether local TTS should later move from macOS say to sherpa-onnx TTS for consistent voice and offline packaging.