[Wake/VAD/STT 与 Live runtime]:完成真实重复语音运行,包含云端ASR/TTS开关、run-live和临时上下文测试

This commit is contained in:
mkbk
2026-06-17 20:00:55 +08:00
parent ac97daa1e7
commit 4b7cd18a0f
20 changed files with 1043 additions and 68 deletions
@@ -4,7 +4,7 @@
`add-voice-pet-pipeline` 已归档并生成 `voice-pet-pipeline` 主规范。当前仓库已有 Python 包、CLI acceptance、`.env` 配置读取、Provider 协议、mock 测试、桌宠资产和 OpenSpec 主规范,但真实运行能力仍不完整:没有 `run-live` 常驻命令,没有真实麦克风输入和扬声器输出,没有本地模型下载/检查,没有证明同一运行进程内可以连续多轮携带历史。
本设计聚焦第一版“无 GUI 的真实实时语音版”。GUI 桌宠窗口、长期记忆、播放中打断跨设备 Transport 和云端 STT/TTS 均不在本变更范围内。
本设计聚焦第一版“无 GUI 的真实实时语音版”。GUI 桌宠窗口、长期记忆、播放中打断跨设备 Transport 不在本变更范围内ASR/TTS 由 `.env``OWNER_SPEECH_PROVIDER=cloud|local` 控制,默认先走云端以保证真实可用
## Goals
@@ -12,8 +12,8 @@
2. 提供 `--once`,便于单轮真实验收和自动化测试。
3. 使用 `.env` 作为默认配置来源,避免要求用户导出 shell 环境变量。
4. 使用 `sounddevice` 真实读取本机麦克风。
5. 使用 `sherpa-onnx` 本地 VAD/STT 模型,模型存放在项目 `models/`
6. 使用 macOS `say/afplay` 先保证本地 TTS 真实播出
5. 使用 `OWNER_SPEECH_PROVIDER=cloud|local` 明确选择语音模型路径
6. 默认 cloud 模式使用 NewAPI 的 `mimo-v2.5-asr``mimo-v2.5-tts`local 模式保留 `sherpa-onnx`本地 TTS 路径
7. 在同一个 `run-live` 进程内保留最近 user/assistant 历史,进程退出即丢弃。
8. 新增自动化测试证明两轮 runtime、临时上下文、进程隔离和错误恢复。
@@ -34,10 +34,10 @@ CLI
-> LiveVoiceRuntimeFactory
-> SoundDeviceAudioTransport
-> WakeDetector("小杰小杰")
-> SherpaOnnxVadProvider
-> SherpaOnnxSttProvider
-> Energy/Sherpa VAD provider
-> CloudAsrSttProvider or SherpaOnnxSttProvider
-> OpenAI/NewAPI LlmProvider
-> MacOsSayTtsProvider
-> CloudTtsProvider or MacOsSayTtsProvider
-> ConversationContext(empty per process)
-> TerminalRuntimeReporter
@@ -58,7 +58,7 @@ LiveVoiceRuntime.run()
1. Parse CLI args.
2. Load `.env` and merge only explicitly supported defaults.
3. Validate LLM config.
4. Validate models when live mode requires local VAD/STT.
4. Validate speech provider mode; cloud mode validates credentials and model names, local mode validates project-local VAD/STT models.
5. Validate audio devices.
6. Create one `ConversationContext` for the process.
7. Enter loop:
@@ -184,12 +184,13 @@ Test requirements:
## Wake/VAD/STT Design
First implementation can use one of two local strategies:
First implementation can use one of two strategies:
1. Preferred: sherpa-onnx VAD for endpointing plus sherpa-onnx STT for short wake windows and user utterances.
2. Future replacement: dedicated local KWS provider for “小杰小杰”.
1. Default cloud mode: local VAD cuts speech segments, `mimo-v2.5-asr` transcribes wake and user segments through NewAPI-compatible `/v1/audio/transcriptions`.
2. Local mode: sherpa-onnx VAD/STT uses project-local model assets under `models/`.
3. Future replacement: dedicated local KWS provider for “小杰小杰”.
The implementation must not send raw audio to the cloud for wake, VAD, or STT. If a dedicated wake model is unavailable, wake detection may transcribe short local segments and search for the normalized wake word.
Cloud mode sends only VAD-cut speech segments to ASR, not the continuous microphone stream. Local mode does not send wake, VAD, or STT audio to the cloud. If a dedicated wake model is unavailable, wake detection may transcribe short speech segments and search for the normalized wake word.
## Model Download Design
@@ -241,4 +242,4 @@ No persistent data migration is needed. Existing `.env` remains valid. New optio
1. Which exact sherpa-onnx Chinese STT model gives the best latency/accuracy tradeoff on this Mac.
2. Whether future wake detection should use a dedicated KWS model instead of short-window STT.
3. Whether TTS should later move from macOS `say` to sherpa-onnx TTS for consistent voice and offline packaging.
3. Whether local TTS should later move from macOS `say` to sherpa-onnx TTS for consistent voice and offline packaging.