[Wake/VAD/STT 与 Live runtime]:完成真实重复语音运行,包含云端ASR/TTS开关、run-live和临时上下文测试
This commit is contained in:
@@ -4,7 +4,7 @@
|
||||
|
||||
`add-voice-pet-pipeline` 已归档并生成 `voice-pet-pipeline` 主规范。当前仓库已有 Python 包、CLI acceptance、`.env` 配置读取、Provider 协议、mock 测试、桌宠资产和 OpenSpec 主规范,但真实运行能力仍不完整:没有 `run-live` 常驻命令,没有真实麦克风输入和扬声器输出,没有本地模型下载/检查,没有证明同一运行进程内可以连续多轮携带历史。
|
||||
|
||||
本设计聚焦第一版“无 GUI 的真实实时语音版”。GUI 桌宠窗口、长期记忆、播放中打断、跨设备 Transport 和云端 STT/TTS 均不在本变更范围内。
|
||||
本设计聚焦第一版“无 GUI 的真实实时语音版”。GUI 桌宠窗口、长期记忆、播放中打断和跨设备 Transport 不在本变更范围内;ASR/TTS 由 `.env` 的 `OWNER_SPEECH_PROVIDER=cloud|local` 控制,默认先走云端以保证真实可用。
|
||||
|
||||
## Goals
|
||||
|
||||
@@ -12,8 +12,8 @@
|
||||
2. 提供 `--once`,便于单轮真实验收和自动化测试。
|
||||
3. 使用 `.env` 作为默认配置来源,避免要求用户导出 shell 环境变量。
|
||||
4. 使用 `sounddevice` 真实读取本机麦克风。
|
||||
5. 使用 `sherpa-onnx` 本地 VAD/STT 模型,模型存放在项目 `models/`。
|
||||
6. 使用 macOS `say/afplay` 先保证本地 TTS 真实播出。
|
||||
5. 使用 `OWNER_SPEECH_PROVIDER=cloud|local` 明确选择语音模型路径。
|
||||
6. 默认 cloud 模式使用 NewAPI 的 `mimo-v2.5-asr` 和 `mimo-v2.5-tts`,local 模式保留 `sherpa-onnx` 和本地 TTS 路径。
|
||||
7. 在同一个 `run-live` 进程内保留最近 user/assistant 历史,进程退出即丢弃。
|
||||
8. 新增自动化测试证明两轮 runtime、临时上下文、进程隔离和错误恢复。
|
||||
|
||||
@@ -34,10 +34,10 @@ CLI
|
||||
-> LiveVoiceRuntimeFactory
|
||||
-> SoundDeviceAudioTransport
|
||||
-> WakeDetector("小杰小杰")
|
||||
-> SherpaOnnxVadProvider
|
||||
-> SherpaOnnxSttProvider
|
||||
-> Energy/Sherpa VAD provider
|
||||
-> CloudAsrSttProvider or SherpaOnnxSttProvider
|
||||
-> OpenAI/NewAPI LlmProvider
|
||||
-> MacOsSayTtsProvider
|
||||
-> CloudTtsProvider or MacOsSayTtsProvider
|
||||
-> ConversationContext(empty per process)
|
||||
-> TerminalRuntimeReporter
|
||||
|
||||
@@ -58,7 +58,7 @@ LiveVoiceRuntime.run()
|
||||
1. Parse CLI args.
|
||||
2. Load `.env` and merge only explicitly supported defaults.
|
||||
3. Validate LLM config.
|
||||
4. Validate models when live mode requires local VAD/STT.
|
||||
4. Validate speech provider mode; cloud mode validates credentials and model names, local mode validates project-local VAD/STT models.
|
||||
5. Validate audio devices.
|
||||
6. Create one `ConversationContext` for the process.
|
||||
7. Enter loop:
|
||||
@@ -184,12 +184,13 @@ Test requirements:
|
||||
|
||||
## Wake/VAD/STT Design
|
||||
|
||||
First implementation can use one of two local strategies:
|
||||
First implementation can use one of two strategies:
|
||||
|
||||
1. Preferred: sherpa-onnx VAD for endpointing plus sherpa-onnx STT for short wake windows and user utterances.
|
||||
2. Future replacement: dedicated local KWS provider for “小杰小杰”.
|
||||
1. Default cloud mode: local VAD cuts speech segments, `mimo-v2.5-asr` transcribes wake and user segments through NewAPI-compatible `/v1/audio/transcriptions`.
|
||||
2. Local mode: sherpa-onnx VAD/STT uses project-local model assets under `models/`.
|
||||
3. Future replacement: dedicated local KWS provider for “小杰小杰”.
|
||||
|
||||
The implementation must not send raw audio to the cloud for wake, VAD, or STT. If a dedicated wake model is unavailable, wake detection may transcribe short local segments and search for the normalized wake word.
|
||||
Cloud mode sends only VAD-cut speech segments to ASR, not the continuous microphone stream. Local mode does not send wake, VAD, or STT audio to the cloud. If a dedicated wake model is unavailable, wake detection may transcribe short speech segments and search for the normalized wake word.
|
||||
|
||||
## Model Download Design
|
||||
|
||||
@@ -241,4 +242,4 @@ No persistent data migration is needed. Existing `.env` remains valid. New optio
|
||||
|
||||
1. Which exact sherpa-onnx Chinese STT model gives the best latency/accuracy tradeoff on this Mac.
|
||||
2. Whether future wake detection should use a dedicated KWS model instead of short-window STT.
|
||||
3. Whether TTS should later move from macOS `say` to sherpa-onnx TTS for consistent voice and offline packaging.
|
||||
3. Whether local TTS should later move from macOS `say` to sherpa-onnx TTS for consistent voice and offline packaging.
|
||||
|
||||
Reference in New Issue
Block a user