mkbk
122 lines
4.4 KiB
Markdown
122 lines
4.4 KiB
Markdown
# 独立本地唤醒与终端转写显示设计
|
|
|
|
## Context
|
|
|
|
真实运行日志显示，当前 `run-live` 会先截取语音段并使用 STT 判断是否包含“小杰小杰”。该实现把 wake detection 与 user utterance transcription 绑定在一起，导致唤醒慢、唤醒词污染正式问题、终端输出难以区分 STT 和 LLM 阶段。本设计把 wake word detection 升级为本地 KWS 模型路径，并把正式问题转写作为独立可见事件输出。
|
|
|
|
## Goals
|
|
|
|
1. 唤醒词“小杰小杰”由本地模型检测。
|
|
2. wake 阶段不调用云 ASR，也不调用正式 STT provider。
|
|
3. wake 命中后才开始正式问题录音和 STT。
|
|
4. 终端在 LLM 前显示正式问题转写文本。
|
|
5. 模型下载和检查覆盖 wake KWS、VAD、STT。
|
|
6. 自动化测试证明 wake/STT 分离、重复对话、上下文和错误恢复仍正常。
|
|
|
|
## Non-Goals
|
|
|
|
1. 不实现 GUI 桌宠窗口。
|
|
2. 不实现跨进程长期记忆。
|
|
3. 不在本阶段实现逐字 partial ASR 字幕；本阶段先保证正式问题 STT 完成后立即可见，且出现在 LLM 前。
|
|
4. 不把连续麦克风流上传到云端。
|
|
|
|
## Architecture
|
|
|
|
```text
|
|
run-live
|
|
-> AppConfig(.env)
|
|
-> SoundDeviceAudioTransport
|
|
-> SherpaOnnxKeywordWakeWordProvider(local models/wake)
|
|
-> VadRecorder(user utterance, hybrid local+energy VAD by default)
|
|
-> CloudAsrSttProvider or SherpaOnnxSttProvider
|
|
-> TerminalRuntimeReporter.transcript()
|
|
-> ConversationContext
|
|
-> OpenAICompatibleLlmProvider
|
|
-> CloudTtsProvider or MacSayTtsProvider
|
|
-> speaker playback
|
|
```
|
|
|
|
## Runtime Lifecycle
|
|
|
|
1. Load config.
|
|
2. Validate `OWNER_WAKE_PROVIDER=local_kws`.
|
|
3. Load KWS model from `OWNER_SPEECH_MODELS_DIR`.
|
|
4. Load VAD, STT, LLM, TTS.
|
|
5. Open microphone stream.
|
|
6. Wait for KWS wake event by feeding frames directly into wake provider.
|
|
7. On wake hit, reset wake stream and VAD recorder.
|
|
8. Record user utterance with VAD. The default provider is `hybrid`: project-local `sherpa-onnx` VAD remains the primary detector, and an energy threshold fallback prevents low microphone gain from being treated as no speech.
|
|
9. Transcribe user utterance.
|
|
10. Emit transcript to terminal.
|
|
11. Append user text and call LLM.
|
|
12. Synthesize/play reply.
|
|
13. Append assistant reply and return to standby.
|
|
|
|
## Interfaces
|
|
|
|
### `SherpaOnnxKeywordWakeWordProvider`
|
|
|
|
```python
|
|
class SherpaOnnxKeywordWakeWordProvider:
|
|
def __init__(self, models_dir, keyword, keywords_file=None, threshold=0.25, score=1.0, sherpa_module=None): ...
|
|
def load(self) -> None: ...
|
|
def detect(self, frame: AudioFrame) -> WakeEvent | None: ...
|
|
def reset(self) -> None: ...
|
|
```
|
|
|
|
### `RuntimeReporter`
|
|
|
|
```python
|
|
class RuntimeReporter(Protocol):
|
|
def status(self, state: str, message: str, *, turn_id: int | None = None) -> None: ...
|
|
def transcript(self, text: str, *, final: bool, turn_id: int | None = None) -> None: ...
|
|
def error(self, stage: str, code: str, message: str, *, turn_id: int | None = None) -> None: ...
|
|
```
|
|
|
|
## Model Files
|
|
|
|
```text
|
|
models/
|
|
manifest.json
|
|
wake/
|
|
sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01-mobile/
|
|
tokens.txt
|
|
encoder-epoch-12-avg-2-chunk-16-left-64.int8.onnx
|
|
decoder-epoch-12-avg-2-chunk-16-left-64.onnx
|
|
joiner-epoch-12-avg-2-chunk-16-left-64.int8.onnx
|
|
keywords.txt
|
|
vad/
|
|
silero_vad.onnx
|
|
stt/
|
|
sherpa-onnx-streaming-zipformer-zh-14M-2023-02-23/
|
|
```
|
|
|
|
## Error Handling
|
|
|
|
1. Missing KWS model: `WAKE_MODEL_MISSING`.
|
|
2. KWS load failure: `WAKE_MODEL_LOAD_FAILED`.
|
|
3. KWS runtime failure: `WAKE_MODEL_LOAD_FAILED` with retryable true.
|
|
4. Empty user STT: existing `STT_EMPTY_TRANSCRIPT`.
|
|
5. Invalid wake provider config: `CONFIG_MISSING_VALUE`.
|
|
6. Real microphone VAD miss: default `OWNER_VAD_PROVIDER=hybrid` SHALL accept speech when either the local model or the energy fallback detects speech.
|
|
|
|
## Testing Strategy
|
|
|
|
1. Unit test fake wake provider detects wake without STT calls.
|
|
2. Unit test repeated runtime runs two turns with exactly two STT calls.
|
|
3. Unit test terminal reporter records transcript before LLM stage.
|
|
4. Unit test LLM user content excludes wake keyword.
|
|
5. Unit test KWS provider missing model raises structured error.
|
|
6. Model-check test validates wake required files.
|
|
|
|
## Migration
|
|
|
|
No database migration. Users should run:
|
|
|
|
```bash
|
|
python3.11 scripts/download_speech_models.py --dir models
|
|
.venv/bin/python -m owner_voice_pet model-check --models-dir models
|
|
```
|
|
|
|
Existing `.env` remains valid because new wake keys have defaults.
|