[OpenSpec 修正]:完成本地唤醒与转写显示规划,包含KWS模型、终端转写和任务分解
This commit is contained in:
@@ -0,0 +1,120 @@
|
||||
# 独立本地唤醒与终端转写显示设计
|
||||
|
||||
## Context
|
||||
|
||||
真实运行日志显示,当前 `run-live` 会先截取语音段并使用 STT 判断是否包含“小杰小杰”。该实现把 wake detection 与 user utterance transcription 绑定在一起,导致唤醒慢、唤醒词污染正式问题、终端输出难以区分 STT 和 LLM 阶段。本设计把 wake word detection 升级为本地 KWS 模型路径,并把正式问题转写作为独立可见事件输出。
|
||||
|
||||
## Goals
|
||||
|
||||
1. 唤醒词“小杰小杰”由本地模型检测。
|
||||
2. wake 阶段不调用云 ASR,也不调用正式 STT provider。
|
||||
3. wake 命中后才开始正式问题录音和 STT。
|
||||
4. 终端在 LLM 前显示正式问题转写文本。
|
||||
5. 模型下载和检查覆盖 wake KWS、VAD、STT。
|
||||
6. 自动化测试证明 wake/STT 分离、重复对话、上下文和错误恢复仍正常。
|
||||
|
||||
## Non-Goals
|
||||
|
||||
1. 不实现 GUI 桌宠窗口。
|
||||
2. 不实现跨进程长期记忆。
|
||||
3. 不在本阶段实现逐字 partial ASR 字幕;本阶段先保证正式问题 STT 完成后立即可见,且出现在 LLM 前。
|
||||
4. 不把连续麦克风流上传到云端。
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
run-live
|
||||
-> AppConfig(.env)
|
||||
-> SoundDeviceAudioTransport
|
||||
-> SherpaOnnxKeywordWakeWordProvider(local models/wake)
|
||||
-> VadRecorder(user utterance)
|
||||
-> CloudAsrSttProvider or SherpaOnnxSttProvider
|
||||
-> TerminalRuntimeReporter.transcript()
|
||||
-> ConversationContext
|
||||
-> OpenAICompatibleLlmProvider
|
||||
-> CloudTtsProvider or MacSayTtsProvider
|
||||
-> speaker playback
|
||||
```
|
||||
|
||||
## Runtime Lifecycle
|
||||
|
||||
1. Load config.
|
||||
2. Validate `OWNER_WAKE_PROVIDER=local_kws`.
|
||||
3. Load KWS model from `OWNER_SPEECH_MODELS_DIR`.
|
||||
4. Load VAD, STT, LLM, TTS.
|
||||
5. Open microphone stream.
|
||||
6. Wait for KWS wake event by feeding frames directly into wake provider.
|
||||
7. On wake hit, reset wake stream and VAD recorder.
|
||||
8. Record user utterance with VAD.
|
||||
9. Transcribe user utterance.
|
||||
10. Emit transcript to terminal.
|
||||
11. Append user text and call LLM.
|
||||
12. Synthesize/play reply.
|
||||
13. Append assistant reply and return to standby.
|
||||
|
||||
## Interfaces
|
||||
|
||||
### `SherpaOnnxKeywordWakeWordProvider`
|
||||
|
||||
```python
|
||||
class SherpaOnnxKeywordWakeWordProvider:
|
||||
def __init__(self, models_dir, keyword, keywords_file=None, threshold=0.25, score=1.0, sherpa_module=None): ...
|
||||
def load(self) -> None: ...
|
||||
def detect(self, frame: AudioFrame) -> WakeEvent | None: ...
|
||||
def reset(self) -> None: ...
|
||||
```
|
||||
|
||||
### `RuntimeReporter`
|
||||
|
||||
```python
|
||||
class RuntimeReporter(Protocol):
|
||||
def status(self, state: str, message: str, *, turn_id: int | None = None) -> None: ...
|
||||
def transcript(self, text: str, *, final: bool, turn_id: int | None = None) -> None: ...
|
||||
def error(self, stage: str, code: str, message: str, *, turn_id: int | None = None) -> None: ...
|
||||
```
|
||||
|
||||
## Model Files
|
||||
|
||||
```text
|
||||
models/
|
||||
manifest.json
|
||||
wake/
|
||||
sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01-mobile/
|
||||
tokens.txt
|
||||
encoder-epoch-12-avg-2-chunk-16-left-64.int8.onnx
|
||||
decoder-epoch-12-avg-2-chunk-16-left-64.onnx
|
||||
joiner-epoch-12-avg-2-chunk-16-left-64.int8.onnx
|
||||
keywords.txt
|
||||
vad/
|
||||
silero_vad.onnx
|
||||
stt/
|
||||
sherpa-onnx-streaming-zipformer-zh-14M-2023-02-23/
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
1. Missing KWS model: `WAKE_MODEL_MISSING`.
|
||||
2. KWS load failure: `WAKE_MODEL_LOAD_FAILED`.
|
||||
3. KWS runtime failure: `WAKE_MODEL_LOAD_FAILED` with retryable true.
|
||||
4. Empty user STT: existing `STT_EMPTY_TRANSCRIPT`.
|
||||
5. Invalid wake provider config: `CONFIG_MISSING_VALUE`.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
1. Unit test fake wake provider detects wake without STT calls.
|
||||
2. Unit test repeated runtime runs two turns with exactly two STT calls.
|
||||
3. Unit test terminal reporter records transcript before LLM stage.
|
||||
4. Unit test LLM user content excludes wake keyword.
|
||||
5. Unit test KWS provider missing model raises structured error.
|
||||
6. Model-check test validates wake required files.
|
||||
|
||||
## Migration
|
||||
|
||||
No database migration. Users should run:
|
||||
|
||||
```bash
|
||||
python3.11 scripts/download_speech_models.py --dir models
|
||||
.venv/bin/python -m owner_voice_pet model-check --models-dir models
|
||||
```
|
||||
|
||||
Existing `.env` remains valid because new wake keys have defaults.
|
||||
Reference in New Issue
Block a user