[唤醒灵敏度与端点恢复]:完成唤醒提示顺序和Hybrid VAD优化,包含阈值默认值、缓冲时序和测试覆盖

This commit is contained in:
mkbk
2026-06-17 21:00:28 +08:00
parent 44f708a3dc
commit 95e4996434
13 changed files with 145 additions and 29 deletions
+3 -3
View File
@@ -8,11 +8,11 @@ OWNER_ASSET_DIR=assets/pet
OWNER_LOG_DIR=logs OWNER_LOG_DIR=logs
OWNER_WAKE_PROVIDER=local_kws OWNER_WAKE_PROVIDER=local_kws
OWNER_WAKE_KEYWORDS_FILE= OWNER_WAKE_KEYWORDS_FILE=
OWNER_WAKE_KWS_THRESHOLD=0.25 OWNER_WAKE_KWS_THRESHOLD=0.15
OWNER_WAKE_KWS_SCORE=1.0 OWNER_WAKE_KWS_SCORE=1.0
OWNER_WAKE_ACK_TEXT=我在 OWNER_WAKE_ACK_TEXT=我在
OWNER_POST_PLAYBACK_DRAIN_MS=250 OWNER_POST_PLAYBACK_DRAIN_MS=50
OWNER_VAD_PROVIDER=local OWNER_VAD_PROVIDER=hybrid
OWNER_VAD_THRESHOLD=0.5 OWNER_VAD_THRESHOLD=0.5
OWNER_VAD_MIN_DURATION_MS=250 OWNER_VAD_MIN_DURATION_MS=250
OWNER_VAD_END_SILENCE_MS=350 OWNER_VAD_END_SILENCE_MS=350
+6 -6
View File
@@ -34,11 +34,11 @@ OWNER_LLM_MODEL=mimo-v2.5
OWNER_LLM_API_STYLE=chat_completions OWNER_LLM_API_STYLE=chat_completions
OWNER_WAKE_PROVIDER=local_kws OWNER_WAKE_PROVIDER=local_kws
OWNER_WAKE_KEYWORDS_FILE= OWNER_WAKE_KEYWORDS_FILE=
OWNER_WAKE_KWS_THRESHOLD=0.25 OWNER_WAKE_KWS_THRESHOLD=0.15
OWNER_WAKE_KWS_SCORE=1.0 OWNER_WAKE_KWS_SCORE=1.0
OWNER_WAKE_ACK_TEXT=我在 OWNER_WAKE_ACK_TEXT=我在
OWNER_POST_PLAYBACK_DRAIN_MS=250 OWNER_POST_PLAYBACK_DRAIN_MS=50
OWNER_VAD_PROVIDER=local OWNER_VAD_PROVIDER=hybrid
OWNER_VAD_THRESHOLD=0.5 OWNER_VAD_THRESHOLD=0.5
OWNER_VAD_MIN_DURATION_MS=250 OWNER_VAD_MIN_DURATION_MS=250
OWNER_VAD_END_SILENCE_MS=350 OWNER_VAD_END_SILENCE_MS=350
@@ -63,9 +63,9 @@ python3.11 scripts/download_speech_models.py --dir models
`models/` 已在 `.gitignore` 中,不会提交大模型文件。 `models/` 已在 `.gitignore` 中,不会提交大模型文件。
本地唤醒关键词文件位于 `models/wake/keywords.txt`如果真人唤醒不灵敏,可以先把 `.env` `OWNER_WAKE_KWS_THRESHOLD` 调低,例如 `0.15`再重新运行 本地唤醒关键词文件位于 `models/wake/keywords.txt`默认 `OWNER_WAKE_KWS_THRESHOLD=0.15` 已偏向灵敏;如果真人唤醒仍不灵敏,可以继续降到 `0.10`若误唤醒变多再回调到 `0.20`
如果唤醒后你已经停说但还长时间显示“录音中”,优先调小 `OWNER_VAD_END_SILENCE_MS`,例如 `250`;如果房间噪声较大,再略调高 `OWNER_VAD_THRESHOLD`,例如 `0.6` 默认 `OWNER_VAD_PROVIDER=hybrid` 会同时使用本地 `sherpa-onnx` VAD 和能量阈值兜底,避免本地 VAD 对麦克风音量过保守导致 `VAD_TIMEOUT_NO_SPEECH`如果唤醒后你已经停说但还长时间显示“录音中”,优先调小 `OWNER_VAD_END_SILENCE_MS`,例如 `250`;如果房间噪声较大,再略调高 `OWNER_VAD_THRESHOLD`,例如 `0.6`
## 设备检查 ## 设备检查
@@ -89,7 +89,7 @@ python3.11 scripts/download_speech_models.py --dir models
.venv/bin/python -m owner_voice_pet run-live .venv/bin/python -m owner_voice_pet run-live
``` ```
运行后终端会显示待机、唤醒命中、录音中、转写中、转写结果、思考中、播放中、恢复待机等状态。说“小杰小杰”等待唤醒命中后再提问;听到回复后可以再次说“小杰小杰”继续下一轮。本次进程内会携带临时历史,程序退出后不保存。 运行后终端会显示待机、唤醒命中、应答中、请说出问题、录音中、转写中、转写结果、思考中、播放中、恢复待机等状态。说“小杰小杰”,听到“我在”且看到“请说出问题”后再提问;听到回复后可以再次说“小杰小杰”继续下一轮。本次进程内会携带临时历史,程序退出后不保存。
## 验证 ## 验证
@@ -27,7 +27,7 @@ run-live
-> AppConfig(.env) -> AppConfig(.env)
-> SoundDeviceAudioTransport -> SoundDeviceAudioTransport
-> SherpaOnnxKeywordWakeWordProvider(local models/wake) -> SherpaOnnxKeywordWakeWordProvider(local models/wake)
-> VadRecorder(user utterance) -> VadRecorder(user utterance, hybrid local+energy VAD by default)
-> CloudAsrSttProvider or SherpaOnnxSttProvider -> CloudAsrSttProvider or SherpaOnnxSttProvider
-> TerminalRuntimeReporter.transcript() -> TerminalRuntimeReporter.transcript()
-> ConversationContext -> ConversationContext
@@ -45,7 +45,7 @@ run-live
5. Open microphone stream. 5. Open microphone stream.
6. Wait for KWS wake event by feeding frames directly into wake provider. 6. Wait for KWS wake event by feeding frames directly into wake provider.
7. On wake hit, reset wake stream and VAD recorder. 7. On wake hit, reset wake stream and VAD recorder.
8. Record user utterance with VAD. 8. Record user utterance with VAD. The default provider is `hybrid`: project-local `sherpa-onnx` VAD remains the primary detector, and an energy threshold fallback prevents low microphone gain from being treated as no speech.
9. Transcribe user utterance. 9. Transcribe user utterance.
10. Emit transcript to terminal. 10. Emit transcript to terminal.
11. Append user text and call LLM. 11. Append user text and call LLM.
@@ -98,6 +98,7 @@ models/
3. KWS runtime failure: `WAKE_MODEL_LOAD_FAILED` with retryable true. 3. KWS runtime failure: `WAKE_MODEL_LOAD_FAILED` with retryable true.
4. Empty user STT: existing `STT_EMPTY_TRANSCRIPT`. 4. Empty user STT: existing `STT_EMPTY_TRANSCRIPT`.
5. Invalid wake provider config: `CONFIG_MISSING_VALUE`. 5. Invalid wake provider config: `CONFIG_MISSING_VALUE`.
6. Real microphone VAD miss: default `OWNER_VAD_PROVIDER=hybrid` SHALL accept speech when either the local model or the energy fallback detects speech.
## Testing Strategy ## Testing Strategy
@@ -196,6 +196,8 @@ standby
| --- | --- | --- | --- | | --- | --- | --- | --- |
| KWS 关键词拼音格式不匹配模型 | 中 | 高 | 默认写入 sherpa 示例格式;保留 keywords 文件可编辑;测试下载后用真实模型加载;README 标明关键词文件位置 | | KWS 关键词拼音格式不匹配模型 | 中 | 高 | 默认写入 sherpa 示例格式;保留 keywords 文件可编辑;测试下载后用真实模型加载;README 标明关键词文件位置 |
| KWS 模型误唤醒或漏唤醒 | 中 | 中 | 暴露 threshold/score 配置;保留状态输出;后续可替换 KWS Provider | | KWS 模型误唤醒或漏唤醒 | 中 | 中 | 暴露 threshold/score 配置;保留状态输出;后续可替换 KWS Provider |
| 唤醒应答期间用户抢说被缓冲清理吞掉 | 中 | 高 | 终端提示顺序改为“唤醒命中 -> 应答中 -> 请说出问题 -> 录音中”,用户只在应答完成后收到提问提示;默认播放后排水从 250 ms 降为 50 ms |
| 本地 VAD 对真实麦克风音量过保守 | 中 | 高 | 默认 VAD provider 改为 `hybrid`,本地模型判断和能量阈值兜底任一命中即认为有语音;保留 `local``energy` 可配置回退 |
| 本地 KWS 增加启动加载时间 | 低 | 中 | 模型约 15 MB,启动加载一次;不在每轮重复加载 | | 本地 KWS 增加启动加载时间 | 低 | 中 | 模型约 15 MB,启动加载一次;不在每轮重复加载 |
| 正式问题和唤醒词连在同一句导致问题前半段丢失 | 中 | 中 | 第一版交互明确为先唤醒再提问;后续可做 pre-roll buffer,但不得把 wake 音频直接进 LLM | | 正式问题和唤醒词连在同一句导致问题前半段丢失 | 中 | 中 | 第一版交互明确为先唤醒再提问;后续可做 pre-roll buffer,但不得把 wake 音频直接进 LLM |
| 终端转写不是逐字流式 | 中 | 低 | 第一版至少在 LLM 前即时显示最终 STT 文本;后续可接入本地 streaming STT partial | | 终端转写不是逐字流式 | 中 | 低 | 第一版至少在 LLM 前即时显示最终 STT 文本;后续可接入本地 streaming STT partial |
@@ -232,6 +234,20 @@ standby
- [ ] 4.3 执行 run-live 单轮/两轮验收;前置条件:模型、设备、.env 齐全;验收标准:唤醒命中不等待云 ASR,终端显示转写结果;优先级:P0;预计:60 分钟。 - [ ] 4.3 执行 run-live 单轮/两轮验收;前置条件:模型、设备、.env 齐全;验收标准:唤醒命中不等待云 ASR,终端显示转写结果;优先级:P0;预计:60 分钟。
- [ ] 4.4 最终门禁、归档、提交;前置条件:全部任务完成;验收标准:compileall、unittest、security-check、model-check、device-check、OpenSpec strict、工作树干净;优先级:P0;预计:45 分钟。 - [ ] 4.4 最终门禁、归档、提交;前置条件:全部任务完成;验收标准:compileall、unittest、security-check、model-check、device-check、OpenSpec strict、工作树干净;优先级:P0;预计:45 分钟。
### 5. 唤醒应答与快速端点修正
- [ ] 5.1 增加唤醒后本地语音应答;前置条件:本地 KWS 已可唤醒;验收标准:wake 命中后播放“我在”再进入录音;测试要点:fake runtime 播放顺序;优先级:P0;预计:45 分钟。
- [ ] 5.2 清理应答播放期间的麦克风缓冲;前置条件:5.1 完成;验收标准:应答音频不进入正式问题 VAD/STT;测试要点:transport flush 测试;优先级:P0;预计:45 分钟。
- [ ] 5.3 live 默认改用本地 `sherpa-onnx` VAD 并缩短静音端点;前置条件:模型已下载;验收标准:`.env` 可配置 VAD provider、静音结束时间和最大录音时长;测试要点:配置和 runtime 构造测试;优先级:P0;预计:45 分钟。
- [ ] 5.4 验证并提交“唤醒应答与快速端点”模块;前置条件:5.1 至 5.3 完成;验收标准:compileall、unittest、security-check、model-check、OpenSpec strict 通过后 commit;优先级:P0;预计:20 分钟。
### 6. 灵敏度与录音端点恢复
- [ ] 6.1 调整唤醒提示顺序;前置条件:真人验收暴露用户会在“我在”播放前抢说;验收标准:终端顺序为“唤醒命中 -> 应答中:我在 -> 请说出问题 -> 录音中:正在听取问题”;测试要点:runtime reporter 顺序断言;优先级:P0;预计:30 分钟。
- [ ] 6.2 降低默认 KWS 阈值;前置条件:真人反馈唤醒难触发;验收标准:默认 `OWNER_WAKE_KWS_THRESHOLD=0.15`README 说明 0.10 至 0.20 调参范围;测试要点:配置默认值测试;优先级:P0;预计:20 分钟。
- [ ] 6.3 增加 `hybrid` VAD;前置条件:真人验收出现 `VAD_TIMEOUT_NO_SPEECH`;验收标准:本地 VAD 和能量阈值任一判断为语音即可开始录音,默认 `OWNER_VAD_PROVIDER=hybrid`;测试要点:能量兜底单测;优先级:P0;预计:45 分钟。
- [ ] 6.4 验证并提交“唤醒灵敏度与端点恢复”模块;前置条件:6.1 至 6.3 完成;验收标准:compileall、unittest、security-check、model-check、device-check、OpenSpec strict 通过后 commit;优先级:P0;预计:30 分钟。
## Spec Deltas ## Spec Deltas
### 新增能力 ### 新增能力
@@ -244,6 +260,8 @@ standby
2. `Live terminal state reporting`:新增终端转写结果输出要求。 2. `Live terminal state reporting`:新增终端转写结果输出要求。
3. `Local speech model management`:新增 wake KWS 模型和关键词文件管理。 3. `Local speech model management`:新增 wake KWS 模型和关键词文件管理。
4. `Testability`:新增 wake/STT 分离测试和唤醒污染回归测试。 4. `Testability`:新增 wake/STT 分离测试和唤醒污染回归测试。
5. `Wake acknowledgement before recording`:明确“请说出问题”必须在应答播放完成后输出,避免用户抢说被清缓冲。
6. `Fast user utterance endpointing`:默认 VAD provider 改为 `hybrid`,用能量阈值兜底真实麦克风音量差异。
### 删除项 ### 删除项
@@ -259,6 +277,7 @@ standby
2. M2KWS 模型下载、manifest、config、model-check 完成并提交。 2. M2KWS 模型下载、manifest、config、model-check 完成并提交。
3. M3Runtime 独立 wake provider 和转写输出完成并提交。 3. M3Runtime 独立 wake provider 和转写输出完成并提交。
4. M4README、真实验收、archive 和最终提交完成。 4. M4README、真实验收、archive 和最终提交完成。
5. M5:真人验收反馈修正唤醒提示顺序、KWS 阈值和 hybrid VAD,并在门禁通过后提交。
估时: 估时:
@@ -23,7 +23,7 @@ The live runtime SHALL provide an audible local acknowledgement after local wake
- **THEN** the runtime SHALL clear buffered microphone input captured during acknowledgement playback before starting VAD recording for the user's question - **THEN** the runtime SHALL clear buffered microphone input captured during acknowledgement playback before starting VAD recording for the user's question
### Requirement: Fast user utterance endpointing ### Requirement: Fast user utterance endpointing
The live runtime SHALL use the project-local VAD model by default for user utterance endpoint detection and SHALL expose configurable silence timing so the recording stops promptly after the user stops speaking. The live runtime SHALL use project-local VAD capability with an energy-threshold fallback by default for user utterance endpoint detection and SHALL expose configurable silence timing so the recording stops promptly after the user stops speaking.
#### Scenario: User stops speaking after wake #### Scenario: User stops speaking after wake
- **WHEN** VAD observes the configured continuous silence duration after a started utterance - **WHEN** VAD observes the configured continuous silence duration after a started utterance
@@ -33,6 +33,14 @@ The live runtime SHALL use the project-local VAD model by default for user utter
- **WHEN** `OWNER_VAD_PROVIDER=local` - **WHEN** `OWNER_VAD_PROVIDER=local`
- **THEN** the runtime SHALL use the project-local `sherpa-onnx` VAD model rather than a raw energy threshold for live user utterance endpointing - **THEN** the runtime SHALL use the project-local `sherpa-onnx` VAD model rather than a raw energy threshold for live user utterance endpointing
#### Scenario: Hybrid VAD is configured
- **WHEN** `OWNER_VAD_PROVIDER=hybrid`
- **THEN** the runtime SHALL load both the project-local VAD model and an energy-threshold fallback and SHALL treat a frame as speech when either detector reports speech
#### Scenario: Wake acknowledgement prompt ordering
- **WHEN** local wake detection succeeds and `OWNER_WAKE_ACK_TEXT` is non-empty
- **THEN** the terminal SHALL emit wake hit, acknowledgement, question prompt, and recording states in that order so the user is cued to speak only after acknowledgement playback
## MODIFIED Requirements ## MODIFIED Requirements
### Requirement: Wake word detection ### Requirement: Wake word detection
@@ -38,3 +38,10 @@
- [x] 5.2 清理应答播放期间的麦克风缓冲;前置条件:5.1 完成;验收标准:应答音频不进入正式问题 VAD/STT;测试要点:transport flush 测试;优先级:P0;预计:45 分钟。 - [x] 5.2 清理应答播放期间的麦克风缓冲;前置条件:5.1 完成;验收标准:应答音频不进入正式问题 VAD/STT;测试要点:transport flush 测试;优先级:P0;预计:45 分钟。
- [x] 5.3 live 默认改用本地 `sherpa-onnx` VAD 并缩短静音端点;前置条件:模型已下载;验收标准:`.env` 可配置 VAD provider、静音结束时间和最大录音时长;测试要点:配置和 runtime 构造测试;优先级:P0;预计:45 分钟。 - [x] 5.3 live 默认改用本地 `sherpa-onnx` VAD 并缩短静音端点;前置条件:模型已下载;验收标准:`.env` 可配置 VAD provider、静音结束时间和最大录音时长;测试要点:配置和 runtime 构造测试;优先级:P0;预计:45 分钟。
- [x] 5.4 验证并提交“唤醒应答与快速端点”模块;前置条件:5.1 至 5.3 完成;验收标准:compileall、unittest、security-check、model-check、OpenSpec strict 通过后 commit;优先级:P0;预计:20 分钟。 - [x] 5.4 验证并提交“唤醒应答与快速端点”模块;前置条件:5.1 至 5.3 完成;验收标准:compileall、unittest、security-check、model-check、OpenSpec strict 通过后 commit;优先级:P0;预计:20 分钟。
## 6. 灵敏度与录音端点恢复
- [x] 6.1 调整唤醒提示顺序;前置条件:真人验收显示用户可能在“我在”播放前抢说;验收标准:终端顺序为“唤醒命中 -> 应答中:我在 -> 请说出问题 -> 录音中:正在听取问题”;测试要点:runtime reporter 顺序断言;优先级:P0;预计:30 分钟。
- [x] 6.2 降低默认 KWS 阈值;前置条件:真人反馈“小杰小杰”难触发;验收标准:默认 `OWNER_WAKE_KWS_THRESHOLD=0.15`README 给出 `0.10``0.20` 调参建议;测试要点:配置默认值测试;优先级:P0;预计:20 分钟。
- [x] 6.3 增加 `hybrid` VAD 并设为默认;前置条件:真人验收出现 `VAD_TIMEOUT_NO_SPEECH`;验收标准:本地 VAD 和能量阈值任一判断为语音即可开始/继续录音,默认 `OWNER_VAD_PROVIDER=hybrid`;测试要点:能量兜底单测、配置校验测试;优先级:P0;预计:45 分钟。
- [x] 6.4 验证并提交“唤醒灵敏度与端点恢复”模块;前置条件:6.1 至 6.3 完成;验收标准:compileall、unittest、security-check、model-check、device-check、OpenSpec strict 通过后 commit;优先级:P0;预计:30 分钟。
+2 -1
View File
@@ -17,7 +17,7 @@ from .models import (
) )
from .transport import AudioRingBuffer, FileReplayTransport, MemoryAudioTransport from .transport import AudioRingBuffer, FileReplayTransport, MemoryAudioTransport
from .wakeword import KeywordWakeWordProvider, SherpaOnnxKeywordWakeWordProvider from .wakeword import KeywordWakeWordProvider, SherpaOnnxKeywordWakeWordProvider
from .vad import EnergyVadProvider, VadRecorder from .vad import EnergyVadProvider, HybridVadProvider, VadRecorder
from .stt import CloudAsrSttProvider, MetadataSttProvider, SherpaOnnxSttProvider, is_valid_transcript_text from .stt import CloudAsrSttProvider, MetadataSttProvider, SherpaOnnxSttProvider, is_valid_transcript_text
from .conversation import ConversationContext from .conversation import ConversationContext
from .llm import MockLlmProvider, OpenAICompatibleLlmProvider from .llm import MockLlmProvider, OpenAICompatibleLlmProvider
@@ -38,6 +38,7 @@ __all__ = [
"KeywordWakeWordProvider", "KeywordWakeWordProvider",
"SherpaOnnxKeywordWakeWordProvider", "SherpaOnnxKeywordWakeWordProvider",
"EnergyVadProvider", "EnergyVadProvider",
"HybridVadProvider",
"VadRecorder", "VadRecorder",
"CloudAsrSttProvider", "CloudAsrSttProvider",
"MetadataSttProvider", "MetadataSttProvider",
+8 -8
View File
@@ -22,11 +22,11 @@ class AppConfig:
log_dir: Path = Path("logs") log_dir: Path = Path("logs")
wake_provider: str = "local_kws" wake_provider: str = "local_kws"
wake_keywords_file: Path | None = None wake_keywords_file: Path | None = None
wake_kws_threshold: float = 0.25 wake_kws_threshold: float = 0.15
wake_kws_score: float = 1.0 wake_kws_score: float = 1.0
wake_ack_text: str = "我在" wake_ack_text: str = "我在"
post_playback_drain_ms: int = 250 post_playback_drain_ms: int = 50
vad_provider: str = "local" vad_provider: str = "hybrid"
vad_threshold: float = 0.5 vad_threshold: float = 0.5
vad_min_duration_ms: int = 250 vad_min_duration_ms: int = 250
vad_end_silence_ms: int = 350 vad_end_silence_ms: int = 350
@@ -63,11 +63,11 @@ class AppConfig:
log_dir=Path(get("LOG_DIR", "logs") or "logs"), log_dir=Path(get("LOG_DIR", "logs") or "logs"),
wake_provider=(get("WAKE_PROVIDER", "local_kws") or "local_kws").lower(), wake_provider=(get("WAKE_PROVIDER", "local_kws") or "local_kws").lower(),
wake_keywords_file=Path(value) if (value := get("WAKE_KEYWORDS_FILE")) else None, wake_keywords_file=Path(value) if (value := get("WAKE_KEYWORDS_FILE")) else None,
wake_kws_threshold=float(get("WAKE_KWS_THRESHOLD", "0.25") or "0.25"), wake_kws_threshold=float(get("WAKE_KWS_THRESHOLD", "0.15") or "0.15"),
wake_kws_score=float(get("WAKE_KWS_SCORE", "1.0") or "1.0"), wake_kws_score=float(get("WAKE_KWS_SCORE", "1.0") or "1.0"),
wake_ack_text=get("WAKE_ACK_TEXT", "我在") or "我在", wake_ack_text=get("WAKE_ACK_TEXT", "我在") or "我在",
post_playback_drain_ms=int(get("POST_PLAYBACK_DRAIN_MS", "250") or "250"), post_playback_drain_ms=int(get("POST_PLAYBACK_DRAIN_MS", "50") or "50"),
vad_provider=(get("VAD_PROVIDER", "local") or "local").lower(), vad_provider=(get("VAD_PROVIDER", "hybrid") or "hybrid").lower(),
vad_threshold=float(get("VAD_THRESHOLD", "0.5") or "0.5"), vad_threshold=float(get("VAD_THRESHOLD", "0.5") or "0.5"),
vad_min_duration_ms=int(get("VAD_MIN_DURATION_MS", "250") or "250"), vad_min_duration_ms=int(get("VAD_MIN_DURATION_MS", "250") or "250"),
vad_end_silence_ms=int(get("VAD_END_SILENCE_MS", "350") or "350"), vad_end_silence_ms=int(get("VAD_END_SILENCE_MS", "350") or "350"),
@@ -178,11 +178,11 @@ class AppConfig:
"startup", "startup",
) )
) )
if self.vad_provider not in {"local", "energy"}: if self.vad_provider not in {"hybrid", "local", "energy"}:
errors.append( errors.append(
ProviderError( ProviderError(
ErrorCode.CONFIG_MISSING_VALUE, ErrorCode.CONFIG_MISSING_VALUE,
"OWNER_VAD_PROVIDER must be local or energy", "OWNER_VAD_PROVIDER must be hybrid, local, or energy",
False, False,
"config", "config",
"startup", "startup",
+9 -3
View File
@@ -12,7 +12,7 @@ from .protocols import AudioTransport, LlmProvider, SttProvider, TtsProvider, Wa
from .stt import CloudAsrSttProvider, SherpaOnnxSttProvider, is_valid_transcript_text from .stt import CloudAsrSttProvider, SherpaOnnxSttProvider, is_valid_transcript_text
from .transport import SoundDeviceAudioTransport from .transport import SoundDeviceAudioTransport
from .tts import CloudTtsProvider, MacSayTtsProvider, SentenceBuffer from .tts import CloudTtsProvider, MacSayTtsProvider, SentenceBuffer
from .vad import EnergyVadProvider, SherpaOnnxVadProvider, VadRecorder from .vad import EnergyVadProvider, HybridVadProvider, SherpaOnnxVadProvider, VadRecorder
from .wakeword import SherpaOnnxKeywordWakeWordProvider from .wakeword import SherpaOnnxKeywordWakeWordProvider
@@ -145,10 +145,11 @@ class LiveVoiceRuntime:
wake_error = self._wait_for_local_wake(turn_id) wake_error = self._wait_for_local_wake(turn_id)
if wake_error is not None: if wake_error is not None:
return wake_error return wake_error
self._state(PipelineState.SPEECH_DETECTING, "唤醒命中:请说出问题", turn_id=turn_id) self._state(PipelineState.SPEECH_DETECTING, "唤醒命中", turn_id=turn_id)
ack_error = self._acknowledge_wake(turn_id) ack_error = self._acknowledge_wake(turn_id)
if ack_error is not None: if ack_error is not None:
return ack_error return ack_error
self._state(PipelineState.SPEECH_DETECTING, "请说出问题", turn_id=turn_id)
user_segment = self._capture_segment(turn_id, state_message="录音中:正在听取问题") user_segment = self._capture_segment(turn_id, state_message="录音中:正在听取问题")
if isinstance(user_segment, ProviderError): if isinstance(user_segment, ProviderError):
return user_segment return user_segment
@@ -277,7 +278,12 @@ def build_live_runtime(config: AppConfig, reporter: RuntimeReporter | None = Non
else: else:
stt = SherpaOnnxSttProvider(str(config.speech_models_dir)) stt = SherpaOnnxSttProvider(str(config.speech_models_dir))
tts = MacSayTtsProvider() tts = MacSayTtsProvider()
if config.vad_provider == "local": if config.vad_provider == "hybrid":
vad_provider = HybridVadProvider(
SherpaOnnxVadProvider(config.speech_models_dir, threshold=config.vad_threshold),
EnergyVadProvider(),
)
elif config.vad_provider == "local":
vad_provider = SherpaOnnxVadProvider(config.speech_models_dir, threshold=config.vad_threshold) vad_provider = SherpaOnnxVadProvider(config.speech_models_dir, threshold=config.vad_threshold)
else: else:
vad_provider = EnergyVadProvider() vad_provider = EnergyVadProvider()
+47
View File
@@ -157,6 +157,53 @@ class SherpaOnnxVadProvider:
self._model.reset() self._model.reset()
class HybridVadProvider:
"""Combine local model VAD with energy fallback for live microphone variance."""
def __init__(self, primary: Any, fallback: Any) -> None:
self.primary = primary
self.fallback = fallback
self.loaded = False
self._speech_ms = 0
self._silence_ms = 0
def load(self) -> None:
self.primary.load()
self.fallback.load()
self.loaded = True
def analyze(self, frame: AudioFrame) -> VadResult:
if not self.loaded:
raise ProviderError(
ErrorCode.VAD_MODEL_LOAD_FAILED,
"hybrid VAD provider is not loaded",
False,
"hybrid-vad",
"vad",
)
primary = self.primary.analyze(frame)
fallback = self.fallback.analyze(frame)
is_speech = primary.is_speech or fallback.is_speech
frame_ms = int(frame.metadata.get("duration_ms", 20))
if is_speech:
self._speech_ms += frame_ms
self._silence_ms = 0
else:
self._silence_ms += frame_ms
return VadResult(
is_speech=is_speech,
confidence=max(primary.confidence, fallback.confidence),
speech_ms=self._speech_ms,
silence_ms=self._silence_ms,
)
def reset(self) -> None:
self._speech_ms = 0
self._silence_ms = 0
self.primary.reset()
self.fallback.reset()
@dataclass(slots=True) @dataclass(slots=True)
class VadRecorder: class VadRecorder:
provider: Any provider: Any
+3
View File
@@ -104,6 +104,9 @@ class LiveRuntimeTests(unittest.TestCase):
self.assertEqual(len(transport.played_segments), 4) self.assertEqual(len(transport.played_segments), 4)
self.assertEqual(reporter.transcripts, ["第一问", "第二问"]) self.assertEqual(reporter.transcripts, ["第一问", "第二问"])
self.assertIn("应答中:我在", reporter.statuses) self.assertIn("应答中:我在", reporter.statuses)
self.assertLess(reporter.statuses.index("唤醒命中"), reporter.statuses.index("应答中:我在"))
self.assertLess(reporter.statuses.index("应答中:我在"), reporter.statuses.index("请说出问题"))
self.assertLess(reporter.statuses.index("请说出问题"), reporter.statuses.index("录音中:正在听取问题"))
self.assertIn("恢复待机:可继续唤醒", reporter.statuses[-1]) self.assertIn("恢复待机:可继续唤醒", reporter.statuses[-1])
def test_temporary_context_is_sent_to_second_llm_call(self) -> None: def test_temporary_context_is_sent_to_second_llm_call(self) -> None:
+4 -4
View File
@@ -70,11 +70,11 @@ class ModelsConfigTests(unittest.TestCase):
self.assertEqual(config.llm_api_key, "secret-value") self.assertEqual(config.llm_api_key, "secret-value")
self.assertEqual(config.llm_model, "test-model") self.assertEqual(config.llm_model, "test-model")
self.assertEqual(config.wake_provider, "local_kws") self.assertEqual(config.wake_provider, "local_kws")
self.assertEqual(config.wake_kws_threshold, 0.25) self.assertEqual(config.wake_kws_threshold, 0.15)
self.assertEqual(config.wake_kws_score, 1.0) self.assertEqual(config.wake_kws_score, 1.0)
self.assertEqual(config.wake_ack_text, "我在") self.assertEqual(config.wake_ack_text, "我在")
self.assertEqual(config.post_playback_drain_ms, 250) self.assertEqual(config.post_playback_drain_ms, 50)
self.assertEqual(config.vad_provider, "local") self.assertEqual(config.vad_provider, "hybrid")
self.assertEqual(config.vad_threshold, 0.5) self.assertEqual(config.vad_threshold, 0.5)
self.assertEqual(config.vad_min_duration_ms, 250) self.assertEqual(config.vad_min_duration_ms, 250)
self.assertEqual(config.vad_end_silence_ms, 350) self.assertEqual(config.vad_end_silence_ms, 350)
@@ -98,7 +98,7 @@ class ModelsConfigTests(unittest.TestCase):
errors = config.validate_basic() errors = config.validate_basic()
self.assertTrue(any("OWNER_WAKE_PROVIDER" in error.message for error in errors)) self.assertTrue(any("OWNER_WAKE_PROVIDER" in error.message for error in errors))
def test_vad_provider_must_be_local_or_energy(self) -> None: def test_vad_provider_must_be_hybrid_local_or_energy(self) -> None:
config = AppConfig(vad_provider="invalid") config = AppConfig(vad_provider="invalid")
errors = config.validate_basic() errors = config.validate_basic()
self.assertTrue(any("OWNER_VAD_PROVIDER" in error.message for error in errors)) self.assertTrue(any("OWNER_VAD_PROVIDER" in error.message for error in errors))
+25 -1
View File
@@ -7,7 +7,7 @@ import unittest
from owner_voice_pet.models import AudioFrame, AudioSegment, ErrorCode, ProviderError from owner_voice_pet.models import AudioFrame, AudioSegment, ErrorCode, ProviderError
from owner_voice_pet.config import AppConfig from owner_voice_pet.config import AppConfig
from owner_voice_pet.stt import CloudAsrSttProvider, MetadataSttProvider, SherpaOnnxSttProvider, is_valid_transcript_text from owner_voice_pet.stt import CloudAsrSttProvider, MetadataSttProvider, SherpaOnnxSttProvider, is_valid_transcript_text
from owner_voice_pet.vad import EnergyVadProvider, VadRecorder from owner_voice_pet.vad import EnergyVadProvider, HybridVadProvider, VadRecorder
from owner_voice_pet.wakeword import ( from owner_voice_pet.wakeword import (
KeywordWakeWordProvider, KeywordWakeWordProvider,
MissingWakeWordModelProvider, MissingWakeWordModelProvider,
@@ -87,6 +87,30 @@ class WakeVadSttTests(unittest.TestCase):
self.assertIsInstance(result, ProviderError) self.assertIsInstance(result, ProviderError)
self.assertEqual(result.code, ErrorCode.VAD_TIMEOUT_NO_SPEECH) self.assertEqual(result.code, ErrorCode.VAD_TIMEOUT_NO_SPEECH)
def test_hybrid_vad_accepts_energy_fallback_speech(self) -> None:
class SilentVadProvider:
def __init__(self) -> None:
self.loaded = False
def load(self) -> None:
self.loaded = True
def analyze(self, frame: AudioFrame):
return type(
"Result",
(),
{"is_speech": False, "confidence": 0.1, "speech_ms": 0, "silence_ms": 20},
)()
def reset(self) -> None:
return None
provider = HybridVadProvider(SilentVadProvider(), EnergyVadProvider())
provider.load()
result = provider.analyze(make_frame(1, 0, speech=True))
self.assertTrue(result.is_speech)
self.assertEqual(result.speech_ms, 20)
def test_metadata_stt_transcribes_fixture_text(self) -> None: def test_metadata_stt_transcribes_fixture_text(self) -> None:
provider = MetadataSttProvider() provider = MetadataSttProvider()
provider.load() provider.load()