Compare commits

...

10 Commits

Author SHA1 Message Date
mkbk ac72738fc8 [持续对话判定]:完成恢复待机延迟修复,包含本地规则收敛和分类器跳过回归测试 2026-06-18 13:51:35 +08:00
mkbk 86a429f018 [唤醒应答加速]:完成ACK音频预热缓存,包含唤醒后复用播放和回归测试 2026-06-18 13:44:44 +08:00
mkbk 3408a30e25 [播报文本净化]:完成TTS表情包过滤,包含emoji清理、上下文净化和回归测试 2026-06-18 12:45:31 +08:00
mkbk 25255f178e [回声字幕修复]:完成播报阶段实时字幕隔离,包含静默打断判定和回归测试 2026-06-18 12:32:03 +08:00
mkbk d7dbec5c29 [播放平滑修复]:完成可打断播报平滑播放,包含单输出流分块播放和TTS采样率保真 2026-06-18 12:21:01 +08:00
mkbk 7519725321 [连续对话判断]:完成自动持续对话和播报打断,包含回复意图判断、免唤醒追问和打断回归测试 2026-06-18 12:08:41 +08:00
mkbk 1e956f5eb6 [实时字幕端点]:完成无新文字快速结束录音,包含1.5秒停滞配置、Capture端点和回归测试 2026-06-18 11:23:20 +08:00
mkbk 39da9dd09f [真实流程计时]:完成完整链路耗时输出,包含命令时间、LLM发送时间和阶段响应耗时 2026-06-18 00:45:10 +08:00
mkbk b9b501cf74 [真实流程命令]:完成完整链路自测入口,包含真实Provider CLI、播放验收和文档测试 2026-06-17 23:57:24 +08:00
mkbk b2822dc1af [真实链路自测]:完成完整流程自测和ACK缓冲修正,包含真实Provider验收、播放回声清理和回归测试 2026-06-17 23:47:21 +08:00
28 changed files with 2587 additions and 133 deletions
+8
View File
@@ -3,6 +3,7 @@ OWNER_LLM_API_KEY=
OWNER_LLM_MODEL=mimo-v2.5
OWNER_LLM_API_STYLE=chat_completions
OWNER_REALTIME_TRANSCRIPT_ENABLED=1
OWNER_REALTIME_TRANSCRIPT_IDLE_TIMEOUT_MS=1500
OWNER_AUDIO_INPUT_DEVICE=
OWNER_AUDIO_OUTPUT_DEVICE=
OWNER_ASSET_DIR=assets/pet
@@ -37,6 +38,13 @@ OWNER_SPEECH_MODELS_DIR=models
OWNER_CONTEXT_MODE=session_memory
OWNER_CONTEXT_MAX_MESSAGES=12
OWNER_CONTEXT_MAX_CHARS=12000
OWNER_CONTINUOUS_DIALOG_ENABLED=1
OWNER_CONTINUATION_DECISION_PROVIDER=hybrid
OWNER_CONTINUATION_CONFIDENCE_THRESHOLD=0.65
OWNER_FOLLOWUP_LISTEN_TIMEOUT_MS=3000
OWNER_BARGE_IN_ENABLED=1
OWNER_BARGE_IN_MIN_SPEECH_MS=250
OWNER_BARGE_IN_ECHO_GUARD_MS=500
OWNER_WAKE_WORD=小杰小杰
OWNER_SAMPLE_RATE=16000
OWNER_CHANNELS=1
+28 -3
View File
@@ -16,6 +16,8 @@
- 本地模型:`models/` 存放 `sherpa-onnx` wake/VAD/STT/denoiser 模型,目录不提交 Git。
- Pipeline`run-live` 使用 stage 化 `VoiceAssistantPipeline`,通过事件总线输出终端状态。
- 主说话人端点:默认 `OWNER_ENDPOINT_MODE=primary_speaker`,本轮音色消失后结束录音,避免背景噪声拖慢 STT。
- 自动持续对话:助手回复后自动判断是否需要继续听用户回答,默认规则优先、LLM 小分类器兜底,不确定就恢复待机。
- 播报打断:助手播报超过回声保护期后,如果检测到有效用户语音和实时字幕,会停止剩余播报并处理新输入。
- 临时上下文:同一次 `run-live` 进程内携带最近 user/assistant 历史,退出即清空。
## 首次准备
@@ -35,6 +37,7 @@ OWNER_LLM_API_KEY=
OWNER_LLM_MODEL=mimo-v2.5
OWNER_LLM_API_STYLE=chat_completions
OWNER_REALTIME_TRANSCRIPT_ENABLED=1
OWNER_REALTIME_TRANSCRIPT_IDLE_TIMEOUT_MS=1500
OWNER_WAKE_PROVIDER=local_kws
OWNER_WAKE_KEYWORDS_FILE=
OWNER_WAKE_KWS_THRESHOLD=0.15
@@ -62,13 +65,24 @@ OWNER_ASR_MODEL=mimo-v2.5-asr
OWNER_TTS_MODEL=mimo-v2.5-tts
OWNER_TTS_VOICE=mimo_default
OWNER_CONTEXT_MODE=session_memory
OWNER_CONTINUOUS_DIALOG_ENABLED=1
OWNER_CONTINUATION_DECISION_PROVIDER=hybrid
OWNER_CONTINUATION_CONFIDENCE_THRESHOLD=0.65
OWNER_FOLLOWUP_LISTEN_TIMEOUT_MS=3000
OWNER_BARGE_IN_ENABLED=1
OWNER_BARGE_IN_MIN_SPEECH_MS=250
OWNER_BARGE_IN_ECHO_GUARD_MS=500
```
`OWNER_WAKE_PROVIDER=local_kws` 表示唤醒词“小杰小杰”由本地模型检测。唤醒命中后会先本地播报 `OWNER_WAKE_ACK_TEXT=我在`,再开始听取问题。`OWNER_SPEECH_PROVIDER=local` 表示正式问题 STT、实时字幕和 TTS 都走本地模型或 macOS 本地能力;云端只接收 final 文本和本次会话历史用于 LLM 回复。
`OWNER_NOISE_FILTER_ENABLED=1` 表示唤醒后的正式问题阶段默认启用本地 GTCRN 降噪。降噪后的同一份音频会进入 VAD、实时字幕和 final STT;wake 阶段默认保持原始音频,`OWNER_WAKE_DENOISE_ENABLED=0` 可以避免降噪影响“小杰小杰”的 KWS 特征。
`OWNER_REALTIME_TRANSCRIPT_ENABLED=1` 表示录音期间会使用本地 streaming STT 实时显示中间转写,终端会输出 `实时转写:...`。为了过滤噪音,实时字幕默认不会显示 `家``家确` 这类很短的瞬时误识别;最终发送给 LLM 的内容仍只以 `转写结果:...` 为准。设置为 `0` 可以临时关闭实时显示。
`OWNER_REALTIME_TRANSCRIPT_ENABLED=1` 表示录音期间会使用本地 streaming STT 实时显示中间转写,终端会输出 `实时转写:...`。为了过滤噪音,实时字幕默认不会显示 `家``家确` 这类很短的瞬时误识别;最终发送给 LLM 的内容仍只以 `转写结果:...` 为准。`OWNER_REALTIME_TRANSCRIPT_IDLE_TIMEOUT_MS=1500` 表示已有实时字幕后,如果 1.5 秒内没有新的文字输出,就直接结束本轮录音进入 final STT;设置为 `0` 可以关闭这个停滞端点。`OWNER_REALTIME_TRANSCRIPT_ENABLED=0` 可以临时关闭实时显示。
`OWNER_CONTINUOUS_DIALOG_ENABLED=1` 表示每轮回复播放后会自动判断是否继续对话。若助手回复里明显在问用户、要求补充信息或让用户选择,终端会输出 `继续对话:3秒内可直接回答`,这 3 秒内可以不用再说“小杰小杰”。若助手只是完成回答、报错、拒绝或判断不确定,就直接恢复待机。`OWNER_CONTINUATION_DECISION_PROVIDER=hybrid` 表示先用本地规则判断,规则不确定时才调用云端 LLM 做小分类;低于 `OWNER_CONTINUATION_CONFIDENCE_THRESHOLD=0.65` 的结果按待机处理。
`OWNER_BARGE_IN_ENABLED=1` 表示播报中允许打断。播放开始后的 `OWNER_BARGE_IN_ECHO_GUARD_MS=500` 毫秒内忽略麦克风输入,之后如果检测到至少 `OWNER_BARGE_IN_MIN_SPEECH_MS=250` 毫秒有效用户语音,并且 realtime STT 给出有效 partial,就停止剩余播报。上下文只记录已经完整播出的 assistant 句子,未播出的内容不会写入临时历史。
## 本地模型
@@ -114,6 +128,16 @@ python3.11 scripts/download_speech_models.py --dir models
.venv/bin/python -m owner_voice_pet simulate-live --turns 2 --fixture /tmp/owner-simulated-mic.jsonl
```
真实 Provider 完整链路自测:
```bash
.venv/bin/python -m owner_voice_pet real-live-check --turns 2
```
这个命令不用真人对着麦克风说话,会用 macOS `say/afconvert` 生成“小杰小杰”和两轮问题音频,再驱动真实 `VoiceAssistantPipeline`:本地 KWS、本地 VAD、本地降噪、本地实时字幕、本地 final STT、云端 LLM、本地 TTS、Transport 播放、恢复待机都会跑到。默认会真实播放 ACK 和回复;如果只想检查链路但不发声,可以加 `--no-playback`。输出 JSON 中 `success=true` 表示完整链路通过,并会检查第二轮 LLM 请求是否携带第一轮临时历史。
输出里的 `timing` 会给出命令 `started_at``finished_at`、总 `duration_ms`、各 phase 耗时,以及按 turn/stage 聚合的 `stage_timings`,例如 wake 等待、ACK、等待说话、录音、final STT、LLM 到首个 TTS、TTS 播放、整轮总耗时。`llm_request_timings` 会记录每次云端 LLM 请求真正发出去的 `sent_at`、首个回复文本耗时 `first_delta_ms` 和请求总耗时 `duration_ms`
单轮验收:
```bash
@@ -126,14 +150,15 @@ python3.11 scripts/download_speech_models.py --dir models
.venv/bin/python -m owner_voice_pet run-live
```
运行后终端状态来自 pipeline event bus,会显示待机、唤醒命中、应答中、请说出问题、录音中、检测到用户语音、实时转写、用户语音结束、转写中、转写结果、思考中、播放中、恢复待机等状态。说“小杰小杰”,听到“我在”且看到“请说出问题”后再提问;听到回复后可以再次说“小杰小杰”继续下一轮。本次进程内会携带临时历史,程序退出后不保存。背景噪声下如果实时字幕仍偶发短错字,先看最终 `转写结果`;最终文本才会进入 LLM。
运行后终端状态来自 pipeline event bus,会显示待机、唤醒命中、应答中、请说出问题、录音中、检测到用户语音、实时转写、用户语音结束、转写中、转写结果、思考中、播放中、继续对话或恢复待机等状态。说“小杰小杰”,听到“我在”且看到“请说出问题”后再提问;如果助手回复后判断需要你继续回答,可以在 3 秒内直接说下一句,不需要再次唤醒。若助手已经完成回答,会自动恢复待机。本次进程内会携带临时历史,程序退出后不保存。背景噪声下如果实时字幕仍偶发短错字,先看最终 `转写结果`;最终文本才会进入 LLM。
## 验证
```bash
.venv/bin/python -m compileall src tests scripts
.venv/bin/python -m unittest discover -s tests
.venv/bin/python -m owner_voice_pet simulate-live --turns 2
.venv/bin/python -m owner_voice_pet simulate-live --turns 3
.venv/bin/python -m owner_voice_pet real-live-check --turns 2 --no-playback
.venv/bin/python -m owner_voice_pet acceptance
.venv/bin/python -m owner_voice_pet validate-assets
.venv/bin/python -m owner_voice_pet security-check
@@ -147,6 +147,18 @@ The first implementation is a per-turn heuristic endpoint, not persistent voicep
4. `VoiceAssistantPipeline``speech_started` 后把 capture 阶段已开始录音的帧 feed 给 realtime STT session;当 partial 文本变化时才 emit,避免刷屏。
5. `OWNER_REALTIME_TRANSCRIPT_ENABLED=1` 默认启用;设置为 `0` 可临时回退到只显示 final transcript。
## Realtime Transcript Idle Endpoint
真人 `run-live` 日志显示,用户已看到实时字幕完整推进后,capture 仍可能继续等待 VAD 或主说话人端点。新增 `OWNER_REALTIME_TRANSCRIPT_IDLE_TIMEOUT_MS` 作为文本级端点,默认 `1500` ms
1. CaptureStage 只在已经输出过至少一次有效 `transcript_partial` 后启用该端点。
2. 每次 `transcript_partial` 真实发出时,记录该帧的音频时间戳为 `last_partial_ms`
3. 后续已开始录音但没有新的 partial 文本输出时,如果当前帧时间戳与 `last_partial_ms` 的差值达到配置阈值,调用 `VadRecorder.finish("partial_transcript_idle")` 构建当前录音片段。
4. 构建出的 segment 继续走 final STT、LLM、TTS,不把 partial 文本写入上下文。
5. 配置为 `0` 时关闭该端点,保留只依赖主说话人/VAD 的旧行为。
该端点使用音频帧时间戳而不是 wall-clock,避免批量读帧或 CPU 抖动导致误判。它不会在没有 partial 的场景提前结束,避免 STT 尚未给出第一段文本时切掉用户开头。
## Model Files
```text
@@ -238,6 +250,82 @@ The generated sequence intentionally contains:
The transport is bounded: if frames are exhausted unexpectedly, it raises a structured validation error instead of letting the pipeline loop forever. `--write-fixture` writes the exact simulated microphone frames as JSONL, and `--fixture` replays them for repeatable debugging.
## Real Provider Fixture Acceptance and Playback Drain
The simulated microphone command validates pipeline ordering with fake providers, but it does not prove that real local models and real playback can run together. The next acceptance layer uses generated 16 kHz PCM utterances as microphone input while keeping the provider chain real:
```text
owner_voice_pet real-live-check
-> generated wake/question PCM via macOS say + afconvert
-> MemoryAudioTransport(flush preserves prefilled fixture frames)
-> SherpaOnnxKeywordWakeWordProvider(real KWS)
-> HybridVadProvider(real sherpa VAD + energy fallback)
-> SherpaOnnxDenoiserPreprocessor(real GTCRN)
-> PrimarySpeakerVadRecorder
-> SherpaOnnxSttProvider(real local CTC partial/final)
-> OpenAICompatibleLlmProvider(real cloud LLM from .env)
-> MacSayTtsProvider(real macOS TTS)
-> SoundDeviceAudioTransport.play_pcm(real speaker smoke)
```
The CLI defaults to two generated turns: “我叫阿明,请你记住我的名字。” followed by “我叫什么名字?”. A recording wrapper around the real LLM provider stores the messages sent to each LLM call, so the command can verify that the second request contains first-turn process-local user/assistant history without exposing the API key. `--question` can override the generated prompts, `--voice` controls the macOS `say` voice, and `--no-playback` keeps TTS synthesis and playback events while skipping speaker output for quiet automated runs.
Playback drain semantics are intentionally split by transport type:
1. `SoundDeviceAudioTransport.flush_input()` clears the realtime callback queue that accumulated during ACK/TTS playback.
2. `MemoryAudioTransport` defaults to the same destructive flush behavior for direct unit tests.
3. Simulated and fixture-driven live tests set `flush_clears_input=False` because their queued frames represent future time-ordered microphone input, not already accumulated realtime echo.
`_drain_input_after_playback()` therefore always calls `flush_input()` once after actual playback. If `OWNER_POST_PLAYBACK_DRAIN_MS=0`, it returns immediately and does not perform any timed read/drop loop. If the value is positive, the positive window is treated as an explicit user tuning and the runtime reads and discards only that configured duration before a final flush. When `OWNER_WAKE_ACK_TEXT` is empty and no ACK playback occurs, the runtime skips playback drain entirely.
## Real Provider Timing Observability
`owner_voice_pet real-live-check` is also the repeatable latency diagnostic entry for the full chain. The command records timing at three levels:
1. Command lifecycle: `started_at`, `finished_at`, epoch millisecond timestamps, and total `duration_ms`.
2. Preparation phases: `prepare_config`, `generate_fixture_audio`, `build_pipeline`, and `pipeline_run`, each with start offset and duration.
3. Pipeline events: every emitted stage event is recorded with turn id, ISO timestamp, command-relative offset, and message. Derived `stage_timings` pair stable stage boundaries such as `wake_listening -> wake_detected`, `capture_started -> speech_started`, `speech_started -> speech_ended`, `stt_started -> transcript_final`, `llm_started -> tts_started`, `tts_started -> playback_finished`, and `wake_listening -> standby_resumed`.
The real LLM provider is wrapped by `RecordingLlmProvider`. For every cloud LLM call it records:
1. `sent_at` and `sent_offset_ms`, captured immediately before delegating to the OpenAI-compatible provider.
2. `first_delta_at` and `first_delta_ms`, captured when the first non-empty text delta is yielded.
3. `finished_at` and `duration_ms`, captured when the stream ends or unwinds through an error.
4. `message_count` and `last_user_preview`, enough to correlate the request with a turn without serializing credentials or full request payloads.
Timing output is diagnostic metadata only. It SHALL NOT include `OWNER_LLM_API_KEY`, raw audio, denoised audio, full provider request headers, or any other secret-bearing configuration.
## Automatic Continuous Dialog Decision
持续对话不再等同于“每次回复后固定打开 3 秒追问窗口”。新的回复后状态流为:
```text
llm_started
-> tts_started/playback_finished
-> continuation_decision_started
-> continuation_decision_made(continue|standby)
-> standby_resumed
OR followup_listening -> capture_started -> transcript_final -> llm_started ...
```
`ContinuationDecisionStage` 使用 hybrid provider
1. `RuleContinuationDecisionProvider` 先用可解释规则判断。明确问题、补充信息请求、选择题、等待用户下一步的表达判定为 `continue`;完成性陈述、播报结果、拒绝/报错、寒暄结束和不确定表达判定为 `standby`
2. `LlmContinuationDecisionProvider` 只在规则返回 uncertain 时调用云端 LLM 小分类器。分类器 prompt 要求只输出 `continue``standby`,可附带 `confidence`;解析失败或低于 `OWNER_CONTINUATION_CONFIDENCE_THRESHOLD` 时按 `standby`
3. `HybridContinuationDecisionProvider` 是默认 provider,由 `OWNER_CONTINUATION_DECISION_PROVIDER=hybrid` 启用;`rule` 可用于完全离线决策。
Follow-up 捕获复用现有 capture、实时字幕、final STT、LLM、TTS 流程,但跳过 wake 和 ACK。为了让“3 秒内无回答”真正快退,follow-up 捕获临时把 recorder no-speech timeout 限制为 `OWNER_FOLLOWUP_LISTEN_TIMEOUT_MS`;若返回 `VAD_TIMEOUT_NO_SPEECH`,事件层输出 `followup_timeout``continuous_session_ended`,然后恢复 standby。
## Barge-in Playback Interruption
第一版打断只解决 turn-based 播放中的用户抢话,不做电话式 full-duplex。实现边界如下:
1. `MacSayTtsProvider` 产物优先转成 16 kHz/16-bit/mono PCM,使播放阶段可以切成约 100 ms 小段,避免整段 `afplay` 阻塞。
2. `SpeakStage` 播放每个小段后检查 microphone queue。`OWNER_BARGE_IN_ECHO_GUARD_MS` 内只 flush/忽略,降低播报回声误触发。
3. echo guard 之后,若 VAD 累计有效语音达到 `OWNER_BARGE_IN_MIN_SPEECH_MS` 且 realtime STT partial 有效,发出 `barge_in_detected` 并停止剩余句子/剩余回复播放。
4. 已完整播放的句子文本才写入 process-local assistant context;未播出的句子和后续未播放 token 不写入上下文。
5. 打断后的用户输入进入免唤醒 follow-up 捕获路径,保留实时字幕和 final STT,最终仍只有 final transcript 进入 LLM。
## Migration
No database migration. Users should run:
@@ -57,6 +57,7 @@
15. 降噪缺失问题:正式问题录音直接把原始麦克风帧送入 VAD、partial STT 和 final STT,背景噪声会同时影响端点、实时字幕和最终识别。
16. 本地模型落后问题:默认 STT 仍是 2023 年 14M 小模型,适合最小验收但不适合作为默认实时字幕质量基线;应升级为 sherpa-onnx 官方 2025 中文 CTC int8 模型。
17. 自测闭环问题:现有 `acceptance` 只覆盖旧单轮 pipeline,不能证明当前 `VoiceAssistantPipeline` 在“模拟麦克风 -> wake -> capture -> partial -> final -> LLM -> TTS -> standby -> 第二轮”路径上完整正常;真人测试前缺少可重复的自动调试入口。
18. ACK 缓冲实现偏差:文档要求 `OWNER_POST_PLAYBACK_DRAIN_MS=0` 时仍应清理播放期间已积压的输入队列,但当前实现 `<=0` 直接返回;真实麦克风上可能把“我在”播放回声或旧缓冲送入正式问题 VAD/STT,模拟预排帧又不能直接用真实 flush 语义清空。
## 详细需求
@@ -97,6 +98,16 @@
33. 模拟麦克风输入 SHALL 包含 wake 帧、正式问题主说话人帧、短噪声 partial、背景噪声/非主说话人帧和第二轮重复唤醒帧。
34. 模拟 transport SHALL 有边界保护:如果帧提前耗尽,命令必须结构化失败并退出,不能无限等待。
35. `simulate-live` SHALL 支持写入和回放 JSONL fixture,便于后续持续复现同一组模拟麦克风输入。
36. ACK/TTS 播放完成后的输入清理 SHALL 在 `OWNER_POST_PLAYBACK_DRAIN_MS=0` 时仍执行一次 `flush_input()`,清掉播放期间已进入真实输入队列的回声或旧帧,但不得再主动读取并丢弃新音频。
37. 当 ACK 文本为空且没有实际播放行为时,runtime SHALL NOT 执行 post-playback drain,避免无播放场景误清正式问题开头。
38. 模拟麦克风和 fixture replay 场景 SHALL 能保留预排的未来输入帧,并仍记录 flush 调用次数,用于验证真实队列清理动作不吞掉测试中尚未“发生”的后续用户语音。
39. 完整链路自测 SHALL 尽量使用真实本地 provider 和真实云端 LLM:真实 KWS、VAD、降噪、STT、LLM、TTS 和播放;无法自动使用真人麦克风时,允许用生成的 16 kHz PCM 音频帧替代物理麦克风输入。
40. 系统 SHALL 提供 `owner_voice_pet real-live-check` 命令,把真实 Provider 完整链路自测固化为可重复入口,而不是依赖一次性内联脚本。
41. `real-live-check` SHALL 默认生成两轮音频并执行真实本地 KWS/VAD/降噪/STT、本地 TTS、Transport 播放和云端 LLM;输出 JSON 必须包含完成轮数、final transcripts、事件计数、播放段数、flush 次数、临时上下文检查和错误列表。
42. `real-live-check --no-playback` SHALL 仍执行本地 TTS 合成和 pipeline 播放事件记录,但不实际向扬声器发声,便于自动化测试和无声环境排查。
43. `real-live-check` SHALL 输出命令级开始/结束时间、总耗时、阶段耗时、pipeline 事件时间线和按 turn/stage 聚合的响应耗时。
44. `real-live-check` SHALL 输出每次云端 LLM 请求真正发出的 `sent_at`、首个文本回复耗时和请求总耗时,且不得泄露 API key。
45. `run-live` capture 阶段 SHALL 支持实时字幕停滞端点:当已经输出过 `transcript_partial` 后,如果 `OWNER_REALTIME_TRANSCRIPT_IDLE_TIMEOUT_MS` 时间内没有新的实时字幕文本输出,当前正式问题采集必须立即结束并进入 final STT。
### 非功能需求
@@ -127,6 +138,8 @@
12. partial 单字误识别:本地 streaming STT 输出 `家``确``a` 等单字时,终端不得显示为 `实时转写`
13. partial 瞬态跳变:本地 streaming STT 从“你是谁”短暂跳到“加”再回到“你是谁”时,不得显示中间极短跳变。
14. CTC 模型缺旧 transducer 文件:当 manifest 类型为 `sherpa-onnx-streaming-zipformer2-ctc` 时,不应再要求 encoder/decoder/joiner 文件存在。
15. ACK 禁用:当 `OWNER_WAKE_ACK_TEXT=` 时,系统不得因为“播放后清理”逻辑而清空唤醒后的正式问题帧。
16. 模拟预排帧:当 `MemoryAudioTransport` 用于模拟完整 turn 序列时,flush 调用不得把未来 turn 的预排帧删除,否则模拟验收会掩盖真实 pipeline 顺序。
### 输入输出规格
@@ -151,6 +164,7 @@
17. `OWNER_NOISE_FILTER_ENABLED=1`:正式问题阶段默认启用本地降噪。
18. `OWNER_WAKE_DENOISE_ENABLED=0`wake 阶段默认不启用降噪。
19. `OWNER_NOISE_FILTER_PROVIDER=sherpa_onnx_gtcrn`:第一版本地降噪 provider。
20. `OWNER_REALTIME_TRANSCRIPT_IDLE_TIMEOUT_MS=1500`:已有实时字幕后,若 1.5 秒没有新的文字输出,则结束本轮录音;设置为 `0` 可关闭该端点。
终端输出:
@@ -290,6 +304,9 @@ standby
10. 捕获阶段降噪:`AudioPreprocessStage` 将 int16 PCM 转为 float32,调用 `sherpa_onnx.OnlineSpeechDenoiser.run(samples, sample_rate)`,把返回的 `DenoisedAudio.samples` 转回 int16 PCM,并在 metadata 中加入 `denoised=True``noise_filter_provider=sherpa_onnx_gtcrn`
11. CTC STT 加载:manifest `stt.type=sherpa-onnx-streaming-zipformer2-ctc` 时,只校验 `tokens``model`,调用 `OnlineRecognizer.from_zipformer2_ctc`;旧 manifest 仍兼容 transducer 路径。
12. partial 稳定过滤:实时字幕会先去除空白和标点,计算有效字符数;有效字符数小于 2 的文本直接忽略;若新文本比上一次已显示文本短且不构成稳定前缀推进,也忽略,避免把瞬时噪声显示给用户。
13. 零时长播放后清理:`_drain_input_after_playback()` 先执行一次 `transport.flush_input()`;若 `post_playback_drain_ms <= 0` 立即返回;若配置为正数,才按用户显式配置继续 read/drop drain window 并最终再次 flush。
14. 模拟 flush 隔离:`MemoryAudioTransport` 默认保持 flush 清空语义;模拟 live pipeline 显式设置 `flush_clears_input=False`,只记录 flush 调用,不删除预排帧,避免将真实时间队列与离线 fixture 序列混淆。
15. 实时字幕停滞端点:CaptureStage 记录最后一次成功输出 `transcript_partial` 的音频时间戳;后续已开始录音但没有新的 partial 文本推进时,若当前帧时间戳与最后 partial 输出时间差达到 `OWNER_REALTIME_TRANSCRIPT_IDLE_TIMEOUT_MS`,调用 recorder `finish("partial_transcript_idle")` 构建当前片段并发出 `speech_ended`
### 数据库/状态管理变更
@@ -341,6 +358,7 @@ standby
| 模拟验收与真实麦克风仍有差异 | 中 | 中 | 明确模拟验收用于自动调试 live pipeline 状态机和事件顺序;仍保留 4.3 真人 `run-live` 验收,不在用户确认前归档 |
| 模拟帧耗尽导致测试卡死 | 中 | 高 | 使用 bounded simulated transport,空读超过阈值直接抛结构化错误并让命令返回非 0 |
| 模拟 provider 掩盖真实模型加载问题 | 中 | 中 | 模拟命令只验证 pipeline;模型加载仍由 `model-check` 真实加载 KWS/VAD/STT/denoiser 覆盖 |
| `0ms` drain 不执行 flush 导致播放回声污染正式问题 | 中 | 高 | `_drain_input_after_playback()` 无论 drain window 是否为 0 都先 flush 一次真实输入队列;新增回归测试验证 flush 发生且首句不丢 |
## 任务分解
@@ -414,6 +432,35 @@ standby
- [ ] 11.4 补充 README 和自动化测试;前置条件:11.3 完成;验收标准:README 包含模拟验收命令,测试验证两轮闭环、partial 噪声过滤、第二轮临时上下文、fixture 写入/回放;测试要点:unittest;优先级:P0;预计:35 分钟。
- [ ] 11.5 执行全量门禁并提交“模拟麦克风验收”模块;前置条件:11.1 至 11.4 完成;验收标准:`simulate-live --turns 2`、compileall、unittest、security-check、model-check、device-check、OpenSpec strict、git diff check 通过后中文 commit;优先级:P0;预计:45 分钟。
### 12. 真实完整链路自测与 ACK 缓冲修正
- [ ] 12.1 更新 OpenSpec 描述真实完整链路自测和 ACK 零时长 flush 偏差;前置条件:模拟验收已提交且本机模型齐全;验收标准:proposal/design/spec/tasks/spec delta 覆盖真实 provider 验收、0ms flush、模拟预排帧隔离和 ACK 禁用边缘案例;测试要点:OpenSpec strict;优先级:P0;预计:35 分钟。
- [ ] 12.2 修正 `VoiceAssistantPipeline` 和兼容 `LiveVoiceRuntime` 播放后输入清理;前置条件:12.1 完成;验收标准:ACK/TTS 播放后即使 `OWNER_POST_PLAYBACK_DRAIN_MS=0` 也执行一次 flush,ACK 文本为空时不执行 post-playback drain;测试要点:零时长 flush 单测、ACK 禁用不清空单测;优先级:P0;预计:45 分钟。
- [ ] 12.3 调整 `MemoryAudioTransport` 和模拟验收;前置条件:12.2 完成;验收标准:默认 flush 仍清空输入,模拟 live 显式保留预排帧并记录 flush 调用次数;测试要点:transport flush 双语义测试、两轮 simulated live 回归;优先级:P0;预计:35 分钟。
- [ ] 12.4 执行真实 provider 脚本化闭环验收;前置条件:本地模型、`.env` LLM key、macOS TTS/播放可用;验收标准:生成 wake/question 音频帧驱动真实 KWS、VAD、denoiser、本地 STT、云端 LLM、本地 TTS、播放,两轮上下文可引用第一轮历史;测试要点:输出不泄露 key,模型文件不进 Git;优先级:P0;预计:60 分钟。
- [ ] 12.5 执行全量门禁并提交“真实链路自测”模块;前置条件:12.1 至 12.4 完成;验收标准:compileall、unittest、simulate-live、security-check、model-check、device-check、OpenSpec strict、git diff check 全通过后中文 commit;优先级:P0;预计:45 分钟。
### 13. 固化真实 Provider 完整流程 CLI
- [ ] 13.1 更新 OpenSpec 和 README 描述 `real-live-check`;前置条件:12 组发现临时脚本证据不可重复;验收标准:proposal/design/spec/tasks/README 明确命令用途、默认真实播放、`--no-playback` 和 JSON 检查项;测试要点:OpenSpec strict;优先级:P0;预计:35 分钟。
- [ ] 13.2 实现 `owner_voice_pet real-live-check`;前置条件:13.1 完成;验收标准:命令从 `.env` 读取配置,生成 wake/question PCM,运行真实 KWS/VAD/denoise/STT/LLM/TTS/pipeline/playback,并输出结构化 JSON;测试要点:CLI wiring 测试、transport 单测;优先级:P0;预计:60 分钟。
- [ ] 13.3 执行真实命令验收和全量门禁;前置条件:13.2 完成;验收标准:`real-live-check --turns 2`、compileall、unittest、simulate-live、security-check、model-check、device-check、OpenSpec strict、git diff check 通过;优先级:P0;预计:60 分钟。
- [ ] 13.4 提交“真实流程命令”模块;前置条件:13.3 通过;验收标准:中文 commit 信息为 `[真实流程命令]:完成完整链路自测入口,包含真实Provider CLI、播放验收和文档测试`,提交后 `git status --short` 为空;优先级:P0;预计:10 分钟。
### 14. 真实流程命令计时输出
- [ ] 14.1 更新 OpenSpec 和 README 描述真实流程计时字段;前置条件:`real-live-check` 已可运行;验收标准:proposal/design/spec/tasks/README 明确 `started_at``finished_at``duration_ms``stage_timings``llm_request_timings.sent_at`;测试要点:OpenSpec strict;优先级:P0;预计:30 分钟。
- [ ] 14.2 实现命令级和事件级计时;前置条件:14.1 完成;验收标准:`real-live-check` JSON 包含命令总耗时、生成音频/build/pipeline phase 耗时、每轮 wake/ACK/capture/STT/LLM/TTS/turn_total 耗时;测试要点:stage timing 单测;优先级:P0;预计:45 分钟。
- [ ] 14.3 实现 LLM 请求发送时间和响应耗时;前置条件:14.2 完成;验收标准:每次 LLM 调用记录 `sent_at``first_delta_ms``duration_ms`、最后 user 预览,且不包含 key;测试要点:RecordingLlmProvider 单测;优先级:P0;预计:35 分钟。
- [ ] 14.4 执行真实命令验收和全量门禁并提交;前置条件:14.1 至 14.3 完成;验收标准:`real-live-check --turns 2 --no-playback` 和默认播放版本均输出 timing 且成功,全量门禁通过后中文提交;优先级:P0;预计:60 分钟。
### 15. 实时字幕停滞端点
- [ ] 15.1 更新 OpenSpec 和 README 描述 1.5 秒无新实时字幕即结束录音;前置条件:真人 `run-live` 日志确认 final STT 前仍等待过久;验收标准:proposal/design/spec/tasks/README 明确 `OWNER_REALTIME_TRANSCRIPT_IDLE_TIMEOUT_MS=1500`、关闭方式和 end_reason;测试要点:OpenSpec strict;优先级:P0;预计:30 分钟。
- [ ] 15.2 实现配置与 capture 端点;前置条件:15.1 完成;验收标准:`AppConfig``.env.example``--show-config` 支持新配置,`VoiceAssistantPipeline` 在已有 partial 后 1500 ms 无新文本时调用 recorder finish;测试要点:配置解析和 show-config 单测;优先级:P0;预计:45 分钟。
- [ ] 15.3 补充 recorder finish 和回归测试;前置条件:15.2 完成;验收标准:持续有语音但 partial 不再推进时,segment end_reason 为 `partial_transcript_idle`final STT 仍执行且 partial 不进上下文;测试要点:live runtime 单测;优先级:P0;预计:45 分钟。
- [ ] 15.4 执行门禁并提交“实时字幕端点”模块;前置条件:15.1 至 15.3 完成;验收标准:compileall、unittest、simulate-live、security-check、OpenSpec strict、git diff check 通过后中文提交;优先级:P0;预计:45 分钟。
## Spec Deltas
### 新增能力
@@ -434,6 +481,12 @@ standby
10. `Realtime partial transcript output`:新增录音期间 partial transcript 事件、终端显示和上下文隔离要求。
11. `Local voice chain and noise filtering`:新增本地 STT/TTS 默认、GTCRN 降噪、CTC 模型、partial 稳定过滤和音频不上传要求。
12. `Simulated microphone live acceptance`:新增不依赖真人麦克风的 live pipeline 模拟验收,覆盖两轮重复对话、短噪声过滤、背景噪声端点、fixture 写入/回放和空帧防卡死。
13. `Real provider live fixture acceptance`:新增真实 provider 脚本化闭环验收和 ACK/TTS 播放后 0ms flush 语义,确保真实本地模型、云端 LLM、本地 TTS/播放在同一 pipeline 中可连续运行。
14. `Real live check CLI`:新增 `real-live-check` 可重复命令,把完整真实 Provider 链路验收从一次性脚本升级为仓库内正式工具。
15. `Real live check timing`:新增真实流程命令计时输出,覆盖命令发起时间、LLM 请求发送时间、各阶段响应耗时和 pipeline 事件时间线。
16. `Realtime transcript idle endpoint`:新增已有实时字幕后 1.5 秒无新文字即结束录音的端点要求,避免用户已停说但 capture 继续等待 VAD/音色端点。
17. `Automatic continuous dialog decision`:新增回复后自动判断继续/待机能力,使用规则优先、LLM 小分类兜底,不确定时默认回到待机。
18. `Barge-in playback interruption`:新增播报中用户有效说话打断能力,使用 echo guard、最短语音时长和 realtime partial 共同确认,避免播放回声误触发。
### 删除项
@@ -455,6 +508,11 @@ standby
8. M8:录音期间实时转写显示完成后提交,保留真人 `run-live` 验收任务,不在用户确认前归档。
9. M9:本地语音链路、高质量实时字幕和正式问题降噪完成后提交,继续保留真人 `run-live` 验收任务,不在用户确认前归档。
10. M10:模拟麦克风自动验收完成后提交,作为真人验收前的可重复自测入口;仍不归档变更,直到真实 `run-live` 行为由用户确认。
11. M11:真实 provider 脚本化闭环和 ACK 缓冲修正完成后提交;仍保留用户真人 `run-live` 体验验收,不在用户确认前归档。
12. M12`real-live-check` 正式 CLI、文档、测试和真实命令验收完成后提交;仍保留物理麦克风 `run-live` 真人体验验收。
13. M13`real-live-check` 计时输出完成后提交,使后续排查可以直接看到每段耗时。
14. M14:实时字幕停滞端点完成后提交,用户真实 `run-live` 若已有 partial 后 1.5 秒无新文字,应快速进入 final STT。
15. M15:自动持续对话判断与播报打断完成后提交,用户真实 `run-live` 应能在助手反问时免唤醒继续回答,在助手已完成回答时自动恢复待机。
估时:
@@ -56,6 +56,11 @@ The live runtime SHALL display recognized user utterance text in the terminal bo
- **WHEN** a partial transcript is emitted
- **THEN** it SHALL be treated as user-visible feedback only and SHALL NOT be appended to the conversation context or sent to the LLM
#### Scenario: Realtime transcript stops advancing
- **WHEN** the user utterance has started, at least one partial transcript has been displayed, and no new partial transcript text is emitted for `OWNER_REALTIME_TRANSCRIPT_IDLE_TIMEOUT_MS`
- **THEN** capture SHALL end the current utterance with end reason `partial_transcript_idle`
- **AND** the runtime SHALL proceed to final STT for the captured audio instead of waiting for VAD maximum recording duration
#### Scenario: User utterance is transcribed
- **WHEN** a live turn captures a user utterance and STT returns non-empty text
- **THEN** the terminal output SHALL include a transcript message containing the recognized text before the thinking/LLM status is emitted
@@ -130,6 +135,90 @@ The system SHALL provide a deterministic simulated microphone acceptance command
- **WHEN** the simulated microphone transport runs out of frames before the expected turn completes
- **THEN** the command SHALL fail with a structured error instead of waiting indefinitely
### Requirement: Automatic continuous dialog decision
The live runtime SHALL automatically decide after each assistant reply whether to continue listening for a direct follow-up answer or to end the continuous session and return to wake standby.
#### Scenario: Assistant asks the user a direct follow-up question
- **WHEN** an assistant reply clearly asks the user a question, requests missing information, asks the user to choose among options, or otherwise waits for the user's next answer
- **THEN** the continuation decision SHALL be `continue` with confidence at or above the configured threshold
- **AND** the runtime SHALL enter follow-up listening after playback without requiring the wake word again
#### Scenario: Assistant completes the answer
- **WHEN** an assistant reply only provides an answer, reports a result, refuses, reports an error, ends a greeting, or is ambiguous about needing more user input
- **THEN** the continuation decision SHALL be `standby`
- **AND** the runtime SHALL return to wake standby after playback
#### Scenario: Hybrid decision falls back to an LLM classifier
- **WHEN** deterministic rules cannot decide continue or standby
- **THEN** the hybrid provider SHALL call the configured cloud LLM with a classifier prompt that can only resolve to `continue` or `standby`
- **AND** a low-confidence, malformed, timed-out, or failed classifier result SHALL be treated as `standby`
#### Scenario: Follow-up answer arrives in the listening window
- **WHEN** the decision is `continue` and the user starts speaking within `OWNER_FOLLOWUP_LISTEN_TIMEOUT_MS`
- **THEN** the runtime SHALL capture, realtime-transcribe, final-transcribe, send to LLM, synthesize, and play the answer without local wake detection and without the wake acknowledgement phrase
#### Scenario: Follow-up listening times out
- **WHEN** the decision is `continue` but no user speech is detected before `OWNER_FOLLOWUP_LISTEN_TIMEOUT_MS`
- **THEN** the runtime SHALL emit `followup_timeout`
- **AND** it SHALL emit `continuous_session_ended`
- **AND** it SHALL return to wake standby
### Requirement: Barge-in playback interruption
The live runtime SHALL support a first-version barge-in path that can stop assistant playback when the user starts a valid new utterance during TTS playback.
#### Scenario: Echo guard suppresses early playback echo
- **WHEN** TTS playback starts and microphone input appears before `OWNER_BARGE_IN_ECHO_GUARD_MS`
- **THEN** the runtime SHALL NOT treat that input as a user barge-in
#### Scenario: Valid user speech interrupts playback
- **WHEN** TTS playback has passed the echo guard, VAD observes at least `OWNER_BARGE_IN_MIN_SPEECH_MS` of user speech, and realtime STT produces a valid partial transcript
- **THEN** the runtime SHALL emit `barge_in_detected`
- **AND** it SHALL stop remaining playback chunks and emit `playback_interrupted`
- **AND** it SHALL process the user's new utterance as the next input without requiring wake
#### Scenario: Interrupted assistant context is partial
- **WHEN** playback is interrupted before the full assistant reply is spoken
- **THEN** the process-local conversation context SHALL append only the assistant text that had been fully spoken before interruption
- **AND** unspoken assistant text SHALL NOT be appended to the session history
### Requirement: Real provider live fixture acceptance
The system SHALL support scripted full-chain acceptance that drives the live pipeline with generated microphone PCM while using real local speech providers, the configured cloud LLM, and local TTS/playback capability.
#### Scenario: Real provider fixture completes one turn
- **WHEN** the developer runs `.venv/bin/python -m owner_voice_pet real-live-check --turns 1`
- **THEN** the pipeline SHALL use the real local KWS, local VAD, local denoiser, local STT, configured cloud LLM, local TTS, and playback provider path to complete a wake-to-standby turn
#### Scenario: Real provider fixture completes repeated turns with temporary context
- **WHEN** the developer runs `.venv/bin/python -m owner_voice_pet real-live-check --turns 2` and the first generated turn gives a user fact
- **THEN** the second LLM request SHALL include the first turn's user and assistant history from process-local memory
- **AND** no history SHALL be read from disk or persisted after the process exits
- **AND** JSON output SHALL include checks for completed turns, event counts, final transcripts, playback, flush count, and temporary context
#### Scenario: Zero playback drain still flushes queued echo
- **WHEN** ACK or TTS playback completes and `OWNER_POST_PLAYBACK_DRAIN_MS=0`
- **THEN** the runtime SHALL call `flush_input()` once to clear already queued playback echo
- **AND** it SHALL NOT perform an additional timed microphone read-and-discard loop
#### Scenario: No acknowledgement playback is configured
- **WHEN** `OWNER_WAKE_ACK_TEXT` is empty and no wake acknowledgement audio is played
- **THEN** the runtime SHALL NOT run post-playback drain before capture starts
#### Scenario: Fixture transport preserves future frames
- **WHEN** a memory or fixture transport represents pre-generated future microphone frames
- **THEN** tests SHALL be able to configure flush to record the cleanup call without deleting those future frames
#### Scenario: Real provider check runs quietly
- **WHEN** the developer runs `.venv/bin/python -m owner_voice_pet real-live-check --turns 2 --no-playback`
- **THEN** the command SHALL still synthesize local TTS output and emit pipeline playback events
- **AND** it SHALL skip actual speaker output for automated or quiet environments
#### Scenario: Real provider check reports timing diagnostics
- **WHEN** the developer runs `.venv/bin/python -m owner_voice_pet real-live-check --turns 2`
- **THEN** the JSON output SHALL include `timing.started_at`, `timing.finished_at`, `timing.duration_ms`, `timing.phases`, `timing.events`, and `timing.stage_timings`
- **AND** `timing.stage_timings` SHALL include turn-scoped durations for wake wait, acknowledgement, wait-for-speech, capture, final STT, LLM-to-first-TTS, TTS playback, and full turn when the corresponding events are emitted
- **AND** the JSON output SHALL include `llm_request_timings` with one item per LLM call containing `sent_at`, `sent_offset_ms`, `first_delta_ms`, `finished_at`, `duration_ms`, `message_count`, and a short last-user preview
- **AND** the timing output SHALL NOT contain API keys, raw audio payloads, denoised audio payloads, or full secret-bearing provider request headers
### Requirement: Wake acknowledgement before recording
The live runtime SHALL provide an audible local acknowledgement after local wake detection and before it starts recording the user's formal question.
@@ -89,3 +89,40 @@
- [x] 11.3 新增 `owner_voice_pet simulate-live` CLI;前置条件:11.2 完成;验收标准:默认两轮模拟,输出 JSON checks,可写入/回放 JSONL fixture;测试要点:CLI 单测和真实命令;优先级:P0;预计:40 分钟。
- [x] 11.4 补充 README 和自动化测试;前置条件:11.3 完成;验收标准:README 包含模拟验收命令,测试验证两轮闭环、partial 噪声过滤、第二轮临时上下文、fixture 写入/回放;测试要点:unittest;优先级:P0;预计:35 分钟。
- [x] 11.5 执行全量门禁并提交“模拟麦克风验收”模块;前置条件:11.1 至 11.4 完成;验收标准:`simulate-live --turns 2`、compileall、unittest、security-check、model-check、device-check、OpenSpec strict、git diff check 通过后中文 commit;优先级:P0;预计:45 分钟。
## 12. 真实完整链路自测与 ACK 缓冲修正
- [x] 12.1 更新 OpenSpec 描述真实完整链路自测和 ACK 零时长 flush 偏差;前置条件:模拟验收已提交且本机模型齐全;验收标准:proposal/design/spec/tasks/spec delta 覆盖真实 provider 验收、0ms flush、模拟预排帧隔离和 ACK 禁用边缘案例;测试要点:OpenSpec strict;优先级:P0;预计:35 分钟。
- [x] 12.2 修正 `VoiceAssistantPipeline` 和兼容 `LiveVoiceRuntime` 播放后输入清理;前置条件:12.1 完成;验收标准:ACK/TTS 播放后即使 `OWNER_POST_PLAYBACK_DRAIN_MS=0` 也执行一次 flush,ACK 文本为空时不执行 post-playback drain;测试要点:零时长 flush 单测、ACK 禁用不清空单测;优先级:P0;预计:45 分钟。
- [x] 12.3 调整 `MemoryAudioTransport` 和模拟验收;前置条件:12.2 完成;验收标准:默认 flush 仍清空输入,模拟 live 显式保留预排帧并记录 flush 调用次数;测试要点:transport flush 双语义测试、两轮 simulated live 回归;优先级:P0;预计:35 分钟。
- [x] 12.4 执行真实 provider 脚本化闭环验收;前置条件:本地模型、`.env` LLM key、macOS TTS/播放可用;验收标准:生成 wake/question 音频帧驱动真实 KWS、VAD、denoiser、本地 STT、云端 LLM、本地 TTS、播放,两轮上下文可引用第一轮历史;测试要点:输出不泄露 key,模型文件不进 Git;优先级:P0;预计:60 分钟。
- [x] 12.5 执行全量门禁并提交“真实链路自测”模块;前置条件:12.1 至 12.4 完成;验收标准:compileall、unittest、simulate-live、security-check、model-check、device-check、OpenSpec strict、git diff check 全通过后中文 commit;优先级:P0;预计:45 分钟。
## 13. 固化真实 Provider 完整流程 CLI
- [x] 13.1 更新 OpenSpec 和 README 描述 `real-live-check`;前置条件:12 组发现临时脚本证据不可重复;验收标准:proposal/design/spec/tasks/README 明确命令用途、默认真实播放、`--no-playback` 和 JSON 检查项;测试要点:OpenSpec strict;优先级:P0;预计:35 分钟。
- [x] 13.2 实现 `owner_voice_pet real-live-check`;前置条件:13.1 完成;验收标准:命令从 `.env` 读取配置,生成 wake/question PCM,运行真实 KWS/VAD/denoise/STT/LLM/TTS/pipeline/playback,并输出结构化 JSON;测试要点:CLI wiring 测试、transport 单测;优先级:P0;预计:60 分钟。
- [x] 13.3 执行真实命令验收和全量门禁;前置条件:13.2 完成;验收标准:`real-live-check --turns 2`、compileall、unittest、simulate-live、security-check、model-check、device-check、OpenSpec strict、git diff check 通过;优先级:P0;预计:60 分钟。
- [x] 13.4 提交“真实流程命令”模块;前置条件:13.3 通过;验收标准:中文 commit 信息为 `[真实流程命令]:完成完整链路自测入口,包含真实Provider CLI、播放验收和文档测试`,提交后 `git status --short` 为空;优先级:P0;预计:10 分钟。
## 14. 真实流程命令计时输出
- [x] 14.1 更新 OpenSpec 和 README 描述真实流程计时字段;前置条件:`real-live-check` 已可运行;验收标准:proposal/design/spec/tasks/README 明确 `started_at``finished_at``duration_ms``stage_timings``llm_request_timings.sent_at`;测试要点:OpenSpec strict;优先级:P0;预计:30 分钟。
- [x] 14.2 实现命令级和事件级计时;前置条件:14.1 完成;验收标准:`real-live-check` JSON 包含命令总耗时、生成音频/build/pipeline phase 耗时、每轮 wake/ACK/capture/STT/LLM/TTS/turn_total 耗时;测试要点:stage timing 单测;优先级:P0;预计:45 分钟。
- [x] 14.3 实现 LLM 请求发送时间和响应耗时;前置条件:14.2 完成;验收标准:每次 LLM 调用记录 `sent_at``first_delta_ms``duration_ms`、最后 user 预览,且不包含 key;测试要点:RecordingLlmProvider 单测;优先级:P0;预计:35 分钟。
- [x] 14.4 执行真实命令验收和全量门禁并提交;前置条件:14.1 至 14.3 完成;验收标准:`real-live-check --turns 2 --no-playback` 和默认播放版本均输出 timing 且成功,全量门禁通过后中文提交;优先级:P0;预计:60 分钟。
## 15. 实时字幕停滞端点
- [x] 15.1 更新 OpenSpec 和 README 描述 1.5 秒无新实时字幕即结束录音;前置条件:真人 `run-live` 日志确认 final STT 前仍等待过久;验收标准:proposal/design/spec/tasks/README 明确 `OWNER_REALTIME_TRANSCRIPT_IDLE_TIMEOUT_MS=1500`、关闭方式和 end_reason;测试要点:OpenSpec strict;优先级:P0;预计:30 分钟。
- [x] 15.2 实现配置与 capture 端点;前置条件:15.1 完成;验收标准:`AppConfig``.env.example``--show-config` 支持新配置,`VoiceAssistantPipeline` 在已有 partial 后 1500 ms 无新文本时调用 recorder finish;测试要点:配置解析和 show-config 单测;优先级:P0;预计:45 分钟。
- [x] 15.3 补充 recorder finish 和回归测试;前置条件:15.2 完成;验收标准:持续有语音但 partial 不再推进时,segment end_reason 为 `partial_transcript_idle`final STT 仍执行且 partial 不进上下文;测试要点:live runtime 单测;优先级:P0;预计:45 分钟。
- [x] 15.4 执行门禁并提交“实时字幕端点”模块;前置条件:15.1 至 15.3 完成;验收标准:compileall、unittest、simulate-live、security-check、OpenSpec strict、git diff check 通过后中文提交;优先级:P0;预计:45 分钟。
## 16. 自动持续对话判断与播报打断
- [x] 16.1 更新 OpenSpec 和 README 描述自动持续对话、免唤醒追问和播报打断;前置条件:用户确认不再固定 3 秒追问窗口;验收标准:proposal/design/spec/tasks/README 明确 `ContinuationDecisionStage`、hybrid 规则、follow-up 3 秒窗口、barge-in echo guard 和默认 standby 策略;测试要点:OpenSpec strict;优先级:P0;预计:45 分钟。
- [x] 16.2 实现配置、事件和续聊决策 Provider;前置条件:16.1 完成;验收标准:`AppConfig``.env.example``--show-config` 支持持续对话配置,新增事件可被 reporter/test 消费,规则优先和 LLM 兜底 provider 可单测;测试要点:规则 continue/standby、不确定低置信 standby、高置信 continue;优先级:P0;预计:60 分钟。
- [x] 16.3 重构 `VoiceAssistantPipeline/TurnController` 支持自动 follow-up;前置条件:16.2 完成;验收标准:回复后先判定 continue/standbycontinue 时播放完进入 `followup_listening`,3 秒内用户直接回答无需 wake/ACK,超时恢复 standby;测试要点:连续对话、follow-up timeout、上下文只写 final STT;优先级:P0;预计:60 分钟。
- [x] 16.4 实现可打断播放路径;前置条件:16.3 完成;验收标准:TTS 播放前 500 ms 忽略回声,之后检测到至少 250 ms 有效用户语音且 realtime partial 有效时触发 `barge_in_detected -> playback_interrupted`,上下文只写已播出的 assistant 文本;测试要点:guard window 不打断、有效输入打断、未播文本不进上下文;优先级:P0;预计:60 分钟。
- [x] 16.5 执行门禁并提交“连续对话判断”模块;前置条件:16.1 至 16.4 完成;验收标准:compileall、unittest、simulate-live --turns 3、real-live-check --turns 2 --no-playback、security-check、model-check、device-check、OpenSpec strict、git diff check 通过后提交;优先级:P0;预计:60 分钟。
+2 -1
View File
@@ -27,7 +27,7 @@ from .llm import MockLlmProvider, OpenAICompatibleLlmProvider
from .pipeline import PipelineResult, VoicePipeline
from .runtime import LiveVoiceRuntime, RuntimeSummary, TerminalRuntimeReporter, TurnResult, build_live_runtime
from .simulation import run_simulated_live
from .tts import CloudTtsProvider, MacSayTtsProvider, SentenceBuffer, SineTtsProvider
from .tts import CloudTtsProvider, MacSayTtsProvider, SentenceBuffer, SineTtsProvider, sanitize_tts_text
from .assets import validate_pet_assets
from .ui import ConsolePetWindow, PetStateController, PetVisualState
@@ -70,6 +70,7 @@ __all__ = [
"MacSayTtsProvider",
"SentenceBuffer",
"SineTtsProvider",
"sanitize_tts_text",
"validate_pet_assets",
"ConsolePetWindow",
"PetStateController",
+373 -58
View File
@@ -5,12 +5,20 @@ from typing import Protocol
from .audio_preprocess import NoopAudioPreprocessor
from .config import AppConfig
from .continuation import ContinuationDecision, ContinuationDecisionProvider, build_continuation_decider
from .conversation import ConversationContext
from .events import (
ACK_STARTED,
BARGE_IN_DETECTED,
CAPTURE_STARTED,
CONTINUATION_DECISION_MADE,
CONTINUATION_DECISION_STARTED,
CONTINUOUS_SESSION_ENDED,
FOLLOWUP_LISTENING,
FOLLOWUP_TIMEOUT,
LLM_STARTED,
PLAYBACK_FINISHED,
PLAYBACK_INTERRUPTED,
QUESTION_PROMPT,
RECOVERING,
SPEECH_ENDED,
@@ -39,7 +47,7 @@ from .protocols import (
WakeWordProvider,
)
from .stt import is_valid_transcript_text
from .tts import SentenceBuffer
from .tts import SentenceBuffer, sanitize_tts_text
from .vad import VadRecorder
@@ -61,6 +69,8 @@ class TurnResult:
assistant_text: str = ""
error: ProviderError | None = None
states: list[PipelineState] = field(default_factory=list)
completed_turns: int = 0
failed_turns: int = 0
@dataclass(slots=True)
@@ -71,6 +81,12 @@ class RuntimeSummary:
last_error: ProviderError | None = None
@dataclass(slots=True)
class SpeakResult:
spoken_text: str
interrupted: bool = False
class TurnController:
def __init__(
self,
@@ -87,6 +103,7 @@ class TurnController:
ack_tts: TtsProvider,
context: ConversationContext,
event_bus: PipelineEventBus,
continuation_decider: ContinuationDecisionProvider,
sentence_buffer: SentenceBuffer | None = None,
) -> None:
self.config = config
@@ -101,8 +118,12 @@ class TurnController:
self.ack_tts = ack_tts
self.context = context
self.event_bus = event_bus
self.continuation_decider = continuation_decider
self.sentence_buffer = sentence_buffer or SentenceBuffer()
self._states: list[PipelineState] = []
self._pending_capture_frames: list[AudioFrame] = []
self._cached_ack_text: str | None = None
self._cached_ack_segment: AudioSegment | None = None
def run_turn(self, turn_id: int) -> TurnResult:
self._states = []
@@ -120,6 +141,15 @@ class TurnController:
except ProviderError as exc:
return self._recover(exc, turn_id)
def prepare_ack_audio(self) -> None:
text = self.config.wake_ack_text.strip()
if not text:
self._cached_ack_text = None
self._cached_ack_segment = None
return
self._cached_ack_text = text
self._cached_ack_segment = self.ack_tts.synthesize(text)
def _wait_for_wake_and_user_text(self, turn_id: int) -> str | ProviderError:
wake_error = self._wait_for_local_wake()
if wake_error is not None:
@@ -129,7 +159,20 @@ class TurnController:
if ack_error is not None:
return ack_error
self._event(QUESTION_PROMPT, PipelineState.SPEECH_DETECTING, "请说出问题", turn_id=turn_id)
user_segment = self._capture_segment(turn_id, state_message="录音中:正在听取问题")
return self._capture_and_transcribe(turn_id, state_message="录音中:正在听取问题")
def _capture_and_transcribe(
self,
turn_id: int,
*,
state_message: str,
no_speech_timeout_ms: int | None = None,
) -> str | ProviderError:
user_segment = self._capture_segment(
turn_id,
state_message=state_message,
no_speech_timeout_ms=no_speech_timeout_ms,
)
if isinstance(user_segment, ProviderError):
return user_segment
self._event(STT_STARTED, PipelineState.TRANSCRIBING, "转写中:正在识别问题", turn_id=turn_id)
@@ -158,40 +201,73 @@ class TurnController:
self.wakeword.reset()
return None
def _capture_segment(self, turn_id: int, *, state_message: str) -> AudioSegment | ProviderError:
self.vad_recorder.reset()
self.vad_recorder.provider.reset()
self.audio_preprocessor.reset()
realtime_session = self._start_realtime_transcript()
self._event(CAPTURE_STARTED, PipelineState.RECORDING, state_message, turn_id=turn_id)
while True:
frames = self.transport.read_frames(timeout_ms=100)
if not frames:
continue
for frame in frames:
try:
frame = self.audio_preprocessor.process_frame(frame)
except ProviderError as exc:
return exc
was_started = self.vad_recorder.started
result = self.vad_recorder.feed(frame)
if not was_started and self.vad_recorder.started:
self._event(SPEECH_STARTED, PipelineState.RECORDING, "检测到用户语音", turn_id=turn_id)
if isinstance(result, ProviderError):
return result
if self.vad_recorder.started and realtime_session is not None:
self._emit_realtime_transcript(realtime_session, frame, turn_id)
if isinstance(result, AudioSegment):
if realtime_session is not None:
self._finish_realtime_transcript(realtime_session, turn_id)
self._event(
SPEECH_ENDED,
PipelineState.RECORDING,
"用户语音结束",
turn_id=turn_id,
payload={"end_reason": result.metadata.get("end_reason", "")},
)
return result
def _capture_segment(
self,
turn_id: int,
*,
state_message: str,
no_speech_timeout_ms: int | None = None,
) -> AudioSegment | ProviderError:
original_no_speech_timeout_ms = self.vad_recorder.no_speech_timeout_ms
if no_speech_timeout_ms is not None:
self.vad_recorder.no_speech_timeout_ms = no_speech_timeout_ms
try:
self.vad_recorder.reset()
self.vad_recorder.provider.reset()
self.audio_preprocessor.reset()
realtime_session = self._start_realtime_transcript()
last_partial_ms: int | None = None
self._event(CAPTURE_STARTED, PipelineState.RECORDING, state_message, turn_id=turn_id)
while True:
frames = self._read_capture_frames(timeout_ms=100)
if not frames:
continue
for frame in frames:
try:
frame = self.audio_preprocessor.process_frame(frame)
except ProviderError as exc:
return exc
was_started = self.vad_recorder.started
result = self.vad_recorder.feed(frame)
if not was_started and self.vad_recorder.started:
self._event(SPEECH_STARTED, PipelineState.RECORDING, "检测到用户语音", turn_id=turn_id)
if isinstance(result, ProviderError):
return result
if self.vad_recorder.started and realtime_session is not None:
if self._emit_realtime_transcript(realtime_session, frame, turn_id):
last_partial_ms = frame.timestamp_ms
if isinstance(result, AudioSegment):
if realtime_session is not None:
self._finish_realtime_transcript(realtime_session, turn_id)
self._event(
SPEECH_ENDED,
PipelineState.RECORDING,
"用户语音结束",
turn_id=turn_id,
payload={"end_reason": result.metadata.get("end_reason", "")},
)
return result
if self._should_end_after_realtime_idle(last_partial_ms, frame.timestamp_ms):
if realtime_session is not None:
self._finish_realtime_transcript(realtime_session, turn_id)
result = self.vad_recorder.finish("partial_transcript_idle")
self._event(
SPEECH_ENDED,
PipelineState.RECORDING,
"用户语音结束",
turn_id=turn_id,
payload={"end_reason": result.metadata.get("end_reason", "")},
)
return result
finally:
self.vad_recorder.no_speech_timeout_ms = original_no_speech_timeout_ms
def _read_capture_frames(self, *, timeout_ms: int) -> list[AudioFrame]:
if self._pending_capture_frames:
frames = list(self._pending_capture_frames)
self._pending_capture_frames.clear()
return frames
return self.transport.read_frames(timeout_ms=timeout_ms)
def _start_realtime_transcript(self) -> RealtimeTranscriptSession | None:
if not self.config.realtime_transcript_enabled or self.realtime_stt is None:
@@ -203,14 +279,19 @@ class TurnController:
realtime_session: RealtimeTranscriptSession,
frame: AudioFrame,
turn_id: int,
) -> None:
) -> bool:
transcript = realtime_session.accept_frame(frame)
if transcript is None:
return
return False
text = transcript.normalized_text
if not is_valid_transcript_text(text):
return
return False
self._event(TRANSCRIPT_PARTIAL, PipelineState.RECORDING, "", turn_id=turn_id, payload={"text": text})
return True
def _should_end_after_realtime_idle(self, last_partial_ms: int | None, current_ms: int) -> bool:
timeout_ms = self.config.realtime_transcript_idle_timeout_ms
return timeout_ms > 0 and last_partial_ms is not None and current_ms - last_partial_ms >= timeout_ms
def _finish_realtime_transcript(
self,
@@ -227,11 +308,10 @@ class TurnController:
def _acknowledge_wake(self, turn_id: int) -> ProviderError | None:
text = self.config.wake_ack_text.strip()
if not text:
self._drain_input_after_playback()
return None
try:
self._event(ACK_STARTED, PipelineState.SPEAKING, f"应答中:{text}", turn_id=turn_id)
segment = self.ack_tts.synthesize(text)
segment = self._ack_segment(text)
playback = self.transport.play_pcm(segment)
if playback.error:
return playback.error
@@ -240,19 +320,92 @@ class TurnController:
except ProviderError as exc:
return exc
def _ack_segment(self, text: str) -> AudioSegment:
if self._cached_ack_text != text or self._cached_ack_segment is None:
self._cached_ack_text = text
self._cached_ack_segment = self.ack_tts.synthesize(text)
return self._cached_ack_segment
def _reply_to_user(self, user_text: str, turn_id: int) -> TurnResult:
completed_turns = 0
current_user_text = user_text
current_turn_id = turn_id
last_assistant_text = ""
while True:
reply_result = self._reply_once(current_user_text, current_turn_id, completed_turns=completed_turns)
completed_turns += reply_result.completed_turns
if not reply_result.success:
return reply_result
last_assistant_text = reply_result.assistant_text
if reply_result.error is not None:
return reply_result
if reply_result.states and reply_result.states[-1] == PipelineState.INTERRUPTED:
followup = self._listen_for_followup(current_turn_id + 1, interrupted=True)
else:
decision = self._decide_continuation(current_user_text, last_assistant_text, current_turn_id)
if not decision.should_continue:
self._event(
CONTINUOUS_SESSION_ENDED,
PipelineState.WAKE_LISTENING,
"",
turn_id=current_turn_id,
payload={"decision": decision.action, "reason": decision.reason},
)
self._event(
STANDBY_RESUMED,
PipelineState.WAKE_LISTENING,
"恢复待机:可继续唤醒",
turn_id=current_turn_id,
)
return TurnResult(
True,
current_user_text,
last_assistant_text,
states=list(self._states),
completed_turns=completed_turns,
)
followup = self._listen_for_followup(current_turn_id + 1, interrupted=False)
if followup is None:
return TurnResult(
True,
current_user_text,
last_assistant_text,
states=list(self._states),
completed_turns=completed_turns,
)
if isinstance(followup, ProviderError):
return self._recover(followup, current_turn_id + 1, completed_turns=completed_turns)
current_turn_id += 1
current_user_text = followup
def _reply_once(self, user_text: str, turn_id: int, *, completed_turns: int) -> TurnResult:
self.context.append_user(user_text)
self._event(LLM_STARTED, PipelineState.THINKING, "思考中:正在生成回复", turn_id=turn_id)
assistant_text = ""
spoken_parts: list[str] = []
interrupted = False
try:
for delta in self.llm.stream_reply(self.context.build_llm_messages()):
assistant_text += delta.text_delta
for sentence in self.sentence_buffer.feed(delta.text_delta, bool(delta.finish_reason)):
self._speak(sentence, turn_id)
for sentence in self.sentence_buffer.flush():
self._speak(sentence, turn_id)
speak_result = self._speak(sentence, turn_id)
if speak_result.interrupted:
interrupted = True
break
spoken_parts.append(speak_result.spoken_text)
if interrupted:
break
if interrupted:
self.sentence_buffer.flush()
else:
for sentence in self.sentence_buffer.flush():
speak_result = self._speak(sentence, turn_id)
if speak_result.interrupted:
interrupted = True
break
spoken_parts.append(speak_result.spoken_text)
except ProviderError as exc:
return self._recover(exc, turn_id)
return self._recover(exc, turn_id, completed_turns=completed_turns)
if not assistant_text.strip():
return self._recover(
ProviderError(
@@ -263,24 +416,177 @@ class TurnController:
"llm",
),
turn_id,
completed_turns=completed_turns,
)
self.context.append_assistant(assistant_text)
self._event(STANDBY_RESUMED, PipelineState.WAKE_LISTENING, "恢复待机:可继续唤醒", turn_id=turn_id)
return TurnResult(True, user_text, assistant_text, states=list(self._states))
spoken_text = "".join(spoken_parts)
if not spoken_text.strip() and not interrupted:
return self._recover(
ProviderError(
ErrorCode.TTS_EMPTY_AUDIO,
"LLM reply contained no speakable text after TTS sanitization",
True,
"voice-assistant-pipeline",
"tts",
),
turn_id,
completed_turns=completed_turns,
)
if spoken_text.strip():
self.context.append_assistant(spoken_text)
states = list(self._states)
if interrupted:
states.append(PipelineState.INTERRUPTED)
return TurnResult(
True,
user_text,
spoken_text,
states=states,
completed_turns=1,
)
def _speak(self, sentence: str, turn_id: int) -> None:
def _decide_continuation(self, user_text: str, assistant_text: str, turn_id: int) -> ContinuationDecision:
if not self.config.continuous_dialog_enabled:
return ContinuationDecision("standby", 1.0, "continuous dialog disabled", "config")
self._event(CONTINUATION_DECISION_STARTED, PipelineState.THINKING, "", turn_id=turn_id)
decision = self.continuation_decider.decide(
user_text=user_text,
assistant_text=assistant_text,
history=self.context.messages(),
)
self._event(
CONTINUATION_DECISION_MADE,
PipelineState.THINKING,
"",
turn_id=turn_id,
payload={
"action": decision.action,
"confidence": decision.confidence,
"reason": decision.reason,
"provider": decision.provider,
},
)
return decision
def _listen_for_followup(self, turn_id: int, *, interrupted: bool) -> str | ProviderError | None:
if not interrupted:
seconds = max(1, round(self.config.followup_listen_timeout_ms / 1000))
self._event(
FOLLOWUP_LISTENING,
PipelineState.RECORDING,
f"继续对话:{seconds}秒内可直接回答",
turn_id=turn_id,
)
user_text = self._capture_and_transcribe(
turn_id,
state_message="录音中:正在听取追问" if not interrupted else "录音中:正在听取打断内容",
no_speech_timeout_ms=self.config.followup_listen_timeout_ms,
)
if isinstance(user_text, ProviderError) and user_text.code == ErrorCode.VAD_TIMEOUT_NO_SPEECH:
self._event(
FOLLOWUP_TIMEOUT,
PipelineState.WAKE_LISTENING,
"追问超时:未检测到用户回答",
turn_id=turn_id,
)
self._event(CONTINUOUS_SESSION_ENDED, PipelineState.WAKE_LISTENING, "", turn_id=turn_id)
self._event(STANDBY_RESUMED, PipelineState.WAKE_LISTENING, "恢复待机:可继续唤醒", turn_id=turn_id)
return None
return user_text
def _speak(self, sentence: str, turn_id: int) -> SpeakResult:
spoken_sentence = sanitize_tts_text(sentence)
if not spoken_sentence:
return SpeakResult("")
self._event(TTS_STARTED, PipelineState.SPEAKING, "播放中:正在播报回复", turn_id=turn_id)
segment = self.tts.synthesize(sentence)
playback = self.transport.play_pcm(segment)
if playback.error:
raise playback.error
segment = self.tts.synthesize(spoken_sentence)
if not self._can_interrupt_playback(segment):
playback = self.transport.play_pcm(segment)
if playback.error:
raise playback.error
self._event(PLAYBACK_FINISHED, PipelineState.SPEAKING, "", turn_id=turn_id)
self._drain_input_after_playback()
return SpeakResult(spoken_sentence)
if self._play_interruptible(segment, turn_id=turn_id):
return SpeakResult("", interrupted=True)
self._event(PLAYBACK_FINISHED, PipelineState.SPEAKING, "", turn_id=turn_id)
self._drain_input_after_playback()
return SpeakResult(spoken_sentence)
def _can_interrupt_playback(self, segment: AudioSegment) -> bool:
return (
self.config.barge_in_enabled
and self.realtime_stt is not None
and segment.duration_ms > self.config.barge_in_echo_guard_ms
and not segment.metadata.get("format")
)
def _play_interruptible(self, segment: AudioSegment, *, turn_id: int) -> bool:
guard_cleared = False
self.vad_recorder.provider.reset()
realtime_session = self._start_realtime_transcript()
speech_ms = 0
partial_seen = False
pending_frames: list[AudioFrame] = []
interrupted = False
def after_chunk(_chunk: AudioSegment, elapsed_ms: int) -> bool:
nonlocal guard_cleared, speech_ms, partial_seen, interrupted
if elapsed_ms < self.config.barge_in_echo_guard_ms:
return False
if not guard_cleared:
self.transport.flush_input()
guard_cleared = True
return False
detected, speech_ms, partial_seen, new_frames = self._detect_barge_in(
realtime_session,
turn_id=turn_id,
speech_ms=speech_ms,
partial_seen=partial_seen,
)
pending_frames.extend(new_frames)
if detected:
self._pending_capture_frames.extend(pending_frames)
self._event(BARGE_IN_DETECTED, PipelineState.INTERRUPTED, "检测到用户打断", turn_id=turn_id)
self._event(PLAYBACK_INTERRUPTED, PipelineState.INTERRUPTED, "播报已打断", turn_id=turn_id)
interrupted = True
return True
return False
playback = self.transport.play_pcm_chunks(segment, chunk_ms=100, after_chunk=after_chunk)
if playback.error:
raise playback.error
if realtime_session is not None:
realtime_session.finish()
return interrupted
def _detect_barge_in(
self,
realtime_session: RealtimeTranscriptSession | None,
*,
turn_id: int,
speech_ms: int,
partial_seen: bool,
) -> tuple[bool, int, bool, list[AudioFrame]]:
frames = self.transport.read_frames(timeout_ms=0)
if not frames:
return False, speech_ms, partial_seen, []
for frame in frames:
result = self.vad_recorder.provider.analyze(frame)
if result.is_speech:
speech_ms += int(frame.metadata.get("duration_ms", 20))
if realtime_session is not None:
transcript = realtime_session.accept_frame(frame)
if transcript is not None and is_valid_transcript_text(transcript.normalized_text):
partial_seen = True
else:
speech_ms = 0
detected = speech_ms >= self.config.barge_in_min_speech_ms and partial_seen
return detected, speech_ms, partial_seen, frames
def _drain_input_after_playback(self) -> None:
self.transport.flush_input()
if self.config.post_playback_drain_ms <= 0:
return
self.transport.flush_input()
remaining_ms = self.config.post_playback_drain_ms
while remaining_ms > 0:
timeout_ms = min(50, remaining_ms)
@@ -288,11 +594,11 @@ class TurnController:
remaining_ms -= timeout_ms
self.transport.flush_input()
def _recover(self, error: ProviderError, turn_id: int) -> TurnResult:
def _recover(self, error: ProviderError, turn_id: int, *, completed_turns: int = 0) -> TurnResult:
self._event(STAGE_ERROR, PipelineState.ERROR_RECOVERING, error.message, turn_id=turn_id, payload={"error": error})
self._event(RECOVERING, PipelineState.ERROR_RECOVERING, "恢复待机:本轮已结束", turn_id=turn_id)
self._event(STANDBY_RESUMED, PipelineState.WAKE_LISTENING, "恢复待机:可继续唤醒", turn_id=turn_id)
return TurnResult(False, error=error, states=list(self._states))
return TurnResult(False, error=error, states=list(self._states), completed_turns=completed_turns, failed_turns=1)
def _event(
self,
@@ -324,6 +630,7 @@ class VoiceAssistantPipeline:
ack_tts: TtsProvider | None = None,
reporter: RuntimeReporter | None = None,
event_bus: PipelineEventBus | None = None,
continuation_decider: ContinuationDecisionProvider | None = None,
sentence_buffer: SentenceBuffer | None = None,
) -> None:
self.config = config
@@ -339,6 +646,11 @@ class VoiceAssistantPipeline:
self.context = context
self.reporter = reporter
self.event_bus = event_bus or PipelineEventBus()
self.continuation_decider = continuation_decider or build_continuation_decider(
config.continuation_decision_provider,
llm,
threshold=config.continuation_confidence_threshold,
)
if reporter is not None:
self.event_bus.subscribe(self._report_event)
self.sentence_buffer = sentence_buffer or SentenceBuffer()
@@ -355,6 +667,7 @@ class VoiceAssistantPipeline:
ack_tts=self.ack_tts,
context=context,
event_bus=self.event_bus,
continuation_decider=self.continuation_decider,
sentence_buffer=self.sentence_buffer,
)
@@ -368,6 +681,7 @@ class VoiceAssistantPipeline:
self.tts.load()
if self.ack_tts is not self.tts:
self.ack_tts.load()
self.controller.prepare_ack_audio()
def run(self, *, once: bool = False, max_turns: int | None = None) -> RuntimeSummary:
completed = 0
@@ -383,10 +697,11 @@ class VoiceAssistantPipeline:
while True:
turn_id = completed + failed + 1
result = self.run_turn(turn_id)
completed += result.completed_turns
if result.success:
completed += 1
completed += 0 if result.completed_turns else 1
else:
failed += 1
failed += result.failed_turns or 1
last_error = result.error
if once:
break
+34 -45
View File
@@ -4,6 +4,7 @@ import argparse
import json
import re
import subprocess
from dataclasses import replace
from pathlib import Path
from .audio_preprocess import SherpaOnnxDenoiserPreprocessor
@@ -13,6 +14,7 @@ from .conversation import ConversationContext
from .llm import MockLlmProvider, OpenAICompatibleLlmProvider
from .models import AudioFrame, ProviderError
from .pipeline import VoicePipeline
from .real_live_check import run_real_live_check
from .runtime import build_live_runtime
from .simulation import run_simulated_live
from .speech_models import check_speech_models, model_status_errors
@@ -40,6 +42,12 @@ def main(argv: list[str] | None = None) -> int:
simulate.add_argument("--turns", type=int, default=2, help="Number of simulated turns. Default: 2")
simulate.add_argument("--fixture", default=None, help="Replay simulated microphone frames from JSONL")
simulate.add_argument("--write-fixture", default=None, help="Write generated simulated microphone frames to JSONL")
real_check = subparsers.add_parser("real-live-check", help="Run generated-audio live check with real providers")
real_check.add_argument("--turns", type=int, default=2, help="Number of generated live turns. Default: 2")
real_check.add_argument("--voice", default="Tingting", help="macOS say voice used for generated microphone input")
real_check.add_argument("--wake-text", default="小杰小杰。", help="Generated wake utterance")
real_check.add_argument("--question", action="append", default=None, help="Generated user question; can be repeated")
real_check.add_argument("--no-playback", action="store_true", help="Synthesize but do not play generated TTS output")
smoke = subparsers.add_parser("llm-smoke", help="Call configured OpenAI/NewAPI endpoint")
smoke.add_argument("--message", default="用一句中文回复:小杰在线。")
smoke.add_argument("--no-stream", action="store_true")
@@ -58,6 +66,7 @@ def main(argv: list[str] | None = None) -> int:
"llm_api_style": config.llm_api_style,
"llm_stream": config.llm_stream,
"realtime_transcript_enabled": config.realtime_transcript_enabled,
"realtime_transcript_idle_timeout_ms": config.realtime_transcript_idle_timeout_ms,
"llm_api_key_present": bool(config.llm_api_key),
"asset_dir": str(config.asset_dir),
"wake_provider": config.wake_provider,
@@ -88,6 +97,13 @@ def main(argv: list[str] | None = None) -> int:
"tts_voice": config.tts_voice,
"speech_models_dir": str(config.speech_models_dir),
"context_mode": config.context_mode,
"continuous_dialog_enabled": config.continuous_dialog_enabled,
"continuation_decision_provider": config.continuation_decision_provider,
"continuation_confidence_threshold": config.continuation_confidence_threshold,
"followup_listen_timeout_ms": config.followup_listen_timeout_ms,
"barge_in_enabled": config.barge_in_enabled,
"barge_in_min_speech_ms": config.barge_in_min_speech_ms,
"barge_in_echo_guard_ms": config.barge_in_echo_guard_ms,
},
ensure_ascii=False,
sort_keys=True,
@@ -159,6 +175,23 @@ def main(argv: list[str] | None = None) -> int:
print(json.dumps(data, ensure_ascii=False, sort_keys=True))
return 0 if data["success"] else 1
if args.command == "real-live-check":
config = AppConfig.from_dotenv(args.env_file)
try:
data = run_real_live_check(
config=config,
turns=args.turns,
voice=args.voice,
wake_text=args.wake_text,
questions=args.question,
play_audio=not args.no_playback,
)
except (ProviderError, ValueError) as exc:
print(json.dumps({"success": False, "error": str(exc)}, ensure_ascii=False, sort_keys=True))
return 1
print(json.dumps(data, ensure_ascii=False, sort_keys=True))
return 0 if data["success"] else 1
if args.command == "acceptance":
result = run_acceptance()
print(json.dumps(result, ensure_ascii=False, sort_keys=True))
@@ -167,51 +200,7 @@ def main(argv: list[str] | None = None) -> int:
if args.command == "llm-smoke":
config = AppConfig.from_dotenv(args.env_file)
if args.no_stream:
config = AppConfig(
wake_word=config.wake_word,
sample_rate=config.sample_rate,
channels=config.channels,
llm_base_url=config.llm_base_url,
llm_api_key=config.llm_api_key,
llm_model=config.llm_model,
llm_api_style=config.llm_api_style,
llm_stream=False,
realtime_transcript_enabled=config.realtime_transcript_enabled,
audio_input_device=config.audio_input_device,
audio_output_device=config.audio_output_device,
asset_dir=config.asset_dir,
log_dir=config.log_dir,
wake_provider=config.wake_provider,
wake_keywords_file=config.wake_keywords_file,
wake_kws_threshold=config.wake_kws_threshold,
wake_kws_score=config.wake_kws_score,
wake_ack_text=config.wake_ack_text,
post_playback_drain_ms=config.post_playback_drain_ms,
pipeline_mode=config.pipeline_mode,
endpoint_mode=config.endpoint_mode,
noise_filter_enabled=config.noise_filter_enabled,
noise_filter_provider=config.noise_filter_provider,
wake_denoise_enabled=config.wake_denoise_enabled,
speaker_profile_ms=config.speaker_profile_ms,
speaker_profile_min_ms=config.speaker_profile_min_ms,
speaker_absent_ms=config.speaker_absent_ms,
speaker_similarity_threshold=config.speaker_similarity_threshold,
speaker_min_rms=config.speaker_min_rms,
vad_provider=config.vad_provider,
vad_threshold=config.vad_threshold,
vad_min_duration_ms=config.vad_min_duration_ms,
vad_end_silence_ms=config.vad_end_silence_ms,
vad_no_speech_timeout_ms=config.vad_no_speech_timeout_ms,
vad_max_recording_ms=config.vad_max_recording_ms,
speech_provider=config.speech_provider,
asr_model=config.asr_model,
tts_model=config.tts_model,
tts_voice=config.tts_voice,
speech_models_dir=config.speech_models_dir,
context_mode=config.context_mode,
context_max_messages=config.context_max_messages,
context_max_chars=config.context_max_chars,
)
config = replace(config, llm_stream=False)
try:
provider = OpenAICompatibleLlmProvider(config, timeout_s=30)
messages = [ConversationContext().build_llm_messages()[0]]
+68
View File
@@ -17,6 +17,7 @@ class AppConfig:
llm_api_style: str = "chat_completions"
llm_stream: bool = True
realtime_transcript_enabled: bool = True
realtime_transcript_idle_timeout_ms: int = 1500
audio_input_device: str | None = None
audio_output_device: str | None = None
asset_dir: Path = Path("assets/pet")
@@ -51,6 +52,13 @@ class AppConfig:
context_mode: str = "session_memory"
context_max_messages: int = 12
context_max_chars: int = 12000
continuous_dialog_enabled: bool = True
continuation_decision_provider: str = "hybrid"
continuation_confidence_threshold: float = 0.65
followup_listen_timeout_ms: int = 3000
barge_in_enabled: bool = True
barge_in_min_speech_ms: int = 250
barge_in_echo_guard_ms: int = 500
@classmethod
def from_dotenv(cls, path: str | Path = ".env", prefix: str = "OWNER_") -> "AppConfig":
@@ -71,6 +79,9 @@ class AppConfig:
llm_stream=(get("LLM_STREAM", "1") or "1").lower() not in {"0", "false", "no"},
realtime_transcript_enabled=(get("REALTIME_TRANSCRIPT_ENABLED", "1") or "1").lower()
not in {"0", "false", "no"},
realtime_transcript_idle_timeout_ms=int(
get("REALTIME_TRANSCRIPT_IDLE_TIMEOUT_MS", "1500") or "1500"
),
audio_input_device=get("AUDIO_INPUT_DEVICE"),
audio_output_device=get("AUDIO_OUTPUT_DEVICE"),
asset_dir=Path(get("ASSET_DIR", "assets/pet") or "assets/pet"),
@@ -107,6 +118,18 @@ class AppConfig:
context_mode=(get("CONTEXT_MODE", "session_memory") or "session_memory").lower(),
context_max_messages=int(get("CONTEXT_MAX_MESSAGES", "12") or "12"),
context_max_chars=int(get("CONTEXT_MAX_CHARS", "12000") or "12000"),
continuous_dialog_enabled=(get("CONTINUOUS_DIALOG_ENABLED", "1") or "1").lower()
not in {"0", "false", "no"},
continuation_decision_provider=(
get("CONTINUATION_DECISION_PROVIDER", "hybrid") or "hybrid"
).lower(),
continuation_confidence_threshold=float(
get("CONTINUATION_CONFIDENCE_THRESHOLD", "0.65") or "0.65"
),
followup_listen_timeout_ms=int(get("FOLLOWUP_LISTEN_TIMEOUT_MS", "3000") or "3000"),
barge_in_enabled=(get("BARGE_IN_ENABLED", "1") or "1").lower() not in {"0", "false", "no"},
barge_in_min_speech_ms=int(get("BARGE_IN_MIN_SPEECH_MS", "250") or "250"),
barge_in_echo_guard_ms=int(get("BARGE_IN_ECHO_GUARD_MS", "500") or "500"),
)
@classmethod
@@ -165,6 +188,16 @@ class AppConfig:
"startup",
)
)
if self.realtime_transcript_idle_timeout_ms < 0:
errors.append(
ProviderError(
ErrorCode.CONFIG_MISSING_VALUE,
"OWNER_REALTIME_TRANSCRIPT_IDLE_TIMEOUT_MS must be non-negative",
False,
"config",
"startup",
)
)
if self.wake_provider not in {"local_kws"}:
errors.append(
ProviderError(
@@ -326,6 +359,41 @@ class AppConfig:
"startup",
)
)
if self.continuation_decision_provider not in {"hybrid", "rule", "llm"}:
errors.append(
ProviderError(
ErrorCode.CONFIG_MISSING_VALUE,
"OWNER_CONTINUATION_DECISION_PROVIDER must be hybrid, rule, or llm",
False,
"config",
"startup",
)
)
if not 0 < self.continuation_confidence_threshold <= 1:
errors.append(
ProviderError(
ErrorCode.CONFIG_MISSING_VALUE,
"OWNER_CONTINUATION_CONFIDENCE_THRESHOLD must be in (0, 1]",
False,
"config",
"startup",
)
)
for name, value in {
"OWNER_FOLLOWUP_LISTEN_TIMEOUT_MS": self.followup_listen_timeout_ms,
"OWNER_BARGE_IN_MIN_SPEECH_MS": self.barge_in_min_speech_ms,
"OWNER_BARGE_IN_ECHO_GUARD_MS": self.barge_in_echo_guard_ms,
}.items():
if value < 0:
errors.append(
ProviderError(
ErrorCode.CONFIG_MISSING_VALUE,
f"{name} must be non-negative",
False,
"config",
"startup",
)
)
return errors
def api_url(self, path: str) -> str:
+211
View File
@@ -0,0 +1,211 @@
from __future__ import annotations
import json
import re
from collections.abc import Sequence
from dataclasses import dataclass
from typing import Protocol
from .models import Message
from .protocols import LlmProvider
@dataclass(frozen=True, slots=True)
class ContinuationDecision:
action: str
confidence: float
reason: str
provider: str
@property
def should_continue(self) -> bool:
return self.action == "continue"
class ContinuationDecisionProvider(Protocol):
def decide(
self,
*,
user_text: str,
assistant_text: str,
history: Sequence[Message],
) -> ContinuationDecision:
...
class RuleContinuationDecisionProvider:
_continue_patterns = (
"可以告诉我",
"请告诉我",
"请选择",
"选一个",
"请补充",
"补充一下",
"继续吗",
)
_ambiguous_continue_patterns = (
"我还可以",
"我可以继续",
"还可以继续",
"可以继续",
"继续展开",
"继续讲",
"继续聊",
"两个方向",
"三个方向",
"几个方向",
"几个部分",
)
_standby_patterns = (
"这是",
"已经",
"完成",
"好了",
"好的",
"没问题",
"不客气",
"无法",
"不能",
"抱歉",
"出错",
"失败",
"先这样",
"到这里",
"随时叫我",
"有需要再叫我",
"需要时再叫我",
)
def decide(
self,
*,
user_text: str,
assistant_text: str,
history: Sequence[Message],
) -> ContinuationDecision:
text = _normalize(assistant_text)
if not text:
return ContinuationDecision("standby", 1.0, "empty assistant reply", "rule")
if _has_direct_question(text) or any(pattern in text for pattern in self._continue_patterns):
return ContinuationDecision("continue", 0.86, "assistant asks for user input", "rule")
if any(pattern in text for pattern in self._standby_patterns):
return ContinuationDecision("standby", 0.82, "assistant appears to finish the answer", "rule")
if len(text) <= 18 and not text.endswith(("?", "", "", "")):
return ContinuationDecision("standby", 0.72, "short non-question reply", "rule")
if any(pattern in text for pattern in self._ambiguous_continue_patterns):
return ContinuationDecision("unknown", 0.0, "ambiguous continuation offer", "rule")
return ContinuationDecision("standby", 0.78, "assistant did not ask for immediate input", "rule")
class LlmContinuationDecisionProvider:
def __init__(self, llm: LlmProvider, *, threshold: float = 0.65) -> None:
self.llm = llm
self.threshold = threshold
def decide(
self,
*,
user_text: str,
assistant_text: str,
history: Sequence[Message],
) -> ContinuationDecision:
prompt = (
"你是语音助手的持续对话分类器。判断助手刚才的回复是否需要用户继续直接回答。\n"
"只允许输出 JSON: {\"action\":\"continue|standby\",\"confidence\":0到1,\"reason\":\"短原因\"}。\n"
"如果不确定,action 必须是 standby。\n"
f"用户上一句: {user_text}\n"
f"助手回复: {assistant_text}\n"
)
messages = [
Message("system", "你只做二分类,不生成正常对话回复。", 0.0),
Message("user", prompt, 0.0),
]
try:
raw = "".join(delta.text_delta for delta in self.llm.stream_reply(messages)).strip()
except Exception as exc:
return ContinuationDecision("standby", 0.0, f"classifier failed: {exc}", "llm")
parsed = _parse_classifier_output(raw)
if parsed is None:
return ContinuationDecision("standby", 0.0, "classifier output malformed", "llm")
action, confidence, reason = parsed
if action not in {"continue", "standby"}:
return ContinuationDecision("standby", 0.0, "classifier action invalid", "llm")
if confidence < self.threshold:
return ContinuationDecision("standby", confidence, "classifier confidence below threshold", "llm")
return ContinuationDecision(action, confidence, reason or "classifier decision", "llm")
class HybridContinuationDecisionProvider:
def __init__(
self,
rule_provider: RuleContinuationDecisionProvider,
llm_provider: LlmContinuationDecisionProvider | None,
*,
threshold: float = 0.65,
) -> None:
self.rule_provider = rule_provider
self.llm_provider = llm_provider
self.threshold = threshold
def decide(
self,
*,
user_text: str,
assistant_text: str,
history: Sequence[Message],
) -> ContinuationDecision:
rule = self.rule_provider.decide(user_text=user_text, assistant_text=assistant_text, history=history)
if rule.action in {"continue", "standby"} and rule.confidence >= self.threshold:
return rule
if self.llm_provider is None:
return ContinuationDecision("standby", rule.confidence, "rule uncertain and no llm fallback", "hybrid")
llm = self.llm_provider.decide(user_text=user_text, assistant_text=assistant_text, history=history)
if llm.action == "continue" and llm.confidence >= self.threshold:
return ContinuationDecision("continue", llm.confidence, llm.reason, "hybrid")
return ContinuationDecision("standby", llm.confidence, llm.reason, "hybrid")
def build_continuation_decider(
provider_name: str,
llm: LlmProvider,
*,
threshold: float,
) -> ContinuationDecisionProvider:
rule = RuleContinuationDecisionProvider()
if provider_name == "rule":
return rule
llm_provider = LlmContinuationDecisionProvider(llm, threshold=threshold)
if provider_name == "llm":
return llm_provider
return HybridContinuationDecisionProvider(rule, llm_provider, threshold=threshold)
def _normalize(text: str) -> str:
return re.sub(r"\s+", "", text.strip())
def _has_direct_question(text: str) -> bool:
if text.endswith(("?", "")):
return True
return bool(re.search(r"(你|您).{0,12}(吗|呢|么|什么|哪|是否|要不要|需要不需要)", text))
def _parse_classifier_output(raw: str) -> tuple[str, float, str] | None:
clean = raw.strip()
if not clean:
return None
try:
data = json.loads(clean)
action = str(data.get("action", "")).strip().lower()
confidence = float(data.get("confidence", 1.0 if action in {"continue", "standby"} else 0.0))
reason = str(data.get("reason", "")).strip()
return action, max(0.0, min(1.0, confidence)), reason
except (ValueError, TypeError, json.JSONDecodeError):
lowered = clean.lower()
if lowered in {"continue", "standby"}:
return lowered, 1.0, "plain classifier output"
if "continue" in lowered and "standby" not in lowered:
return "continue", 1.0, "plain classifier output"
if "standby" in lowered and "continue" not in lowered:
return "standby", 1.0, "plain classifier output"
return None
+1 -1
View File
@@ -9,7 +9,7 @@ from .models import Message
@dataclass(slots=True)
class ConversationContext:
system_prompt: str = "你是一个中文桌宠助手,回答要简洁、自然、适合语音播报。"
system_prompt: str = "你是一个中文桌宠助手,回答要简洁、自然、适合语音播报。不要输出 emoji、表情包、Markdown 图片。"
max_messages: int = 12
max_chars: int = 12000
_messages: list[Message] = field(default_factory=list)
+7
View File
@@ -21,6 +21,13 @@ LLM_STARTED = "llm_started"
TTS_STARTED = "tts_started"
PLAYBACK_FINISHED = "playback_finished"
STANDBY_RESUMED = "standby_resumed"
CONTINUATION_DECISION_STARTED = "continuation_decision_started"
CONTINUATION_DECISION_MADE = "continuation_decision_made"
FOLLOWUP_LISTENING = "followup_listening"
FOLLOWUP_TIMEOUT = "followup_timeout"
BARGE_IN_DETECTED = "barge_in_detected"
PLAYBACK_INTERRUPTED = "playback_interrupted"
CONTINUOUS_SESSION_ENDED = "continuous_session_ended"
STAGE_ERROR = "stage_error"
RECOVERING = "recovering"
+25 -5
View File
@@ -5,7 +5,7 @@ from dataclasses import dataclass, field
from .conversation import ConversationContext
from .models import AudioSegment, ErrorCode, PipelineState, ProviderError
from .protocols import AudioTransport, LlmProvider, SttProvider, TtsProvider, WakeWordProvider
from .tts import SentenceBuffer
from .tts import SentenceBuffer, sanitize_tts_text
from .vad import VadRecorder
@@ -98,27 +98,36 @@ class VoicePipeline:
self.context.append_user(text)
self._state(PipelineState.THINKING)
assistant_text = ""
spoken_parts: list[str] = []
played = 0
try:
for delta in self.llm.stream_reply(self.context.build_llm_messages()):
assistant_text += delta.text_delta
for sentence in self.sentence_buffer.feed(delta.text_delta, bool(delta.finish_reason)):
spoken_sentence = sanitize_tts_text(sentence)
if not spoken_sentence:
continue
self._state(PipelineState.SPEAKING)
self.suppress_input = True
audio = self.tts.synthesize(sentence)
audio = self.tts.synthesize(spoken_sentence)
playback = self.transport.play_pcm(audio)
self.suppress_input = False
if playback.error:
return self._recover(playback.error)
spoken_parts.append(spoken_sentence)
played += 1
for sentence in self.sentence_buffer.flush():
spoken_sentence = sanitize_tts_text(sentence)
if not spoken_sentence:
continue
self._state(PipelineState.SPEAKING)
self.suppress_input = True
audio = self.tts.synthesize(sentence)
audio = self.tts.synthesize(spoken_sentence)
playback = self.transport.play_pcm(audio)
self.suppress_input = False
if playback.error:
return self._recover(playback.error)
spoken_parts.append(spoken_sentence)
played += 1
except ProviderError as exc:
self.suppress_input = False
@@ -133,9 +142,20 @@ class VoicePipeline:
"llm",
)
)
self.context.append_assistant(assistant_text)
spoken_text = "".join(spoken_parts)
if not spoken_text.strip():
return self._recover(
ProviderError(
ErrorCode.TTS_EMPTY_AUDIO,
"LLM reply contained no speakable text after TTS sanitization",
True,
"voice-pipeline",
"tts",
)
)
self.context.append_assistant(spoken_text)
self._state(PipelineState.WAKE_LISTENING)
return PipelineResult(True, text, assistant_text, list(self.states), played_segments=played)
return PipelineResult(True, text, spoken_text, list(self.states), played_segments=played)
def _recover(self, error: ProviderError) -> PipelineResult:
self._state(PipelineState.ERROR_RECOVERING)
+10 -1
View File
@@ -1,6 +1,6 @@
from __future__ import annotations
from collections.abc import Iterable, Sequence
from collections.abc import Callable, Iterable, Sequence
from typing import Protocol
from .models import (
@@ -28,6 +28,15 @@ class AudioTransport(Protocol):
def play_pcm(self, segment: AudioSegment, interrupt: bool = False) -> PlaybackResult:
...
def play_pcm_chunks(
self,
segment: AudioSegment,
*,
chunk_ms: int,
after_chunk: Callable[[AudioSegment, int], bool] | None = None,
) -> PlaybackResult:
...
def flush_input(self) -> int:
...
+534
View File
@@ -0,0 +1,534 @@
from __future__ import annotations
import time
from contextlib import contextmanager
from datetime import datetime
import subprocess
import tempfile
import wave
from collections import Counter, deque
from collections.abc import Iterable, Sequence
from dataclasses import dataclass, field, replace
from pathlib import Path
from .assistant_pipeline import VoiceAssistantPipeline
from .audio_preprocess import SherpaOnnxDenoiserPreprocessor
from .config import AppConfig
from .conversation import ConversationContext
from .events import PipelineEventBus
from .llm import OpenAICompatibleLlmProvider
from .models import AudioFrame, AudioSegment, Message, PlaybackResult, ReplyDelta, TransportHealth
from .stt import SherpaOnnxSttProvider
from .transport import SoundDeviceAudioTransport
from .tts import MacSayTtsProvider
from .vad import EnergyVadProvider, HybridVadProvider, PrimarySpeakerVadRecorder, SherpaOnnxVadProvider
from .wakeword import SherpaOnnxKeywordWakeWordProvider
FRAME_MS = 20
SAMPLE_RATE = 16000
FRAME_BYTES = int(SAMPLE_RATE * FRAME_MS / 1000) * 2
DEFAULT_WAKE_TEXT = "小杰小杰。"
DEFAULT_QUESTIONS = [
"我叫阿明,请你记住我的名字。",
"我叫什么名字?",
]
STAGE_PAIRS = [
("wake_wait", "wake_listening", "wake_detected"),
("acknowledgement", "ack_started", "question_prompt"),
("wait_for_speech", "capture_started", "speech_started"),
("capture", "speech_started", "speech_ended"),
("final_stt", "stt_started", "transcript_final"),
("llm_to_first_tts", "llm_started", "tts_started"),
("turn_total", "wake_listening", "standby_resumed"),
]
def _now_iso() -> str:
return datetime.now().astimezone().isoformat(timespec="milliseconds")
def _epoch_ms() -> int:
return int(time.time() * 1000)
def _elapsed_ms(start: float, end: float | None = None) -> int:
return int(round(((end if end is not None else time.perf_counter()) - start) * 1000))
class TimingRecorder:
def __init__(self) -> None:
self.started_at = _now_iso()
self.started_epoch_ms = _epoch_ms()
self._started_mono = time.perf_counter()
self.finished_at: str | None = None
self.finished_epoch_ms: int | None = None
self.duration_ms: int | None = None
self.phases: list[dict[str, object]] = []
self.events: list[dict[str, object]] = []
@contextmanager
def phase(self, name: str) -> Iterable[None]:
started_mono = time.perf_counter()
record: dict[str, object] = {
"name": name,
"started_at": _now_iso(),
"start_offset_ms": _elapsed_ms(self._started_mono, started_mono),
}
try:
yield
finally:
finished_mono = time.perf_counter()
record["finished_at"] = _now_iso()
record["duration_ms"] = _elapsed_ms(started_mono, finished_mono)
self.phases.append(record)
def record_event(self, event_type: str, *, turn_id: int | None, message: str) -> None:
self.events.append(
{
"type": event_type,
"turn": turn_id,
"at": _now_iso(),
"offset_ms": _elapsed_ms(self._started_mono),
"message": message,
}
)
def finish(self) -> None:
self.finished_at = _now_iso()
self.finished_epoch_ms = _epoch_ms()
self.duration_ms = _elapsed_ms(self._started_mono)
def to_json(self) -> dict[str, object]:
if self.duration_ms is None:
self.finish()
return {
"started_at": self.started_at,
"started_epoch_ms": self.started_epoch_ms,
"finished_at": self.finished_at,
"finished_epoch_ms": self.finished_epoch_ms,
"duration_ms": self.duration_ms,
"phases": self.phases,
"events": self.events,
"stage_timings": _stage_timings_from_events(self.events),
}
@dataclass(slots=True)
class RealLiveCheckReporter:
statuses: list[dict[str, object]] = field(default_factory=list)
partials: list[dict[str, object]] = field(default_factory=list)
finals: list[dict[str, object]] = field(default_factory=list)
errors: list[dict[str, object]] = field(default_factory=list)
def status(self, state: str, message: str, *, turn_id: int | None = None) -> None:
self.statuses.append({"turn": turn_id, "state": state, "message": message})
def transcript(self, text: str, *, final: bool, turn_id: int | None = None) -> None:
target = self.finals if final else self.partials
target.append({"turn": turn_id, "text": text})
def error(self, stage: str, code: str, message: str, *, turn_id: int | None = None) -> None:
self.errors.append({"turn": turn_id, "stage": stage, "code": code, "message": message})
class RecordingLlmProvider:
def __init__(self, delegate: OpenAICompatibleLlmProvider, timing: TimingRecorder) -> None:
self.delegate = delegate
self.timing = timing
self.calls: list[list[Message]] = []
self.request_timings: list[dict[str, object]] = []
def stream_reply(self, messages: Sequence[Message]) -> Iterable[ReplyDelta]:
self.calls.append(list(messages))
sent_mono = time.perf_counter()
request_timing: dict[str, object] = {
"call_index": len(self.calls),
"sent_at": _now_iso(),
"sent_offset_ms": _elapsed_ms(self.timing._started_mono, sent_mono),
"message_count": len(messages),
"last_user_preview": _last_user_preview(messages),
}
first_delta_mono: float | None = None
try:
for delta in self.delegate.stream_reply(messages):
if first_delta_mono is None and delta.text_delta:
first_delta_mono = time.perf_counter()
request_timing["first_delta_at"] = _now_iso()
request_timing["first_delta_ms"] = _elapsed_ms(sent_mono, first_delta_mono)
yield delta
finally:
finished_mono = time.perf_counter()
request_timing["finished_at"] = _now_iso()
request_timing["duration_ms"] = _elapsed_ms(sent_mono, finished_mono)
if first_delta_mono is None:
request_timing["first_delta_ms"] = None
self.request_timings.append(request_timing)
class FixtureLiveAudioTransport:
"""Generated microphone input with optional real speaker output."""
def __init__(
self,
frames: list[AudioFrame],
*,
play_audio: bool = True,
output_transport: SoundDeviceAudioTransport | None = None,
) -> None:
self._frames: deque[AudioFrame] = deque(frames)
self.play_audio = play_audio
self.output_transport = output_transport or SoundDeviceAudioTransport()
self.played_segments: list[AudioSegment] = []
self.flush_count = 0
self.started = False
def start_input(self, device_id: str | None = None, sample_rate: int = 16000, channels: int = 1) -> None:
self.started = True
def read_frames(self, timeout_ms: int) -> list[AudioFrame]:
if not self.started or not self._frames:
return []
return [self._frames.popleft()]
def play_pcm(self, segment: AudioSegment, interrupt: bool = False) -> PlaybackResult:
if interrupt:
self.played_segments.clear()
self.played_segments.append(segment)
if not self.play_audio:
return PlaybackResult(True, segment.duration_ms)
return self.output_transport.play_pcm(segment, interrupt=interrupt)
def flush_input(self) -> int:
self.flush_count += 1
return 0
def stop(self) -> None:
self.started = False
def health(self) -> TransportHealth:
output = self.output_transport.health()
return TransportHealth(True, output.output_available if self.play_audio else True, "fixture input")
def run_real_live_check(
*,
config: AppConfig,
turns: int = 2,
voice: str = "Tingting",
wake_text: str = DEFAULT_WAKE_TEXT,
questions: list[str] | None = None,
play_audio: bool = True,
) -> dict[str, object]:
timing = TimingRecorder()
if turns <= 0:
raise ValueError("turns must be positive")
with timing.phase("prepare_config"):
selected_questions = _question_list(turns, questions)
config = replace(
config,
speech_provider="local",
noise_filter_enabled=True,
realtime_transcript_enabled=True,
continuous_dialog_enabled=False,
barge_in_enabled=False,
llm_stream=False,
post_playback_drain_ms=0,
)
config.require_llm_credentials()
with timing.phase("generate_fixture_audio"):
frames = _generated_live_frames(config, turns=turns, voice=voice, wake_text=wake_text, questions=selected_questions)
with timing.phase("build_pipeline"):
reporter = RealLiveCheckReporter()
event_bus = PipelineEventBus()
event_bus.subscribe(lambda event: timing.record_event(event.type, turn_id=event.turn_id, message=event.message))
stt = SherpaOnnxSttProvider(str(config.speech_models_dir))
tts = MacSayTtsProvider()
llm = RecordingLlmProvider(OpenAICompatibleLlmProvider(config, timeout_s=60), timing)
transport = FixtureLiveAudioTransport(
frames,
play_audio=play_audio,
output_transport=SoundDeviceAudioTransport(output_device=config.audio_output_device),
)
pipeline = VoiceAssistantPipeline(
config=config,
transport=transport,
wakeword=SherpaOnnxKeywordWakeWordProvider(
config.speech_models_dir,
keyword=config.wake_word,
keywords_file=config.wake_keywords_file,
threshold=config.wake_kws_threshold,
score=config.wake_kws_score,
),
vad_recorder=PrimarySpeakerVadRecorder(
HybridVadProvider(
SherpaOnnxVadProvider(config.speech_models_dir, threshold=config.vad_threshold),
EnergyVadProvider(),
),
min_duration_ms=config.vad_min_duration_ms,
end_silence_ms=config.vad_end_silence_ms,
no_speech_timeout_ms=config.vad_no_speech_timeout_ms,
max_recording_ms=config.vad_max_recording_ms,
speaker_profile_ms=config.speaker_profile_ms,
speaker_profile_min_ms=config.speaker_profile_min_ms,
speaker_absent_ms=config.speaker_absent_ms,
similarity_threshold=config.speaker_similarity_threshold,
min_rms=config.speaker_min_rms,
),
audio_preprocessor=SherpaOnnxDenoiserPreprocessor(config.speech_models_dir),
stt=stt,
realtime_stt=stt,
llm=llm,
tts=tts,
ack_tts=tts,
context=ConversationContext(max_messages=config.context_max_messages, max_chars=config.context_max_chars),
reporter=reporter,
event_bus=event_bus,
)
with timing.phase("pipeline_run"):
summary = pipeline.run(max_turns=turns)
timing.finish()
event_counts = Counter(event.type for event in event_bus.events)
timing_data = timing.to_json()
context_messages = [{"role": item.role, "content": item.content[:120]} for item in pipeline.context.messages()]
checks = {
"completed_turns": summary.completed_turns == turns,
"no_failed_turns": summary.failed_turns == 0,
"wake_per_turn": event_counts["wake_detected"] >= turns,
"speech_per_turn": event_counts["speech_started"] >= turns,
"final_stt_per_turn": event_counts["transcript_final"] >= turns,
"llm_per_turn": len(llm.calls) >= turns,
"tts_per_turn": event_counts["tts_started"] >= turns,
"playback_per_turn": event_counts["playback_finished"] >= turns,
"standby_per_turn": event_counts["standby_resumed"] >= turns,
"temporary_context_in_second_llm": turns < 2 or _second_call_contains_first_turn(llm.calls),
"real_output_playback": (not play_audio) or len(transport.played_segments) >= turns * 2,
"timing_present": bool(timing_data.get("duration_ms") and timing_data.get("stage_timings")),
"llm_request_timing_present": len(llm.request_timings) >= turns
and all("sent_at" in item and "duration_ms" in item for item in llm.request_timings),
}
return {
"success": all(checks.values()) and not reporter.errors,
"turns": turns,
"completed_turns": summary.completed_turns,
"failed_turns": summary.failed_turns,
"checks": checks,
"final_transcripts": [str(item["text"]) for item in reporter.finals],
"partials_preview": reporter.partials[:8],
"event_counts": dict(sorted(event_counts.items())),
"played_segments": len(transport.played_segments),
"flush_count": transport.flush_count,
"context_messages": context_messages,
"errors": reporter.errors,
"play_audio": play_audio,
"voice": voice,
"timing": timing_data,
"llm_request_timings": llm.request_timings,
}
def _question_list(turns: int, questions: list[str] | None) -> list[str]:
selected = list(questions or DEFAULT_QUESTIONS)
while len(selected) < turns:
selected.append(f"{len(selected) + 1}轮完整流程测试。")
return selected[:turns]
def _generated_live_frames(
config: AppConfig,
*,
turns: int,
voice: str,
wake_text: str,
questions: list[str],
) -> list[AudioFrame]:
with tempfile.TemporaryDirectory() as tmp_dir:
tmp = Path(tmp_dir)
wake_pcm = _synthesize_text_pcm(wake_text, tmp, "wake", voice=voice)
question_pcm = [_synthesize_text_pcm(text, tmp, f"q{idx + 1}", voice=voice) for idx, text in enumerate(questions)]
wake_frames = _trim_wake_frames_to_detection(config, _frames_from_pcm(wake_pcm, 0, 0, prefix="wake"))
frames: list[AudioFrame] = []
frame_id = 0
timestamp_ms = 0
for turn_index, pcm in enumerate(question_pcm, start=1):
copied_wake = [
AudioFrame(
item.pcm,
item.sample_rate,
item.channels,
timestamp_ms + idx * FRAME_MS,
frame_id + idx,
dict(item.metadata),
)
for idx, item in enumerate(wake_frames)
]
frames.extend(copied_wake)
frame_id += len(copied_wake)
timestamp_ms += len(copied_wake) * FRAME_MS
gap = _silence(frame_id, timestamp_ms, 160, prefix=f"turn{turn_index}-gap")
frames.extend(gap)
frame_id += len(gap)
timestamp_ms += len(gap) * FRAME_MS
question_frames = _frames_from_pcm(pcm, frame_id, timestamp_ms, prefix=f"turn{turn_index}-question")
frames.extend(question_frames)
frame_id += len(question_frames)
timestamp_ms += len(question_frames) * FRAME_MS
tail = _silence(frame_id, timestamp_ms, 900, prefix=f"turn{turn_index}-tail")
frames.extend(tail)
frame_id += len(tail)
timestamp_ms += len(tail) * FRAME_MS
return frames
def _synthesize_text_pcm(text: str, tmp: Path, name: str, *, voice: str) -> bytes:
aiff_path = tmp / f"{name}.aiff"
wav_path = tmp / f"{name}.wav"
try:
subprocess.run(
["say", "-v", voice, "-o", str(aiff_path), text],
check=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
subprocess.run(
["afconvert", "-f", "WAVE", "-d", "LEI16@16000", "-c", "1", str(aiff_path), str(wav_path)],
check=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
except (FileNotFoundError, subprocess.CalledProcessError) as exc:
raise ValueError(f"failed to generate fixture speech with macOS say/afconvert: {exc}") from exc
with wave.open(str(wav_path), "rb") as handle:
if handle.getframerate() != SAMPLE_RATE or handle.getnchannels() != 1 or handle.getsampwidth() != 2:
raise ValueError("generated speech must be 16 kHz mono int16 PCM")
return handle.readframes(handle.getnframes())
def _frames_from_pcm(pcm: bytes, start_id: int, start_ms: int, *, prefix: str) -> list[AudioFrame]:
frames: list[AudioFrame] = []
frame_id = start_id
timestamp_ms = start_ms
for offset in range(0, len(pcm), FRAME_BYTES):
chunk = pcm[offset : offset + FRAME_BYTES]
if len(chunk) < FRAME_BYTES:
chunk += b"\x00" * (FRAME_BYTES - len(chunk))
frames.append(
AudioFrame(chunk, SAMPLE_RATE, 1, timestamp_ms, frame_id, {"duration_ms": FRAME_MS, "fixture": prefix})
)
frame_id += 1
timestamp_ms += FRAME_MS
return frames
def _silence(start_id: int, start_ms: int, duration_ms: int, *, prefix: str) -> list[AudioFrame]:
return [
AudioFrame(
b"\x00" * FRAME_BYTES,
SAMPLE_RATE,
1,
start_ms + idx * FRAME_MS,
start_id + idx,
{"duration_ms": FRAME_MS, "fixture": prefix},
)
for idx in range(duration_ms // FRAME_MS)
]
def _trim_wake_frames_to_detection(config: AppConfig, frames: list[AudioFrame]) -> list[AudioFrame]:
probe_frames = list(frames)
if probe_frames:
last = probe_frames[-1]
probe_frames.extend(_silence(last.frame_id + 1, last.timestamp_ms + FRAME_MS, 1400, prefix="wake-tail"))
wake = SherpaOnnxKeywordWakeWordProvider(
config.speech_models_dir,
keyword=config.wake_word,
keywords_file=config.wake_keywords_file,
threshold=config.wake_kws_threshold,
score=config.wake_kws_score,
)
wake.load()
wake.reset()
for idx, frame in enumerate(probe_frames):
if wake.detect(frame) is not None:
return probe_frames[: idx + 1]
raise ValueError("generated wake audio did not trigger local KWS")
def _second_call_contains_first_turn(calls: list[list[Message]]) -> bool:
if len(calls) < 2:
return False
second_contents = "\n".join(message.content for message in calls[1])
return "我叫阿明" in second_contents and any(message.role == "assistant" for message in calls[1])
def _last_user_preview(messages: Sequence[Message]) -> str:
for message in reversed(messages):
if message.role == "user":
return message.content[:80]
return ""
def _stage_timings_from_events(events: list[dict[str, object]]) -> list[dict[str, object]]:
timings: list[dict[str, object]] = []
turns = sorted({int(event["turn"]) for event in events if event.get("turn") is not None})
for turn_id in turns:
turn_events = [event for event in events if event.get("turn") == turn_id]
for stage_name, start_type, end_type in STAGE_PAIRS:
timing = _pair_timing(turn_events, stage_name, start_type, end_type)
if timing is not None:
timings.append(timing)
timings.extend(_repeated_pair_timings(turn_events, "tts_playback", "tts_started", "playback_finished"))
return timings
def _pair_timing(
events: list[dict[str, object]],
name: str,
start_type: str,
end_type: str,
) -> dict[str, object] | None:
start = next((event for event in events if event.get("type") == start_type), None)
if start is None:
return None
end = next(
(event for event in events if event.get("type") == end_type and int(event["offset_ms"]) >= int(start["offset_ms"])),
None,
)
if end is None:
return None
return _timing_record(name, start, end)
def _repeated_pair_timings(
events: list[dict[str, object]],
name: str,
start_type: str,
end_type: str,
) -> list[dict[str, object]]:
timings: list[dict[str, object]] = []
pending: dict[str, object] | None = None
segment_index = 1
for event in events:
if event.get("type") == start_type:
pending = event
continue
if pending is not None and event.get("type") == end_type:
timing = _timing_record(name, pending, event)
timing["segment_index"] = segment_index
timings.append(timing)
segment_index += 1
pending = None
return timings
def _timing_record(name: str, start: dict[str, object], end: dict[str, object]) -> dict[str, object]:
return {
"turn": start.get("turn"),
"name": name,
"started_at": start.get("at"),
"finished_at": end.get("at"),
"start_offset_ms": start.get("offset_ms"),
"end_offset_ms": end.get("offset_ms"),
"duration_ms": int(end["offset_ms"]) - int(start["offset_ms"]),
"start_event": start.get("type"),
"end_event": end.get("type"),
}
+48 -10
View File
@@ -33,7 +33,7 @@ from .models import AudioSegment, ErrorCode, PipelineState, ProviderError
from .protocols import AudioTransport, LlmProvider, RealtimeSttProvider, SttProvider, TtsProvider, WakeWordProvider
from .stt import CloudAsrSttProvider, SherpaOnnxSttProvider, is_valid_transcript_text
from .transport import SoundDeviceAudioTransport
from .tts import CloudTtsProvider, MacSayTtsProvider, SentenceBuffer
from .tts import CloudTtsProvider, MacSayTtsProvider, SentenceBuffer, sanitize_tts_text
from .vad import EnergyVadProvider, HybridVadProvider, PrimarySpeakerVadRecorder, SherpaOnnxVadProvider, VadRecorder
from .wakeword import SherpaOnnxKeywordWakeWordProvider
@@ -115,6 +115,8 @@ class LiveVoiceRuntime:
self.event_bus.subscribe(self._report_event)
self.sentence_buffer = sentence_buffer or SentenceBuffer()
self._states: list[PipelineState] = []
self._cached_ack_text: str | None = None
self._cached_ack_segment: AudioSegment | None = None
def load(self) -> None:
self.wakeword.load()
@@ -123,6 +125,16 @@ class LiveVoiceRuntime:
self.tts.load()
if self.ack_tts is not self.tts:
self.ack_tts.load()
self.prepare_ack_audio()
def prepare_ack_audio(self) -> None:
text = self.config.wake_ack_text.strip()
if not text:
self._cached_ack_text = None
self._cached_ack_segment = None
return
self._cached_ack_text = text
self._cached_ack_segment = self.ack_tts.synthesize(text)
def run(self, *, once: bool = False, max_turns: int | None = None) -> RuntimeSummary:
completed = 0
@@ -240,11 +252,10 @@ class LiveVoiceRuntime:
def _acknowledge_wake(self, turn_id: int) -> ProviderError | None:
text = self.config.wake_ack_text.strip()
if not text:
self._drain_input_after_playback()
return None
try:
self._event(ACK_STARTED, PipelineState.SPEAKING, f"应答中:{text}", turn_id=turn_id)
segment = self.ack_tts.synthesize(text)
segment = self._ack_segment(text)
playback = self.transport.play_pcm(segment)
if playback.error:
return playback.error
@@ -253,10 +264,16 @@ class LiveVoiceRuntime:
except ProviderError as exc:
return exc
def _ack_segment(self, text: str) -> AudioSegment:
if self._cached_ack_text != text or self._cached_ack_segment is None:
self._cached_ack_text = text
self._cached_ack_segment = self.ack_tts.synthesize(text)
return self._cached_ack_segment
def _drain_input_after_playback(self) -> None:
self.transport.flush_input()
if self.config.post_playback_drain_ms <= 0:
return
self.transport.flush_input()
remaining_ms = self.config.post_playback_drain_ms
while remaining_ms > 0:
timeout_ms = min(50, remaining_ms)
@@ -268,13 +285,18 @@ class LiveVoiceRuntime:
self.context.append_user(user_text)
self._event(LLM_STARTED, PipelineState.THINKING, "思考中:正在生成回复", turn_id=turn_id)
assistant_text = ""
spoken_parts: list[str] = []
try:
for delta in self.llm.stream_reply(self.context.build_llm_messages()):
assistant_text += delta.text_delta
for sentence in self.sentence_buffer.feed(delta.text_delta, bool(delta.finish_reason)):
self._speak(sentence, turn_id)
spoken = self._speak(sentence, turn_id)
if spoken:
spoken_parts.append(spoken)
for sentence in self.sentence_buffer.flush():
self._speak(sentence, turn_id)
spoken = self._speak(sentence, turn_id)
if spoken:
spoken_parts.append(spoken)
except ProviderError as exc:
return self._recover(exc, turn_id)
if not assistant_text.strip():
@@ -288,18 +310,34 @@ class LiveVoiceRuntime:
),
turn_id,
)
self.context.append_assistant(assistant_text)
spoken_text = "".join(spoken_parts)
if not spoken_text.strip():
return self._recover(
ProviderError(
ErrorCode.TTS_EMPTY_AUDIO,
"LLM reply contained no speakable text after TTS sanitization",
True,
"live-runtime",
"tts",
),
turn_id,
)
self.context.append_assistant(spoken_text)
self._event(STANDBY_RESUMED, PipelineState.WAKE_LISTENING, "恢复待机:可继续唤醒", turn_id=turn_id)
return TurnResult(True, user_text, assistant_text, states=list(self._states))
return TurnResult(True, user_text, spoken_text, states=list(self._states))
def _speak(self, sentence: str, turn_id: int) -> None:
def _speak(self, sentence: str, turn_id: int) -> str:
spoken_sentence = sanitize_tts_text(sentence)
if not spoken_sentence:
return ""
self._event(TTS_STARTED, PipelineState.SPEAKING, "播放中:正在播报回复", turn_id=turn_id)
segment = self.tts.synthesize(sentence)
segment = self.tts.synthesize(spoken_sentence)
playback = self.transport.play_pcm(segment)
if playback.error:
raise playback.error
self._event(PLAYBACK_FINISHED, PipelineState.SPEAKING, "播放完成", turn_id=turn_id)
self._drain_input_after_playback()
return spoken_sentence
def _recover(self, error: ProviderError, turn_id: int) -> TurnResult:
self._event(STAGE_ERROR, PipelineState.ERROR_RECOVERING, error.message, turn_id=turn_id, payload={"error": error})
+1 -1
View File
@@ -55,7 +55,7 @@ class SimulationReporter:
class BoundedMemoryAudioTransport(MemoryAudioTransport):
def __init__(self, frames: list[AudioFrame], *, max_empty_reads: int = 5) -> None:
super().__init__(frames)
super().__init__(frames, flush_clears_input=False)
self.max_empty_reads = max_empty_reads
self.empty_reads = 0
+104
View File
@@ -6,6 +6,7 @@ import shutil
import subprocess
import tempfile
import time
from collections.abc import Callable
from collections import deque
from pathlib import Path
from typing import Any
@@ -58,12 +59,15 @@ class MemoryAudioTransport:
frames: list[AudioFrame] | None = None,
input_available: bool = True,
output_available: bool = True,
flush_clears_input: bool = True,
) -> None:
self._frames: deque[AudioFrame] = deque(frames or [])
self.played_segments: list[AudioSegment] = []
self.flush_count = 0
self.started = False
self._input_available = input_available
self._output_available = output_available
self._flush_clears_input = flush_clears_input
def start_input(
self, device_id: str | None = None, sample_rate: int = 16000, channels: int = 1
@@ -103,7 +107,27 @@ class MemoryAudioTransport:
self.played_segments.append(segment)
return PlaybackResult(True, segment.duration_ms)
def play_pcm_chunks(
self,
segment: AudioSegment,
*,
chunk_ms: int,
after_chunk: Callable[[AudioSegment, int], bool] | None = None,
) -> PlaybackResult:
elapsed_ms = 0
for chunk in _raw_pcm_chunks(segment, chunk_ms=chunk_ms):
playback = self.play_pcm(chunk)
if playback.error:
return playback
elapsed_ms += chunk.duration_ms
if after_chunk is not None and after_chunk(chunk, elapsed_ms):
return PlaybackResult(True, elapsed_ms)
return PlaybackResult(True, elapsed_ms)
def flush_input(self) -> int:
self.flush_count += 1
if not self._flush_clears_input:
return 0
count = len(self._frames)
self._frames.clear()
return count
@@ -282,6 +306,59 @@ class SoundDeviceAudioTransport:
)
return PlaybackResult(True, segment.duration_ms)
def play_pcm_chunks(
self,
segment: AudioSegment,
*,
chunk_ms: int,
after_chunk: Callable[[AudioSegment, int], bool] | None = None,
) -> PlaybackResult:
if self._sd is None or segment.metadata.get("format") in {"aiff", "wav", "mp3", "m4a", "aac"}:
playback = self.play_pcm(segment)
if playback.error:
return playback
if after_chunk is not None:
after_chunk(segment, segment.duration_ms)
return playback
if not segment.pcm:
return PlaybackResult(
False,
0,
ProviderError(
ErrorCode.AUDIO_STREAM_UNDERRUN,
"cannot play empty audio segment",
True,
"sounddevice-transport",
"transport",
),
)
elapsed_ms = 0
try:
with self._sd.RawOutputStream(
samplerate=segment.sample_rate,
channels=segment.channels,
dtype="int16",
device=_coerce_device_id(self._output_device),
) as stream:
for chunk in _raw_pcm_chunks(segment, chunk_ms=chunk_ms):
stream.write(chunk.pcm)
elapsed_ms += chunk.duration_ms
if after_chunk is not None and after_chunk(chunk, elapsed_ms):
return PlaybackResult(True, elapsed_ms)
except Exception as exc:
return PlaybackResult(
False,
elapsed_ms,
ProviderError(
ErrorCode.AUDIO_OUTPUT_DEVICE_MISSING,
f"cannot play through sounddevice output stream: {exc}",
False,
"sounddevice-transport",
"transport",
),
)
return PlaybackResult(True, elapsed_ms)
def stop(self) -> None:
if self._stream is None:
return None
@@ -412,3 +489,30 @@ def _play_file_bytes_with_afplay(segment: AudioSegment) -> PlaybackResult:
),
)
return PlaybackResult(True, segment.duration_ms)
def _raw_pcm_chunks(segment: AudioSegment, *, chunk_ms: int) -> list[AudioSegment]:
if chunk_ms <= 0 or segment.duration_ms <= chunk_ms:
return [segment]
bytes_per_ms = max(1, int(segment.sample_rate * segment.channels * 2 / 1000))
chunk_bytes = max(2 * segment.channels, bytes_per_ms * chunk_ms)
chunk_bytes -= chunk_bytes % (2 * segment.channels)
chunks: list[AudioSegment] = []
offset = 0
start_ms = segment.start_time_ms
while offset < len(segment.pcm):
data = segment.pcm[offset : offset + chunk_bytes]
duration_ms = max(1, int(len(data) / bytes_per_ms))
chunks.append(
AudioSegment(
data,
segment.sample_rate,
segment.channels,
start_ms,
start_ms + duration_ms,
dict(segment.metadata),
)
)
offset += len(data)
start_ms += duration_ms
return chunks
+149 -4
View File
@@ -3,12 +3,14 @@ from __future__ import annotations
import math
import base64
import json
import re
import socket
import struct
import subprocess
import tempfile
import urllib.error
import urllib.request
import wave
from collections.abc import Callable
from pathlib import Path
from typing import Any
@@ -17,6 +19,131 @@ from .config import AppConfig
from .models import AudioSegment, ErrorCode, ProviderError
_MARKDOWN_IMAGE_RE = re.compile(r"!\[[^\]]*]\([^)]*\)")
_SHORTCODE_EMOJI_RE = re.compile(r":(?:[A-Za-z][A-Za-z0-9_+\-]{1,31}):")
_ASCII_KAOMOJI_RE = re.compile(r"(?:\^_?\^|T_T|QAQ|QwQ|qwq|orz)")
_SYMBOL_KAOMOJI_RE = re.compile(r"[\(][^()()]{0,20}[\u00b0\u2500-\u2bff][^()()]{0,20}[\)][^\s,。!?,.!?]{0,10}")
_EMOJI_RANGES = (
(0x1F000, 0x1FAFF),
(0x2600, 0x27BF),
(0x2300, 0x23FF),
)
_EMOJI_JOINERS = {0x200D, 0x20E3}
_BRACKET_PAIRS = {"[": "]", "": "", "(": ")", "": ""}
_BRACKET_EMOTE_WORDS = {
"ok",
"doge",
"emoji",
"一笑",
"偷笑",
"傻笑",
"加油",
"发呆",
"吐舌",
"呲牙",
"",
"哭泣",
"",
"大哭",
"大笑",
"委屈",
"害羞",
"尴尬",
"开心",
"微笑",
"",
"惊讶",
"惊喜",
"惊恐",
"捂脸",
"抱拳",
"擦汗",
"",
"",
"流汗",
"流泪",
"滑稽",
"爱心",
"玫瑰",
"生气",
"白眼",
"点赞",
"破涕为笑",
"",
"笑哭",
"鼓掌",
"比心",
"亲亲",
"调皮",
"难过",
"高兴",
"鼓励",
"狗头",
}
def sanitize_tts_text(text: str) -> str:
clean = text.strip()
if not clean:
return ""
clean = _MARKDOWN_IMAGE_RE.sub("", clean)
clean = _SHORTCODE_EMOJI_RE.sub("", clean)
clean = _strip_bracket_emotes(clean)
clean = _SYMBOL_KAOMOJI_RE.sub("", clean)
clean = _ASCII_KAOMOJI_RE.sub("", clean)
clean = "".join(ch for ch in clean if not _is_emoji_char(ch))
return _normalize_spoken_text(clean)
def _strip_bracket_emotes(text: str) -> str:
result: list[str] = []
index = 0
while index < len(text):
ch = text[index]
close = _BRACKET_PAIRS.get(ch)
if close is None:
result.append(ch)
index += 1
continue
close_index = text.find(close, index + 1)
if close_index == -1 or close_index - index > 12:
result.append(ch)
index += 1
continue
content = text[index + 1 : close_index]
if _is_bracket_emote(content):
index = close_index + 1
continue
result.append(ch)
index += 1
return "".join(result)
def _is_bracket_emote(content: str) -> bool:
token = re.sub(r"\s+", "", content.strip()).lower()
if not token or len(token) > 8:
return False
if token in _BRACKET_EMOTE_WORDS or token.endswith("表情"):
return True
return all(_is_emoji_char(ch) for ch in token)
def _is_emoji_char(ch: str) -> bool:
codepoint = ord(ch)
if codepoint in _EMOJI_JOINERS or 0xFE00 <= codepoint <= 0xFE0F:
return True
return any(start <= codepoint <= end for start, end in _EMOJI_RANGES)
def _normalize_spoken_text(text: str) -> str:
clean = re.sub(r"[ \t]+", " ", text)
clean = re.sub(r"\s+([,。!?,.!?;:])", r"\1", clean)
clean = re.sub(r"([,])\s*([。!?!?])", r"\2", clean)
clean = re.sub(r"([。!?!?]){2,}", r"\1", clean)
clean = re.sub(r"\s{2,}", " ", clean)
return clean.strip(" \t\r\n,;:")
class SentenceBuffer:
def __init__(self, max_chars: int = 80) -> None:
self.max_chars = max_chars
@@ -104,8 +231,9 @@ class MacSayTtsProvider:
"tts",
)
with tempfile.TemporaryDirectory() as tmp:
output = Path(tmp) / "speech.aiff"
command = ["say", "-o", str(output)]
aiff_output = Path(tmp) / "speech.aiff"
wav_output = Path(tmp) / "speech.wav"
command = ["say", "-o", str(aiff_output)]
if self.voice:
command.extend(["-v", self.voice])
command.append(clean)
@@ -119,7 +247,23 @@ class MacSayTtsProvider:
"macos-say",
"tts",
) from exc
data = output.read_bytes()
try:
subprocess.run(
["afconvert", "-f", "WAVE", "-d", "LEI16", "-c", "1", str(aiff_output), str(wav_output)],
check=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
with wave.open(str(wav_output), "rb") as handle:
sample_rate = handle.getframerate()
channels = handle.getnchannels()
data = handle.readframes(handle.getnframes())
metadata = {"text": clean}
except (FileNotFoundError, subprocess.CalledProcessError, wave.Error):
data = aiff_output.read_bytes()
sample_rate = 16000
channels = 1
metadata = {"text": clean, "format": "aiff"}
if not data:
raise ProviderError(
ErrorCode.TTS_EMPTY_AUDIO,
@@ -128,7 +272,8 @@ class MacSayTtsProvider:
"macos-say",
"tts",
)
return AudioSegment(data, 16000, 1, 0, max(120, len(clean) * 45), {"text": clean, "format": "aiff"})
duration_ms = int(len(data) / max(1, sample_rate * channels * 2) * 1000)
return AudioSegment(data, sample_rate, channels, 0, max(120, duration_ms, len(clean) * 45), metadata)
class CloudTtsProvider:
+3
View File
@@ -283,6 +283,9 @@ class VadRecorder:
self.provider.reset()
return segment
def finish(self, end_reason: str) -> AudioSegment:
return self._build_segment(end_reason)
@dataclass(slots=True)
class PrimarySpeakerVadRecorder(VadRecorder):
+13
View File
@@ -44,6 +44,7 @@ class CliAcceptanceTests(unittest.TestCase):
self.assertEqual(data["llm_model"], "file-model")
self.assertEqual(data["pipeline_mode"], "live_turn_based")
self.assertEqual(data["endpoint_mode"], "primary_speaker")
self.assertEqual(data["realtime_transcript_idle_timeout_ms"], 1500)
self.assertEqual(data["context_mode"], "session_memory")
self.assertNotIn("secret-value", str(data))
@@ -111,6 +112,18 @@ class CliAcceptanceTests(unittest.TestCase):
self.assertEqual(data["final_transcripts"], ["第1轮模拟问题", "第2轮模拟问题"])
self.assertTrue(data["checks"]["partial_noise_filtered"])
def test_real_live_check_command_invokes_real_fixture_runner(self) -> None:
expected = {"success": True, "completed_turns": 2}
with patch("owner_voice_pet.cli.run_real_live_check", return_value=expected) as runner:
code, data = self.call("real-live-check", "--turns", "2", "--no-playback", "--question", "第一问")
self.assertEqual(code, 0)
self.assertEqual(data, expected)
kwargs = runner.call_args.kwargs
self.assertEqual(kwargs["turns"], 2)
self.assertFalse(kwargs["play_audio"])
self.assertEqual(kwargs["questions"], ["第一问"])
if __name__ == "__main__":
unittest.main()
+99
View File
@@ -0,0 +1,99 @@
from __future__ import annotations
import unittest
from owner_voice_pet.continuation import (
HybridContinuationDecisionProvider,
LlmContinuationDecisionProvider,
RuleContinuationDecisionProvider,
)
from owner_voice_pet.models import Message, ReplyDelta
class FakeClassifierLlm:
def __init__(self, text: str) -> None:
self.text = text
self.calls: list[list[Message]] = []
def stream_reply(self, messages):
self.calls.append(list(messages))
yield ReplyDelta(self.text, finish_reason="stop")
class ContinuationDecisionTests(unittest.TestCase):
def test_rule_continue_for_assistant_question(self) -> None:
decision = RuleContinuationDecisionProvider().decide(
user_text="讲讲天气",
assistant_text="你想继续听哪一部分?",
history=[],
)
self.assertEqual(decision.action, "continue")
self.assertGreaterEqual(decision.confidence, 0.65)
def test_rule_standby_for_completed_answer(self) -> None:
decision = RuleContinuationDecisionProvider().decide(
user_text="今天天气怎么样",
assistant_text="这是今天的天气。",
history=[],
)
self.assertEqual(decision.action, "standby")
self.assertGreaterEqual(decision.confidence, 0.65)
def test_hybrid_low_confidence_llm_defaults_to_standby(self) -> None:
llm = FakeClassifierLlm('{"action":"continue","confidence":0.3,"reason":""}')
provider = HybridContinuationDecisionProvider(
RuleContinuationDecisionProvider(),
LlmContinuationDecisionProvider(llm, threshold=0.65),
threshold=0.65,
)
decision = provider.decide(
user_text="继续",
assistant_text="我还可以从背景原因和下一步影响两个方向继续展开",
history=[],
)
self.assertEqual(decision.action, "standby")
self.assertEqual(len(llm.calls), 1)
def test_hybrid_generic_completed_reply_does_not_call_llm_classifier(self) -> None:
llm = FakeClassifierLlm('{"action":"continue","confidence":0.99,"reason":"不应调用"}')
provider = HybridContinuationDecisionProvider(
RuleContinuationDecisionProvider(),
LlmContinuationDecisionProvider(llm, threshold=0.65),
threshold=0.65,
)
decision = provider.decide(
user_text="没有呢",
assistant_text="明白了,我先保持待机。有需要再叫我就行。",
history=[],
)
self.assertEqual(decision.action, "standby")
self.assertEqual(decision.provider, "rule")
self.assertEqual(llm.calls, [])
def test_hybrid_high_confidence_llm_can_continue(self) -> None:
llm = FakeClassifierLlm('{"action":"continue","confidence":0.91,"reason":"等待用户选择"}')
provider = HybridContinuationDecisionProvider(
RuleContinuationDecisionProvider(),
LlmContinuationDecisionProvider(llm, threshold=0.65),
threshold=0.65,
)
decision = provider.decide(
user_text="继续",
assistant_text="我还可以从背景原因和下一步影响两个方向继续展开",
history=[],
)
self.assertEqual(decision.action, "continue")
self.assertEqual(decision.provider, "hybrid")
self.assertEqual(len(llm.calls), 1)
if __name__ == "__main__":
unittest.main()
+417 -3
View File
@@ -6,9 +6,14 @@ from owner_voice_pet.config import AppConfig
from owner_voice_pet.conversation import ConversationContext
from owner_voice_pet.events import (
ACK_STARTED,
BARGE_IN_DETECTED,
CAPTURE_STARTED,
CONTINUATION_DECISION_MADE,
FOLLOWUP_LISTENING,
FOLLOWUP_TIMEOUT,
LLM_STARTED,
PLAYBACK_FINISHED,
PLAYBACK_INTERRUPTED,
SPEECH_ENDED,
SPEECH_STARTED,
STANDBY_RESUMED,
@@ -21,7 +26,7 @@ from owner_voice_pet.events import (
PipelineEventBus,
)
from owner_voice_pet.llm import MockLlmProvider
from owner_voice_pet.models import AudioFrame, AudioSegment, Transcript
from owner_voice_pet.models import AudioFrame, AudioSegment, ErrorCode, PlaybackResult, ReplyDelta, Transcript, TransportHealth
from owner_voice_pet.assistant_pipeline import VoiceAssistantPipeline
from owner_voice_pet.runtime import build_live_runtime
from owner_voice_pet.stt import MetadataSttProvider, SherpaOnnxSttProvider
@@ -47,6 +52,28 @@ def segment_frames(start_id: int, start_ms: int, partials: list[str] | None = No
]
def long_speech_frames_with_stale_partial(start_id: int, start_ms: int, duration_ms: int) -> list[AudioFrame]:
frames: list[AudioFrame] = []
for index in range(duration_ms // 20):
metadata: dict[str, object] = {
"duration_ms": 20,
"speech": True,
"partial_transcript": "你知道",
"transcript": "你知道我在说什么吗",
}
frames.append(
AudioFrame(
b"\xff\x7f",
16000,
1,
start_ms + index * 20,
start_id + index,
metadata,
)
)
return frames
def wake_frame(frame_id: int, timestamp_ms: int) -> AudioFrame:
return AudioFrame(
b"\xff\x7f",
@@ -73,6 +100,32 @@ class QueueSttProvider:
return Transcript(text, "zh", 1.0, segment.duration_ms, "queue-stt")
class QueueLlmProvider:
def __init__(self, replies: list[list[str]]) -> None:
self.replies = [list(item) for item in replies]
self.calls = []
def stream_reply(self, messages):
self.calls.append(list(messages))
chunks = self.replies.pop(0)
for chunk in chunks:
yield ReplyDelta(chunk)
yield ReplyDelta("", finish_reason="stop")
class CountingTtsProvider:
def __init__(self) -> None:
self.delegate = SineTtsProvider()
self.synthesized_texts: list[str] = []
def load(self) -> None:
self.delegate.load()
def synthesize(self, text: str) -> AudioSegment:
self.synthesized_texts.append(text)
return self.delegate.synthesize(text)
class RecordingReporter:
def __init__(self) -> None:
self.statuses: list[str] = []
@@ -134,6 +187,7 @@ def make_runtime(
context: ConversationContext | None = None,
partial_texts: list[list[str]] | None = None,
audio_preprocessor: MarkerAudioPreprocessor | None = None,
wake_ack_text: str = "我在",
) -> tuple[VoiceAssistantPipeline, QueueSttProvider, MockLlmProvider, MemoryAudioTransport, RecordingReporter]:
frames = []
for idx, _text in enumerate(texts):
@@ -142,14 +196,14 @@ def make_runtime(
frames.append(wake_frame(base_id, base_ms))
partials = partial_texts[idx] if partial_texts and idx < len(partial_texts) else None
frames.extend(segment_frames(base_id + 1, base_ms + 20, partials=partials))
transport = MemoryAudioTransport(frames)
transport = MemoryAudioTransport(frames, flush_clears_input=False)
stt = QueueSttProvider(texts)
llm = MockLlmProvider(["这是答复。"])
tts = SineTtsProvider()
reporter = RecordingReporter()
event_bus = PipelineEventBus()
runtime = VoiceAssistantPipeline(
config=AppConfig(llm_api_key="secret", speech_provider="cloud"),
config=AppConfig(llm_api_key="secret", speech_provider="cloud", wake_ack_text=wake_ack_text),
transport=transport,
wakeword=KeywordWakeWordProvider(),
vad_recorder=VadRecorder(EnergyVadProvider(), min_duration_ms=40, end_silence_ms=40),
@@ -165,6 +219,45 @@ def make_runtime(
return runtime, stt, llm, transport, reporter
class PlaybackInjectedTransport(MemoryAudioTransport):
def __init__(
self,
frames: list[AudioFrame],
*,
inject_after_play_count: int,
injected_frames: list[AudioFrame],
) -> None:
super().__init__(frames, flush_clears_input=False)
self.inject_after_play_count = inject_after_play_count
self.injected_frames = list(injected_frames)
self.play_count = 0
def play_pcm(self, segment: AudioSegment, interrupt: bool = False) -> PlaybackResult:
result = super().play_pcm(segment, interrupt=interrupt)
self.play_count += 1
if self.play_count == self.inject_after_play_count:
for frame in self.injected_frames:
self.inject(frame)
return result
def health(self) -> TransportHealth:
return TransportHealth(True, True, "playback injected")
def silence_frames(start_id: int, start_ms: int, count: int) -> list[AudioFrame]:
return [
AudioFrame(
b"\x00\x00",
16000,
1,
start_ms + index * 20,
start_id + index,
{"duration_ms": 20, "speech": False},
)
for index in range(count)
]
class LiveRuntimeTests(unittest.TestCase):
def test_repeated_runtime_runs_two_turns_and_returns_to_standby(self) -> None:
runtime, stt, llm, transport, reporter = make_runtime(["第一问", "第二问"])
@@ -177,6 +270,7 @@ class LiveRuntimeTests(unittest.TestCase):
self.assertEqual(len(stt.calls), 2)
self.assertEqual(len(llm.calls), 2)
self.assertEqual(len(transport.played_segments), 4)
self.assertEqual(transport.flush_count, 4)
self.assertEqual(reporter.transcripts, ["第一问", "第二问"])
self.assertIn("应答中:我在", reporter.statuses)
self.assertLess(reporter.statuses.index("唤醒命中"), reporter.statuses.index("应答中:我在"))
@@ -201,6 +295,59 @@ class LiveRuntimeTests(unittest.TestCase):
positions = [event_types.index(item) for item in expected_order]
self.assertEqual(positions, sorted(positions))
def test_wake_ack_audio_is_prepared_once_and_reused(self) -> None:
frames = []
for idx, _text in enumerate(["第一问", "第二问"]):
base_id = idx * 5
base_ms = idx * 120
frames.append(wake_frame(base_id, base_ms))
frames.extend(segment_frames(base_id + 1, base_ms + 20))
transport = MemoryAudioTransport(frames, flush_clears_input=False)
ack_tts = CountingTtsProvider()
runtime = VoiceAssistantPipeline(
config=AppConfig(llm_api_key="secret", speech_provider="cloud", wake_ack_text="我在"),
transport=transport,
wakeword=KeywordWakeWordProvider(),
vad_recorder=VadRecorder(EnergyVadProvider(), min_duration_ms=40, end_silence_ms=40),
stt=QueueSttProvider(["第一问", "第二问"]),
realtime_stt=None,
llm=MockLlmProvider(["这是答复。"]),
tts=SineTtsProvider(),
ack_tts=ack_tts,
context=ConversationContext(),
reporter=RecordingReporter(),
event_bus=PipelineEventBus(),
)
summary = runtime.run(max_turns=2)
self.assertEqual(summary.completed_turns, 2)
self.assertEqual(ack_tts.synthesized_texts, ["我在"])
self.assertEqual(transport.played_segments[0].metadata["text"], "我在")
self.assertEqual(transport.played_segments[2].metadata["text"], "我在")
def test_zero_post_playback_drain_flushes_without_dropping_prefilled_question(self) -> None:
runtime, stt, _, transport, reporter = make_runtime(["第一问"])
self.assertEqual(runtime.config.post_playback_drain_ms, 0)
summary = runtime.run(max_turns=1)
self.assertEqual(summary.completed_turns, 1)
self.assertEqual(reporter.transcripts, ["第一问"])
self.assertEqual(len(stt.calls), 1)
self.assertEqual(transport.flush_count, 2)
def test_no_ack_text_does_not_drain_before_capture(self) -> None:
runtime, stt, _, transport, reporter = make_runtime(["第一问"], wake_ack_text="")
summary = runtime.run(max_turns=1)
self.assertEqual(summary.completed_turns, 1)
self.assertEqual(reporter.transcripts, ["第一问"])
self.assertEqual(len(stt.calls), 1)
self.assertEqual(len(transport.played_segments), 1)
self.assertEqual(transport.flush_count, 1)
def test_temporary_context_is_sent_to_second_llm_call(self) -> None:
runtime, _, llm, _, _ = make_runtime(["第一问", "第二问"])
runtime.run(max_turns=2)
@@ -242,6 +389,47 @@ class LiveRuntimeTests(unittest.TestCase):
self.assertLess(event_types.index(TRANSCRIPT_PARTIAL), event_types.index(SPEECH_ENDED))
self.assertLess(event_types.index(TRANSCRIPT_PARTIAL), event_types.index(TRANSCRIPT_FINAL))
def test_realtime_transcript_idle_ends_current_utterance(self) -> None:
frames = [wake_frame(0, 0)]
frames.extend(long_speech_frames_with_stale_partial(1, 20, 2200))
transport = MemoryAudioTransport(frames, flush_clears_input=False)
stt = QueueSttProvider(["你知道我在说什么吗"])
llm = MockLlmProvider(["这是答复。"])
reporter = RecordingReporter()
runtime = VoiceAssistantPipeline(
config=AppConfig(
llm_api_key="secret",
speech_provider="cloud",
wake_ack_text="",
realtime_transcript_idle_timeout_ms=1500,
),
transport=transport,
wakeword=KeywordWakeWordProvider(),
vad_recorder=VadRecorder(
EnergyVadProvider(),
min_duration_ms=40,
end_silence_ms=5000,
max_recording_ms=10000,
),
stt=stt,
audio_preprocessor=MarkerAudioPreprocessor(partial_text="你知道"),
realtime_stt=MetadataSttProvider(),
llm=llm,
tts=SineTtsProvider(),
context=ConversationContext(),
reporter=reporter,
event_bus=PipelineEventBus(),
)
summary = runtime.run(max_turns=1)
self.assertEqual(summary.completed_turns, 1)
self.assertEqual(reporter.partials, ["你知道"])
self.assertEqual(reporter.transcripts, ["你知道我在说什么吗"])
self.assertEqual(len(stt.calls), 1)
self.assertEqual(stt.calls[0].metadata["end_reason"], "partial_transcript_idle")
self.assertLess(stt.calls[0].duration_ms, 1700)
def test_capture_uses_denoised_frames_for_partial_and_final_stt(self) -> None:
preprocessor = MarkerAudioPreprocessor(partial_text="降噪后问题")
runtime, stt, _, _, reporter = make_runtime(
@@ -265,6 +453,64 @@ class LiveRuntimeTests(unittest.TestCase):
self.assertEqual(llm.calls[0][-1].content, "第一问")
self.assertNotIn("小杰小杰", llm.calls[0][-1].content)
def test_assistant_reply_sanitizes_tts_text_and_context(self) -> None:
frames = [wake_frame(0, 0)]
frames.extend(segment_frames(1, 20, partials=["第一问", "第一问"]))
transport = MemoryAudioTransport(frames, flush_clears_input=False)
stt = QueueSttProvider(["第一问"])
llm = QueueLlmProvider([["你好 😊。没问题[捂脸],我来帮你。"]])
context = ConversationContext()
runtime = VoiceAssistantPipeline(
config=AppConfig(llm_api_key="secret", speech_provider="cloud", wake_ack_text=""),
transport=transport,
wakeword=KeywordWakeWordProvider(),
vad_recorder=VadRecorder(EnergyVadProvider(), min_duration_ms=40, end_silence_ms=40),
stt=stt,
realtime_stt=MetadataSttProvider(),
llm=llm,
tts=SineTtsProvider(),
context=context,
reporter=RecordingReporter(),
event_bus=PipelineEventBus(),
)
summary = runtime.run(max_turns=1)
context_texts = [message.content for message in context.messages()]
self.assertEqual(summary.completed_turns, 1)
self.assertEqual(transport.played_segments[0].metadata["text"], "你好。没问题,我来帮你。")
self.assertIn("你好。没问题,我来帮你。", context_texts)
self.assertFalse(any("😊" in text or "[捂脸]" in text for text in context_texts))
def test_emoji_only_reply_recovers_without_tts_playback(self) -> None:
frames = [wake_frame(0, 0)]
frames.extend(segment_frames(1, 20, partials=["第一问", "第一问"]))
transport = MemoryAudioTransport(frames, flush_clears_input=False)
runtime = VoiceAssistantPipeline(
config=AppConfig(llm_api_key="secret", speech_provider="cloud", wake_ack_text=""),
transport=transport,
wakeword=KeywordWakeWordProvider(),
vad_recorder=VadRecorder(EnergyVadProvider(), min_duration_ms=40, end_silence_ms=40),
stt=QueueSttProvider(["第一问"]),
realtime_stt=MetadataSttProvider(),
llm=QueueLlmProvider([["😂😂"]]),
tts=SineTtsProvider(),
context=ConversationContext(),
reporter=RecordingReporter(),
event_bus=PipelineEventBus(),
)
summary = runtime.run(once=True)
event_types = [event.type for event in runtime.event_bus.events]
self.assertEqual(summary.completed_turns, 0)
self.assertEqual(summary.failed_turns, 1)
self.assertIsNotNone(summary.last_error)
self.assertEqual(summary.last_error.code, ErrorCode.TTS_EMPTY_AUDIO)
self.assertEqual(transport.played_segments, [])
self.assertNotIn(TTS_STARTED, event_types)
self.assertEqual(event_types[-1], STANDBY_RESUMED)
def test_live_runtime_uses_primary_speaker_endpoint_by_default(self) -> None:
runtime = build_live_runtime(AppConfig(llm_api_key="secret"))
self.assertIsInstance(runtime.vad_recorder, PrimarySpeakerVadRecorder)
@@ -272,6 +518,174 @@ class LiveRuntimeTests(unittest.TestCase):
self.assertIsInstance(runtime.stt, SherpaOnnxSttProvider)
self.assertIsNotNone(runtime.realtime_stt)
def test_assistant_followup_question_enters_listening_without_second_wake(self) -> None:
frames = [wake_frame(0, 0)]
frames.extend(segment_frames(1, 20, partials=["第一问", "第一问"]))
frames.extend(segment_frames(10, 400, partials=["继续内容", "继续内容"]))
transport = MemoryAudioTransport(frames, flush_clears_input=False)
stt = QueueSttProvider(["第一问", "继续内容"])
llm = QueueLlmProvider([["你想继续听哪一部分?"], ["这是补充回答。"]])
reporter = RecordingReporter()
runtime = VoiceAssistantPipeline(
config=AppConfig(
llm_api_key="secret",
speech_provider="cloud",
wake_ack_text="",
followup_listen_timeout_ms=3000,
),
transport=transport,
wakeword=KeywordWakeWordProvider(),
vad_recorder=VadRecorder(EnergyVadProvider(), min_duration_ms=40, end_silence_ms=40),
stt=stt,
realtime_stt=MetadataSttProvider(),
llm=llm,
tts=SineTtsProvider(),
context=ConversationContext(),
reporter=reporter,
event_bus=PipelineEventBus(),
)
summary = runtime.run(max_turns=1)
event_types = [event.type for event in runtime.event_bus.events]
self.assertEqual(summary.completed_turns, 2)
self.assertEqual(event_types.count(WAKE_DETECTED), 1)
self.assertEqual(event_types.count(FOLLOWUP_LISTENING), 1)
self.assertEqual(event_types.count(STT_STARTED), 2)
self.assertEqual(len(llm.calls), 2)
self.assertEqual(reporter.transcripts, ["第一问", "继续内容"])
second_call_text = [message.content for message in llm.calls[1]]
self.assertIn("第一问", second_call_text)
self.assertIn("你想继续听哪一部分?", second_call_text)
def test_followup_timeout_returns_to_standby(self) -> None:
frames = [wake_frame(0, 0)]
frames.extend(segment_frames(1, 20, partials=["第一问", "第一问"]))
frames.extend(silence_frames(20, 400, 170))
transport = MemoryAudioTransport(frames, flush_clears_input=False)
stt = QueueSttProvider(["第一问"])
llm = QueueLlmProvider([["你想继续听哪一部分?"]])
reporter = RecordingReporter()
runtime = VoiceAssistantPipeline(
config=AppConfig(
llm_api_key="secret",
speech_provider="cloud",
wake_ack_text="",
followup_listen_timeout_ms=3000,
),
transport=transport,
wakeword=KeywordWakeWordProvider(),
vad_recorder=VadRecorder(EnergyVadProvider(), min_duration_ms=40, end_silence_ms=40),
stt=stt,
realtime_stt=MetadataSttProvider(),
llm=llm,
tts=SineTtsProvider(),
context=ConversationContext(),
reporter=reporter,
event_bus=PipelineEventBus(),
)
summary = runtime.run(max_turns=1)
event_types = [event.type for event in runtime.event_bus.events]
self.assertEqual(summary.completed_turns, 1)
self.assertEqual(summary.failed_turns, 0)
self.assertIn(FOLLOWUP_TIMEOUT, event_types)
self.assertEqual(event_types[-1], STANDBY_RESUMED)
self.assertEqual(len(llm.calls), 1)
def test_completed_reply_returns_to_standby_without_cloud_classifier_delay(self) -> None:
frames = [wake_frame(0, 0)]
frames.extend(segment_frames(1, 20, partials=["没有呢", "没有呢"]))
transport = MemoryAudioTransport(frames, flush_clears_input=False)
llm = QueueLlmProvider([["明白了,我先保持待机。有需要再叫我就行。"]])
runtime = VoiceAssistantPipeline(
config=AppConfig(
llm_api_key="secret",
speech_provider="cloud",
wake_ack_text="",
followup_listen_timeout_ms=3000,
),
transport=transport,
wakeword=KeywordWakeWordProvider(),
vad_recorder=VadRecorder(EnergyVadProvider(), min_duration_ms=40, end_silence_ms=40),
stt=QueueSttProvider(["没有呢"]),
realtime_stt=MetadataSttProvider(),
llm=llm,
tts=SineTtsProvider(),
context=ConversationContext(),
reporter=RecordingReporter(),
event_bus=PipelineEventBus(),
)
summary = runtime.run(max_turns=1)
event_types = [event.type for event in runtime.event_bus.events]
self.assertEqual(summary.completed_turns, 1)
self.assertNotIn(FOLLOWUP_LISTENING, event_types)
self.assertEqual(event_types[-1], STANDBY_RESUMED)
self.assertEqual(len(llm.calls), 1)
def test_barge_in_interrupts_playback_and_keeps_only_spoken_context(self) -> None:
first_question_frames = [wake_frame(0, 0)]
first_question_frames.extend(segment_frames(1, 20, partials=["第一问", "第一问"]))
barge_frames = [
AudioFrame(b"\xff\x7f", 16000, 1, 600, 20, {"duration_ms": 20, "speech": True, "partial_transcript": "等一下"}),
AudioFrame(b"\xff\x7f", 16000, 1, 620, 21, {"duration_ms": 20, "speech": True, "partial_transcript": "等一下"}),
*silence_frames(22, 640, 3),
]
transport = PlaybackInjectedTransport(
first_question_frames,
inject_after_play_count=8,
injected_frames=barge_frames,
)
stt = QueueSttProvider(["第一问", "打断问题"])
llm = QueueLlmProvider(
[
[
"已播出一句。",
"这是一段需要被打断的很长很长很长很长很长很长很长很长的回复内容,没有播放完。",
],
["这是新回答。"],
]
)
reporter = RecordingReporter()
runtime = VoiceAssistantPipeline(
config=AppConfig(
llm_api_key="secret",
speech_provider="cloud",
wake_ack_text="",
barge_in_enabled=True,
barge_in_echo_guard_ms=500,
barge_in_min_speech_ms=40,
followup_listen_timeout_ms=3000,
),
transport=transport,
wakeword=KeywordWakeWordProvider(),
vad_recorder=VadRecorder(EnergyVadProvider(), min_duration_ms=40, end_silence_ms=40),
stt=stt,
realtime_stt=MetadataSttProvider(),
llm=llm,
tts=SineTtsProvider(),
context=ConversationContext(),
reporter=reporter,
event_bus=PipelineEventBus(),
)
summary = runtime.run(max_turns=2)
event_types = [event.type for event in runtime.event_bus.events]
context_texts = [message.content for message in runtime.context.messages()]
playback_window = event_types[event_types.index(TTS_STARTED) : event_types.index(PLAYBACK_INTERRUPTED)]
self.assertEqual(summary.completed_turns, 2)
self.assertIn(BARGE_IN_DETECTED, event_types)
self.assertIn(PLAYBACK_INTERRUPTED, event_types)
self.assertNotIn(TRANSCRIPT_PARTIAL, playback_window)
self.assertIn("已播出一句。", context_texts)
self.assertNotIn("这是一段需要被打断的很长很长很长很长很长很长很长很长的回复内容,没有播放完。", context_texts)
self.assertEqual(reporter.transcripts[-1], "打断问题")
self.assertEqual(len(llm.calls), 2)
if __name__ == "__main__":
unittest.main()
+29
View File
@@ -78,6 +78,7 @@ class ModelsConfigTests(unittest.TestCase):
self.assertEqual(config.llm_api_key, "secret-value")
self.assertEqual(config.llm_model, "test-model")
self.assertTrue(config.realtime_transcript_enabled)
self.assertEqual(config.realtime_transcript_idle_timeout_ms, 1500)
self.assertEqual(config.wake_provider, "local_kws")
self.assertEqual(config.wake_kws_threshold, 0.15)
self.assertEqual(config.wake_kws_score, 1.0)
@@ -105,6 +106,13 @@ class ModelsConfigTests(unittest.TestCase):
self.assertEqual(config.tts_voice, "mimo_default")
self.assertEqual(str(config.speech_models_dir), "models")
self.assertEqual(config.context_mode, "session_memory")
self.assertTrue(config.continuous_dialog_enabled)
self.assertEqual(config.continuation_decision_provider, "hybrid")
self.assertEqual(config.continuation_confidence_threshold, 0.65)
self.assertEqual(config.followup_listen_timeout_ms, 3000)
self.assertTrue(config.barge_in_enabled)
self.assertEqual(config.barge_in_min_speech_ms, 250)
self.assertEqual(config.barge_in_echo_guard_ms, 500)
self.assertTrue(config.llm_stream)
self.assertEqual(config.validate_basic(), [])
@@ -113,6 +121,11 @@ class ModelsConfigTests(unittest.TestCase):
errors = config.validate_basic()
self.assertTrue(any("OWNER_SPEECH_PROVIDER" in error.message for error in errors))
def test_realtime_transcript_idle_timeout_must_be_non_negative(self) -> None:
config = AppConfig(realtime_transcript_idle_timeout_ms=-1)
errors = config.validate_basic()
self.assertTrue(any("OWNER_REALTIME_TRANSCRIPT_IDLE_TIMEOUT_MS" in error.message for error in errors))
def test_wake_provider_must_be_local_kws(self) -> None:
config = AppConfig(wake_provider="cloud_asr")
errors = config.validate_basic()
@@ -133,6 +146,22 @@ class ModelsConfigTests(unittest.TestCase):
errors = config.validate_basic()
self.assertTrue(any("OWNER_NOISE_FILTER_PROVIDER" in error.message for error in errors))
def test_continuation_config_is_validated(self) -> None:
config = AppConfig(
continuation_decision_provider="invalid",
continuation_confidence_threshold=2.0,
followup_listen_timeout_ms=-1,
barge_in_min_speech_ms=-1,
barge_in_echo_guard_ms=-1,
)
errors = config.validate_basic()
self.assertTrue(any("OWNER_CONTINUATION_DECISION_PROVIDER" in error.message for error in errors))
self.assertTrue(any("OWNER_CONTINUATION_CONFIDENCE_THRESHOLD" in error.message for error in errors))
self.assertTrue(any("OWNER_FOLLOWUP_LISTEN_TIMEOUT_MS" in error.message for error in errors))
self.assertTrue(any("OWNER_BARGE_IN_MIN_SPEECH_MS" in error.message for error in errors))
self.assertTrue(any("OWNER_BARGE_IN_ECHO_GUARD_MS" in error.message for error in errors))
def test_speaker_similarity_threshold_range_is_validated(self) -> None:
config = AppConfig(speaker_similarity_threshold=1.5)
errors = config.validate_basic()
+23 -1
View File
@@ -12,7 +12,7 @@ from owner_voice_pet.models import AudioFrame, ErrorCode, Message, PipelineState
from owner_voice_pet.pipeline import VoicePipeline
from owner_voice_pet.stt import MetadataSttProvider
from owner_voice_pet.transport import MemoryAudioTransport
from owner_voice_pet.tts import CloudTtsProvider, SentenceBuffer, SineTtsProvider
from owner_voice_pet.tts import CloudTtsProvider, SentenceBuffer, SineTtsProvider, sanitize_tts_text
from owner_voice_pet.vad import EnergyVadProvider, VadRecorder
from owner_voice_pet.wakeword import KeywordWakeWordProvider
@@ -60,6 +60,12 @@ class PipelineLlmTtsTests(unittest.TestCase):
self.assertEqual(buffer.feed("剩余").copy(), [])
self.assertEqual(buffer.flush(), ["剩余"])
def test_sanitize_tts_text_removes_unspeakable_expression_tokens(self) -> None:
self.assertEqual(sanitize_tts_text("你好 😊"), "你好")
self.assertEqual(sanitize_tts_text("好的![开心](x.png)"), "好的")
self.assertEqual(sanitize_tts_text("没问题[捂脸],我来帮你。"), "没问题,我来帮你。")
self.assertEqual(sanitize_tts_text("😂😂"), "")
def test_sine_tts_generates_non_empty_audio(self) -> None:
provider = SineTtsProvider()
provider.load()
@@ -126,6 +132,22 @@ class PipelineLlmTtsTests(unittest.TestCase):
self.assertEqual(len(transport.played_segments), 1)
self.assertEqual(pipeline.context.messages()[-1].role, "assistant")
def test_pipeline_sanitizes_tts_text_and_assistant_context(self) -> None:
frames = [
frame(0, 0, {"wake_word": "小杰小杰", "wake_confidence": 0.95}),
frame(1, 20, {"speech": True, "transcript": "你好"}),
frame(2, 40, {"speech": True}),
frame(3, 60, {"speech": False}),
frame(4, 80, {"speech": False}),
]
pipeline, transport = make_pipeline(frames, MockLlmProvider(["你好 😊。没问题[捂脸],我来帮你。"]))
result = pipeline.run_once()
self.assertTrue(result.success)
self.assertEqual(result.assistant_text, "你好。没问题,我来帮你。")
self.assertEqual(transport.played_segments[0].metadata["text"], "你好。没问题,我来帮你。")
self.assertEqual(pipeline.context.messages()[-1].content, "你好。没问题,我来帮你。")
def test_pipeline_skips_llm_on_empty_transcript(self) -> None:
llm = MockLlmProvider(["不应调用"])
frames = [
+100
View File
@@ -0,0 +1,100 @@
from __future__ import annotations
import unittest
from owner_voice_pet.models import AudioFrame, AudioSegment, Message, PlaybackResult, ReplyDelta, TransportHealth
from owner_voice_pet.real_live_check import (
FixtureLiveAudioTransport,
RecordingLlmProvider,
TimingRecorder,
_stage_timings_from_events,
)
def frame(idx: int) -> AudioFrame:
return AudioFrame(b"\x00\x00", 16000, 1, idx * 20, idx, {"duration_ms": 20})
class FakeOutputTransport:
def __init__(self) -> None:
self.played: list[AudioSegment] = []
def play_pcm(self, segment: AudioSegment, interrupt: bool = False) -> PlaybackResult:
self.played.append(segment)
return PlaybackResult(True, segment.duration_ms)
def health(self) -> TransportHealth:
return TransportHealth(True, True, "fake")
class RealLiveCheckTests(unittest.TestCase):
def test_fixture_transport_flush_preserves_generated_future_frames(self) -> None:
transport = FixtureLiveAudioTransport([frame(1), frame(2)], play_audio=False)
transport.start_input()
self.assertEqual(transport.flush_input(), 0)
self.assertEqual(transport.flush_count, 1)
self.assertEqual([item.frame_id for item in transport.read_frames(10)], [1])
def test_fixture_transport_can_delegate_real_playback(self) -> None:
output = FakeOutputTransport()
transport = FixtureLiveAudioTransport([], play_audio=True, output_transport=output)
segment = AudioSegment(b"\x00\x00", 16000, 1, 0, 20)
result = transport.play_pcm(segment)
self.assertTrue(result.played)
self.assertEqual(output.played, [segment])
self.assertEqual(transport.played_segments, [segment])
def test_stage_timings_are_built_from_pipeline_events(self) -> None:
events = [
{"type": "wake_listening", "turn": 1, "at": "t0", "offset_ms": 0, "message": ""},
{"type": "wake_detected", "turn": 1, "at": "t1", "offset_ms": 120, "message": ""},
{"type": "ack_started", "turn": 1, "at": "t2", "offset_ms": 130, "message": ""},
{"type": "question_prompt", "turn": 1, "at": "t3", "offset_ms": 250, "message": ""},
{"type": "capture_started", "turn": 1, "at": "t4", "offset_ms": 260, "message": ""},
{"type": "speech_started", "turn": 1, "at": "t5", "offset_ms": 300, "message": ""},
{"type": "speech_ended", "turn": 1, "at": "t6", "offset_ms": 700, "message": ""},
{"type": "stt_started", "turn": 1, "at": "t7", "offset_ms": 720, "message": ""},
{"type": "transcript_final", "turn": 1, "at": "t8", "offset_ms": 900, "message": ""},
{"type": "llm_started", "turn": 1, "at": "t9", "offset_ms": 910, "message": ""},
{"type": "tts_started", "turn": 1, "at": "t10", "offset_ms": 1300, "message": ""},
{"type": "playback_finished", "turn": 1, "at": "t11", "offset_ms": 1500, "message": ""},
{"type": "standby_resumed", "turn": 1, "at": "t12", "offset_ms": 1520, "message": ""},
]
timings = _stage_timings_from_events(events)
by_name = {str(item["name"]): item for item in timings if "segment_index" not in item}
self.assertEqual(by_name["wake_wait"]["duration_ms"], 120)
self.assertEqual(by_name["acknowledgement"]["duration_ms"], 120)
self.assertEqual(by_name["capture"]["duration_ms"], 400)
self.assertEqual(by_name["final_stt"]["duration_ms"], 180)
self.assertEqual(by_name["llm_to_first_tts"]["duration_ms"], 390)
self.assertEqual(by_name["turn_total"]["duration_ms"], 1520)
tts = [item for item in timings if item["name"] == "tts_playback"]
self.assertEqual(tts[0]["duration_ms"], 200)
def test_recording_llm_provider_records_sent_time_and_response_duration(self) -> None:
class FakeLlm:
def stream_reply(self, messages):
yield ReplyDelta("你好", finish_reason="stop")
timing = TimingRecorder()
provider = RecordingLlmProvider(FakeLlm(), timing) # type: ignore[arg-type]
messages = [Message("user", "现在几点", 0.0)]
deltas = list(provider.stream_reply(messages))
self.assertEqual(deltas[0].text_delta, "你好")
self.assertEqual(len(provider.request_timings), 1)
request = provider.request_timings[0]
self.assertIn("sent_at", request)
self.assertIn("finished_at", request)
self.assertIsInstance(request["duration_ms"], int)
self.assertEqual(request["last_user_preview"], "现在几点")
if __name__ == "__main__":
unittest.main()
+28
View File
@@ -46,8 +46,16 @@ class TransportTests(unittest.TestCase):
transport = MemoryAudioTransport([frame(1, 0), frame(2, 20)])
transport.start_input()
self.assertEqual(transport.flush_input(), 2)
self.assertEqual(transport.flush_count, 1)
self.assertEqual(transport.read_frames(10), [])
def test_memory_transport_can_preserve_prefilled_frames_on_flush(self) -> None:
transport = MemoryAudioTransport([frame(1, 0), frame(2, 20)], flush_clears_input=False)
transport.start_input()
self.assertEqual(transport.flush_input(), 0)
self.assertEqual(transport.flush_count, 1)
self.assertEqual([f.frame_id for f in transport.read_frames(10)], [1])
def test_file_replay_roundtrip(self) -> None:
frames = [frame(1, 0, {"wake": True}), frame(2, 20, {"speech": True})]
with tempfile.TemporaryDirectory() as tmp:
@@ -106,6 +114,24 @@ class TransportTests(unittest.TestCase):
self.assertTrue(result.played)
self.assertEqual(fake.output_writes, [b"\x00\x00\x01\x00"])
self.assertEqual(fake.output_stream_open_count, 1)
def test_sounddevice_chunk_playback_keeps_one_output_stream_open(self) -> None:
fake = FakeSoundDevice()
transport = SoundDeviceAudioTransport(sounddevice_module=fake)
segment = AudioSegment(b"\x00\x00" * 1600, 16000, 1, 0, 100)
callbacks: list[int] = []
result = transport.play_pcm_chunks(
segment,
chunk_ms=20,
after_chunk=lambda _chunk, elapsed_ms: callbacks.append(elapsed_ms) is not None and False,
)
self.assertTrue(result.played)
self.assertEqual(fake.output_stream_open_count, 1)
self.assertGreater(len(fake.output_writes), 1)
self.assertEqual(callbacks[-1], result.duration_ms)
def test_sounddevice_device_report_uses_query_devices(self) -> None:
fake = FakeSoundDevice()
@@ -154,6 +180,7 @@ class FakeSoundDevice:
def __init__(self, callback_payloads: list[bytes] | None = None) -> None:
self.callback_payloads = callback_payloads
self.output_writes: list[bytes] = []
self.output_stream_open_count = 0
def RawInputStream(self, **kwargs):
if self.callback_payloads is not None:
@@ -161,6 +188,7 @@ class FakeSoundDevice:
return FakeInputStream(**kwargs)
def RawOutputStream(self, **kwargs):
self.output_stream_open_count += 1
return FakeOutputStream(self, **kwargs)
def query_devices(self):