[连续对话判断]：完成自动持续对话和播报打断，包含回复意图判断、免唤醒追问和打断回归测试

2026-06-18 12:08:41 +08:00
parent 1e956f5eb6
commit 7519725321
16 changed files with 1059 additions and 119 deletions
@@ -295,6 +295,37 @@ The real LLM provider is wrapped by `RecordingLlmProvider`. For every cloud LLM

 Timing output is diagnostic metadata only. It SHALL NOT include `OWNER_LLM_API_KEY`, raw audio, denoised audio, full provider request headers, or any other secret-bearing configuration.

+## Automatic Continuous Dialog Decision
+
+持续对话不再等同于“每次回复后固定打开 3 秒追问窗口”。新的回复后状态流为：
+
+```text
+llm_started
+  -> tts_started/playback_finished
+  -> continuation_decision_started
+  -> continuation_decision_made(continue|standby)
+  -> standby_resumed
+     OR followup_listening -> capture_started -> transcript_final -> llm_started ...
+```
+
+`ContinuationDecisionStage` 使用 hybrid provider：
+
+1. `RuleContinuationDecisionProvider` 先用可解释规则判断。明确问题、补充信息请求、选择题、等待用户下一步的表达判定为 `continue`；完成性陈述、播报结果、拒绝/报错、寒暄结束和不确定表达判定为 `standby`。
+2. `LlmContinuationDecisionProvider` 只在规则返回 uncertain 时调用云端 LLM 小分类器。分类器 prompt 要求只输出 `continue` 或 `standby`，可附带 `confidence`；解析失败或低于 `OWNER_CONTINUATION_CONFIDENCE_THRESHOLD` 时按 `standby`。
+3. `HybridContinuationDecisionProvider` 是默认 provider，由 `OWNER_CONTINUATION_DECISION_PROVIDER=hybrid` 启用；`rule` 可用于完全离线决策。
+
+Follow-up 捕获复用现有 capture、实时字幕、final STT、LLM、TTS 流程，但跳过 wake 和 ACK。为了让“3 秒内无回答”真正快退，follow-up 捕获临时把 recorder no-speech timeout 限制为 `OWNER_FOLLOWUP_LISTEN_TIMEOUT_MS`；若返回 `VAD_TIMEOUT_NO_SPEECH`，事件层输出 `followup_timeout` 和 `continuous_session_ended`，然后恢复 standby。
+
+## Barge-in Playback Interruption
+
+第一版打断只解决 turn-based 播放中的用户抢话，不做电话式 full-duplex。实现边界如下：
+
+1. `MacSayTtsProvider` 产物优先转成 16 kHz/16-bit/mono PCM，使播放阶段可以切成约 100 ms 小段，避免整段 `afplay` 阻塞。
+2. `SpeakStage` 播放每个小段后检查 microphone queue。`OWNER_BARGE_IN_ECHO_GUARD_MS` 内只 flush/忽略，降低播报回声误触发。
+3. echo guard 之后，若 VAD 累计有效语音达到 `OWNER_BARGE_IN_MIN_SPEECH_MS` 且 realtime STT partial 有效，发出 `barge_in_detected` 并停止剩余句子/剩余回复播放。
+4. 已完整播放的句子文本才写入 process-local assistant context；未播出的句子和后续未播放 token 不写入上下文。
+5. 打断后的用户输入进入免唤醒 follow-up 捕获路径，保留实时字幕和 final STT，最终仍只有 final transcript 进入 LLM。
+
 ## Migration

 No database migration. Users should run:
@@ -485,6 +485,8 @@ standby
 14. `Real live check CLI`：新增 `real-live-check` 可重复命令，把完整真实 Provider 链路验收从一次性脚本升级为仓库内正式工具。
 15. `Real live check timing`：新增真实流程命令计时输出，覆盖命令发起时间、LLM 请求发送时间、各阶段响应耗时和 pipeline 事件时间线。
 16. `Realtime transcript idle endpoint`：新增已有实时字幕后 1.5 秒无新文字即结束录音的端点要求，避免用户已停说但 capture 继续等待 VAD/音色端点。
+17. `Automatic continuous dialog decision`：新增回复后自动判断继续/待机能力，使用规则优先、LLM 小分类兜底，不确定时默认回到待机。
+18. `Barge-in playback interruption`：新增播报中用户有效说话打断能力，使用 echo guard、最短语音时长和 realtime partial 共同确认，避免播放回声误触发。

 ### 删除项

@@ -510,6 +512,7 @@ standby
 12. M12：`real-live-check` 正式 CLI、文档、测试和真实命令验收完成后提交；仍保留物理麦克风 `run-live` 真人体验验收。
 13. M13：`real-live-check` 计时输出完成后提交，使后续排查可以直接看到每段耗时。
 14. M14：实时字幕停滞端点完成后提交，用户真实 `run-live` 若已有 partial 后 1.5 秒无新文字，应快速进入 final STT。
+15. M15：自动持续对话判断与播报打断完成后提交，用户真实 `run-live` 应能在助手反问时免唤醒继续回答，在助手已完成回答时自动恢复待机。

 估时：

@@ -135,6 +135,52 @@ The system SHALL provide a deterministic simulated microphone acceptance command
 - **WHEN** the simulated microphone transport runs out of frames before the expected turn completes
 - **THEN** the command SHALL fail with a structured error instead of waiting indefinitely

+### Requirement: Automatic continuous dialog decision
+The live runtime SHALL automatically decide after each assistant reply whether to continue listening for a direct follow-up answer or to end the continuous session and return to wake standby.
+
+#### Scenario: Assistant asks the user a direct follow-up question
+- **WHEN** an assistant reply clearly asks the user a question, requests missing information, asks the user to choose among options, or otherwise waits for the user's next answer
+- **THEN** the continuation decision SHALL be `continue` with confidence at or above the configured threshold
+- **AND** the runtime SHALL enter follow-up listening after playback without requiring the wake word again
+
+#### Scenario: Assistant completes the answer
+- **WHEN** an assistant reply only provides an answer, reports a result, refuses, reports an error, ends a greeting, or is ambiguous about needing more user input
+- **THEN** the continuation decision SHALL be `standby`
+- **AND** the runtime SHALL return to wake standby after playback
+
+#### Scenario: Hybrid decision falls back to an LLM classifier
+- **WHEN** deterministic rules cannot decide continue or standby
+- **THEN** the hybrid provider SHALL call the configured cloud LLM with a classifier prompt that can only resolve to `continue` or `standby`
+- **AND** a low-confidence, malformed, timed-out, or failed classifier result SHALL be treated as `standby`
+
+#### Scenario: Follow-up answer arrives in the listening window
+- **WHEN** the decision is `continue` and the user starts speaking within `OWNER_FOLLOWUP_LISTEN_TIMEOUT_MS`
+- **THEN** the runtime SHALL capture, realtime-transcribe, final-transcribe, send to LLM, synthesize, and play the answer without local wake detection and without the wake acknowledgement phrase
+
+#### Scenario: Follow-up listening times out
+- **WHEN** the decision is `continue` but no user speech is detected before `OWNER_FOLLOWUP_LISTEN_TIMEOUT_MS`
+- **THEN** the runtime SHALL emit `followup_timeout`
+- **AND** it SHALL emit `continuous_session_ended`
+- **AND** it SHALL return to wake standby
+
+### Requirement: Barge-in playback interruption
+The live runtime SHALL support a first-version barge-in path that can stop assistant playback when the user starts a valid new utterance during TTS playback.
+
+#### Scenario: Echo guard suppresses early playback echo
+- **WHEN** TTS playback starts and microphone input appears before `OWNER_BARGE_IN_ECHO_GUARD_MS`
+- **THEN** the runtime SHALL NOT treat that input as a user barge-in
+
+#### Scenario: Valid user speech interrupts playback
+- **WHEN** TTS playback has passed the echo guard, VAD observes at least `OWNER_BARGE_IN_MIN_SPEECH_MS` of user speech, and realtime STT produces a valid partial transcript
+- **THEN** the runtime SHALL emit `barge_in_detected`
+- **AND** it SHALL stop remaining playback chunks and emit `playback_interrupted`
+- **AND** it SHALL process the user's new utterance as the next input without requiring wake
+
+#### Scenario: Interrupted assistant context is partial
+- **WHEN** playback is interrupted before the full assistant reply is spoken
+- **THEN** the process-local conversation context SHALL append only the assistant text that had been fully spoken before interruption
+- **AND** unspoken assistant text SHALL NOT be appended to the session history
+
 ### Requirement: Real provider live fixture acceptance
 The system SHALL support scripted full-chain acceptance that drives the live pipeline with generated microphone PCM while using real local speech providers, the configured cloud LLM, and local TTS/playback capability.

@@ -118,3 +118,11 @@
 - [x] 15.2 实现配置与 capture 端点；前置条件：15.1 完成；验收标准：`AppConfig`、`.env.example`、`--show-config` 支持新配置，`VoiceAssistantPipeline` 在已有 partial 后 1500 ms 无新文本时调用 recorder finish；测试要点：配置解析和 show-config 单测；优先级：P0；预计：45 分钟。
 - [x] 15.3 补充 recorder finish 和回归测试；前置条件：15.2 完成；验收标准：持续有语音但 partial 不再推进时，segment end_reason 为 `partial_transcript_idle`，final STT 仍执行且 partial 不进上下文；测试要点：live runtime 单测；优先级：P0；预计：45 分钟。
 - [x] 15.4 执行门禁并提交“实时字幕端点”模块；前置条件：15.1 至 15.3 完成；验收标准：compileall、unittest、simulate-live、security-check、OpenSpec strict、git diff check 通过后中文提交；优先级：P0；预计：45 分钟。
+
+## 16. 自动持续对话判断与播报打断
+
+- [x] 16.1 更新 OpenSpec 和 README 描述自动持续对话、免唤醒追问和播报打断；前置条件：用户确认不再固定 3 秒追问窗口；验收标准：proposal/design/spec/tasks/README 明确 `ContinuationDecisionStage`、hybrid 规则、follow-up 3 秒窗口、barge-in echo guard 和默认 standby 策略；测试要点：OpenSpec strict；优先级：P0；预计：45 分钟。
+- [x] 16.2 实现配置、事件和续聊决策 Provider；前置条件：16.1 完成；验收标准：`AppConfig`、`.env.example`、`--show-config` 支持持续对话配置，新增事件可被 reporter/test 消费，规则优先和 LLM 兜底 provider 可单测；测试要点：规则 continue/standby、不确定低置信 standby、高置信 continue；优先级：P0；预计：60 分钟。
+- [x] 16.3 重构 `VoiceAssistantPipeline/TurnController` 支持自动 follow-up；前置条件：16.2 完成；验收标准：回复后先判定 continue/standby，continue 时播放完进入 `followup_listening`，3 秒内用户直接回答无需 wake/ACK，超时恢复 standby；测试要点：连续对话、follow-up timeout、上下文只写 final STT；优先级：P0；预计：60 分钟。
+- [x] 16.4 实现可打断播放路径；前置条件：16.3 完成；验收标准：TTS 播放前 500 ms 忽略回声，之后检测到至少 250 ms 有效用户语音且 realtime partial 有效时触发 `barge_in_detected -> playback_interrupted`，上下文只写已播出的 assistant 文本；测试要点：guard window 不打断、有效输入打断、未播文本不进上下文；优先级：P0；预计：60 分钟。
+- [x] 16.5 执行门禁并提交“连续对话判断”模块；前置条件：16.1 至 16.4 完成；验收标准：compileall、unittest、simulate-live --turns 3、real-live-check --turns 2 --no-playback、security-check、model-check、device-check、OpenSpec strict、git diff check 通过后提交；优先级：P0；预计：60 分钟。