[全双工规格补齐]:完成完整Agent语音助手OpenSpec,包含真实运行边界和自测标准
This commit is contained in:
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-06-19
|
||||
@@ -0,0 +1,63 @@
|
||||
# 完整全双工 Agent 运行时设计
|
||||
|
||||
## 背景
|
||||
|
||||
当前 `run-agent-live` 通过 `build_live_runtime(config).run_agent()` 复用旧 `VoiceAssistantPipeline`。这条路径只在 TTS 播放时启动临时监听,无法满足持续监听、持续识别、全局可取消和 WebRTC APM 音频底座要求。本设计新增独立 `FullDuplexAgentRuntime`,将旧 pipeline 降级为 `run-live` 专用稳定路径。
|
||||
|
||||
## 数据流
|
||||
|
||||
1. `SoundDeviceAudioTransport` 只负责设备 I/O。
|
||||
2. `AudioHub` 读取麦克风 raw frame,写 raw capture ring。
|
||||
3. `WebRtcAudioProcessingStage` 使用 render reference 处理 raw capture,写 processed capture ring。
|
||||
4. `ContinuousVadWorker` 消费 processed capture,发布 speech_start/speech_end。
|
||||
5. `StreamingSttWorker` 消费同一 processed capture,发布 partial/stable/final transcript。
|
||||
6. `InterruptController` 订阅 VAD 事件和 state machine,在 thinking/speaking/tool_running 中触发 cancellation。
|
||||
7. `ConversationManager` 收到 final transcript 后构建短期上下文、memory context、LLM request。
|
||||
8. LLM streaming delta 进入 sentence segmenter。
|
||||
9. Streaming TTS 产出 PCM chunk,`InterruptiblePlaybackQueue` 播放并写 render reference。
|
||||
10. Tool calls 进入 `ToolRouter`,结果回注入 LLM。
|
||||
|
||||
## 并发模型
|
||||
|
||||
后台任务:
|
||||
|
||||
1. `audio_capture_task`
|
||||
2. `apm_process_task`
|
||||
3. `vad_task`
|
||||
4. `streaming_stt_task`
|
||||
5. `conversation_task`
|
||||
6. `playback_task`
|
||||
7. `tool_task`,按需创建
|
||||
|
||||
所有任务使用 `CancellationGraph` 管理。用户打断时取消当前 response 子图,不关闭 audio hub;进程退出时取消 root token 并关闭所有订阅。
|
||||
|
||||
## 状态管理
|
||||
|
||||
状态机使用现有 `PipelineState` 枚举,不再在 `run-agent-live` 中用隐式 list 记录状态。每个转移必须发 event。非法转移在测试中失败。
|
||||
|
||||
## Provider 策略
|
||||
|
||||
1. APM:`webrtc` 为完整模式默认且 required;`fake` 仅 self-test;`disabled` 只允许开发降级,不算完整验收。
|
||||
2. STT:`faster_whisper` 为开发默认;`sherpa_onnx` fallback;`sensevoice` 预留。
|
||||
3. TTS:`cosyvoice` 为默认目标;缺失时完整模式失败,除非显式配置 fallback。
|
||||
4. Memory:`faiss_sqlite` 默认;如果 FAISS 不可用则 memory health error,基本对话可继续但 self-test 标记失败。
|
||||
5. Tools:默认启用 ToolRouter,但高风险 adapter 默认 disabled。
|
||||
|
||||
## 错误处理
|
||||
|
||||
1. 音频设备错误:启动失败或 recovery 到 idle。
|
||||
2. APM 不可用:完整模式启动失败。
|
||||
3. STT final 空文本:丢弃 utterance,回 listening。
|
||||
4. LLM/TTS 错误:取消 response,发 stage_error,回 listening。
|
||||
5. Tool high-risk:发 confirmation_required;第一版终端拒绝自动执行。
|
||||
6. Memory health error:禁用 memory context,继续基本对话并报告诊断。
|
||||
|
||||
## 自测设计
|
||||
|
||||
`agent-self-test` 使用 fake AudioHub/APM/STT/TTS/LLM/tool/memory,必须模拟三轮:
|
||||
|
||||
1. 正常问答。
|
||||
2. 播放时用户打断。
|
||||
3. memory.search tool call。
|
||||
|
||||
`audio-self-test` 检查真实设备、APM provider health、fake echo suppression、interrupt latency 统计。无真实 WebRTC binding 时输出失败 JSON,不伪造通过。
|
||||
@@ -0,0 +1,183 @@
|
||||
# 完整全双工 Agent 语音助手运行时
|
||||
|
||||
## 功能目标
|
||||
|
||||
当前 `run-agent-live` 已能进入无唤醒监听并在 TTS 播放阶段尝试后台打断,但真实架构仍是轮次式:先录完整句话,再 final STT,再 LLM,再 TTS 播放。播放期打断依赖临时 `AsyncBargeInMonitor` 从同一个 Transport 队列抢帧,无法保证麦克风输入、VAD、STT、回声抑制和播放取消之间的并发边界稳定。用户实际反馈“完全没有打断”,说明继续调参数不能解决根因。
|
||||
|
||||
本变更目标是把 `run-agent-live` 替换为完整全双工 Agent 运行时。目标用户是 macOS 本地桌面语音助手使用者,核心场景是用户可以直接说话、助手边生成边播报、用户随时插话打断、系统取消当前回复并立即处理新问题,同时保留长期记忆和安全工具调用能力。
|
||||
|
||||
量化成功指标:
|
||||
|
||||
1. 打断延迟:在自测 fixture 中,用户语音开始到播放停止的 P95 小于 200 ms。
|
||||
2. 回声隔离:纯助手回放 reference 不触发用户打断、不产生有效用户 transcript。
|
||||
3. 帧分发正确性:AudioHub 多消费者订阅时,VAD、STT、interrupt detector 不互相抢帧。
|
||||
4. 响应首音:LLM 输出首个可播报句子后,TTS 首个 PCM chunk 在 1000 ms 内进入播放队列。
|
||||
5. 自测覆盖:`agent-self-test --profile full-duplex --turns 3` 能无真人完成 STT、LLM、TTS、barge-in、memory、tool router 验收。
|
||||
6. 安全边界:工具调用默认执行低风险工具,高风险工具进入确认或拒绝;日志不泄露 `.env` 密钥、raw PCM 或敏感记忆。
|
||||
|
||||
## 详细需求
|
||||
|
||||
功能需求:
|
||||
|
||||
1. `run-agent-live` SHALL 使用新的 `FullDuplexAgentRuntime`,不得继续调用 `VoiceAssistantPipeline.run_agent_turn()`。
|
||||
2. `run-live` SHALL 保持旧 wake-word turn-based 稳定入口。
|
||||
3. 系统 SHALL 新增 `AudioHub`,作为唯一麦克风输入拥有者,内部维护 20 ms、16 kHz、mono、int16 capture ring buffer。
|
||||
4. VAD、STT、interrupt detector 和诊断录制 SHALL 从 AudioHub 获取独立订阅,不得直接抢读 Transport 队列。
|
||||
5. 系统 SHALL 新增 render ring buffer,播放 PCM 写入 speaker 的同时写入 AEC reference。
|
||||
6. `run-agent-live` SHALL 初始化 WebRTC APM;当 `OWNER_AUDIO_APM_PROVIDER=webrtc` 且 `OWNER_AUDIO_APM_REQUIRED=1` 时,真实 provider 不可用必须启动失败,不得静默回退为伪全双工。
|
||||
7. fake APM SHALL 只用于测试、自测和显式 `OWNER_AUDIO_APM_PROVIDER=fake`。
|
||||
8. 持续 VAD SHALL 在 listening/thinking/speaking/tool_running 阶段持续判断用户是否开始说话。
|
||||
9. Streaming STT SHALL 输出 partial、stable partial、final 三层事件;final 或明确提交 stable transcript 才能进入 LLM。
|
||||
10. 打断 SHALL 不依赖 partial STT;VAD + APM 后有效人声即可触发 cancellation。
|
||||
11. LLM、TTS、播放队列、可取消工具 SHALL 共享 cancellation graph;打断时全部取消。
|
||||
12. LLM SHALL 默认启用 streaming。
|
||||
13. TTS SHALL 使用可分块 PCM provider;CosyVoice 为目标 provider,macOS `say` 只做 fallback。
|
||||
14. 播放 SHALL 每 20-30 ms chunk 检查 cancellation,并把 render chunk 写入 APM reference。
|
||||
15. Conversation Manager SHALL 统一短期上下文、长期记忆、LLM、tool call、TTS 和打断恢复。
|
||||
16. MemoryManager SHALL 支持 SQLite 文本/元数据和 FAISS 向量索引;默认启用但敏感内容不自动保存。
|
||||
17. ToolRouter SHALL 接入 LLM tool call loop,第一批工具包括 `memory.search`、`memory.save`、`shell.readonly`、`openinterpreter.run`、`browser.playwright`。
|
||||
18. Open Interpreter、Playwright 默认关闭;显式启用后仍受风险分类、目录限制、超时、输出截断和确认策略约束。
|
||||
19. README 和 `.env.example` SHALL 统一说明 `run-agent-live` 是完整全双工入口,`run-live` 是旧入口。
|
||||
20. 新增 `agent-self-test` 和 `audio-self-test`,作为无人值守验收命令。
|
||||
|
||||
非功能需求:
|
||||
|
||||
1. 音频回调不得执行阻塞 STT、LLM、TTS 或工具逻辑。
|
||||
2. Capture/render ring buffer 必须有容量上限,溢出丢弃旧帧并发诊断事件。
|
||||
3. 所有后台线程/任务必须可关闭;测试后不得残留线程。
|
||||
4. 事件 payload 必须脱敏,不输出 API key、Authorization、raw PCM、完整工具敏感输出。
|
||||
5. 无 APM、无 STT 模型、无 TTS provider、无设备权限时必须给结构化错误。
|
||||
6. 旧 `simulate-live`、`real-live-check`、`run-live` 不得被新 runtime 破坏。
|
||||
|
||||
边缘案例:
|
||||
|
||||
1. 用户在 LLM 尚未开始播放时插话:取消 thinking,保留新用户音频,重启当前输入。
|
||||
2. 用户在 TTS 合成中插话:取消 TTS,未播文本不进上下文。
|
||||
3. 用户和助手声音重叠:APM 后用户声有效时触发打断;纯回声不得触发。
|
||||
4. 工具运行中插话:可取消工具取消;不可安全取消工具进入确认/等待结果后恢复。
|
||||
5. STT final 空文本:不调用 LLM,回 listening。
|
||||
6. Memory index 损坏:禁用长期记忆并发 health error,不影响基本对话。
|
||||
|
||||
## 设计方案
|
||||
|
||||
文字架构图:
|
||||
|
||||
```text
|
||||
Microphone
|
||||
-> AudioHub raw capture ring
|
||||
-> WebRtcAudioProcessingStage(AEC/NS/AGC, render reference)
|
||||
-> processed capture ring
|
||||
-> Continuous VAD
|
||||
-> Streaming STT
|
||||
-> ConversationManager
|
||||
-> MemoryManager(SQLite + FAISS)
|
||||
-> LLM streaming + ToolRouter
|
||||
-> SentenceSegmenter
|
||||
-> Streaming TTS
|
||||
-> InterruptiblePlaybackQueue
|
||||
-> Speaker + render reference ring
|
||||
```
|
||||
|
||||
主状态机:
|
||||
|
||||
```text
|
||||
idle -> listening -> thinking -> speaking -> listening
|
||||
speaking -> interrupted -> listening
|
||||
thinking -> interrupted -> listening
|
||||
tool_running -> interrupted/recovering/listening
|
||||
any recoverable error -> recovering -> listening
|
||||
```
|
||||
|
||||
关键接口:
|
||||
|
||||
1. `AudioHub.subscribe(kind: str) -> AudioSubscription`
|
||||
- 参数:`kind` 为 `processed_capture`、`raw_capture`、`render_reference`、`debug`。
|
||||
- 返回:独立 cursor 的 frame reader。
|
||||
- 错误:设备缺失、buffer closed、format mismatch。
|
||||
2. `AudioProcessingProvider.process_capture(frame) -> AudioFrame`
|
||||
- 输入:raw capture frame。
|
||||
- 输出:AEC/NS/AGC 后 frame。
|
||||
- 错误:`AUDIO_APM_UNAVAILABLE`、`AUDIO_APM_PROCESS_FAILED`。
|
||||
3. `StreamingSttSession.accept_frame(frame) -> list[TranscriptEvent]`
|
||||
- 输出:partial/stable/final。
|
||||
- 错误:模型缺失、推理失败、取消。
|
||||
4. `StreamingTtsSession.accept_text(text) -> list[AudioFrame]`
|
||||
- 输出:可播放 PCM chunks。
|
||||
- 错误:provider 缺失、合成失败、取消。
|
||||
5. `ToolRouter.route(call) -> ToolDecision`
|
||||
- 输出:execute/reject/require_confirmation。
|
||||
|
||||
性能优化路径:
|
||||
|
||||
1. 音频输入只写 ring buffer,不做重计算。
|
||||
2. STT 和 VAD 消费 processed capture ring,互不阻塞。
|
||||
3. 播放 chunk 与 render reference 同步写入,避免打断检测依赖播放回调抢帧。
|
||||
4. LLM streaming + sentence segmentation 让首句尽早进入 TTS。
|
||||
5. Self-test 记录 interrupt latency、STT first partial、TTS first chunk。
|
||||
|
||||
UI/UX 路径:
|
||||
|
||||
1. 终端输出仍由 event bus 驱动。
|
||||
2. 打断事件输出 `检测到用户打断`、`取消当前回复`、`继续听你说`。
|
||||
3. debug 模式输出 APM provider、echo suppression、interrupt latency。
|
||||
4. 未来 GUI 桌宠订阅同一事件总线。
|
||||
|
||||
## 风险与权衡
|
||||
|
||||
| 风险 | 概率 | 影响 | 缓解措施 |
|
||||
|---|---:|---:|---|
|
||||
| macOS 上没有可用 WebRTC APM Python binding | 高 | 高 | 完整模式启动失败并说明安装要求;self-test 使用 fake APM;旧 `run-live` 保留 |
|
||||
| Faster Whisper 流式能力不是真正低延迟 streaming | 中 | 中 | 先实现稳定 provider contract;允许 sherpa-onnx fallback;SenseVoice 作为后续 provider |
|
||||
| CosyVoice 本地依赖重、安装慢 | 高 | 中 | provider 可配置;macOS say fallback 标记为降级,不宣称完整全双工默认 |
|
||||
| AudioHub 并发复杂导致线程泄漏 | 中 | 高 | 所有后台任务使用 cancellation graph;单元测试检查 shutdown |
|
||||
| AEC 质量不足仍误触发 | 中 | 高 | 增加 echo self-test、render drift 诊断、阈值调试输出 |
|
||||
| FAISS/SQLite 不一致 | 中 | 中 | health-check 和 checksum;不返回未验证记忆 |
|
||||
| 工具调用误执行高风险动作 | 中 | 高 | 默认关闭 Open Interpreter/Playwright;风险分类和确认策略;只读 shell allowlist |
|
||||
| 旧 run-live 回归 | 低 | 高 | 保持代码路径分离;旧测试继续跑 |
|
||||
|
||||
## 任务分解
|
||||
|
||||
详见 `tasks.md`。所有任务拆到不超过 1 小时,按 OpenSpec、音频底座、持续识别与打断、流式回复、记忆工具、自测文档分阶段完成。每个阶段完成后验证并中文 commit。
|
||||
|
||||
## Spec Deltas
|
||||
|
||||
新增:
|
||||
|
||||
1. `Complete full-duplex agent runtime`
|
||||
2. `AudioHub fanout`
|
||||
3. `Required WebRTC APM for full-duplex`
|
||||
4. `Always-on interruption controller`
|
||||
5. `Streaming STT/TTS runtime`
|
||||
6. `Conversation manager with memory and tools`
|
||||
7. `Self-test commands`
|
||||
|
||||
修改:
|
||||
|
||||
1. `Live run-agent-live runtime` 从“可真实启动并播放期打断”升级为“完整全双工主入口”。
|
||||
2. `WebRTC audio processing foundation` 从规划/fake provider 验证升级为运行时启动要求。
|
||||
3. `Tool Router` 从模块级测试升级为 LLM tool-call 主链路要求。
|
||||
|
||||
删除/推翻:
|
||||
|
||||
1. 推翻 `AsyncBargeInMonitor` 作为主打断架构;保留为 legacy fallback 或删除。
|
||||
2. 推翻 `run-agent-live` 复用 `VoiceAssistantPipeline.run_agent_turn()` 的实现边界。
|
||||
3. 推翻“真实运行仍走 run-live”的 README 旧描述。
|
||||
|
||||
## 实施计划
|
||||
|
||||
1. M1 OpenSpec:0.5 天。
|
||||
2. M2 AudioHub + APM:1-1.5 天。
|
||||
3. M3 Continuous VAD/STT + interrupt:1-2 天。
|
||||
4. M4 Streaming response/TTS/playback:1 天。
|
||||
5. M5 Memory + Tool Router:1-1.5 天。
|
||||
6. M6 self-test/docs/final gates:0.5-1 天。
|
||||
|
||||
乐观总耗时 5 天,最可能 7 天,悲观 10 天。没有数据迁移;长期记忆首次启用时创建 SQLite/FAISS 文件,旧短期上下文不迁移。
|
||||
|
||||
## Git 提交规范
|
||||
|
||||
每完成一个主要阶段必须立即验证并提交,提交信息格式:
|
||||
|
||||
`[模块名]:完成[具体功能描述],包含[关键变更]`
|
||||
|
||||
提交前至少运行该阶段相关单测;最终阶段运行完整门禁并保证 `git status --short` 为空。
|
||||
@@ -0,0 +1,96 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Complete full-duplex agent runtime
|
||||
The system SHALL provide a complete `run-agent-live` runtime that uses a dedicated full-duplex agent architecture instead of the legacy turn-based `VoiceAssistantPipeline.run_agent_turn()` loop.
|
||||
|
||||
#### Scenario: Full-duplex entry starts the new runtime
|
||||
- **WHEN** the user runs `.venv/bin/python -m owner_voice_pet run-agent-live`
|
||||
- **THEN** the command SHALL construct `FullDuplexAgentRuntime`
|
||||
- **AND** it SHALL NOT call `VoiceAssistantPipeline.run_agent_turn()` as the primary runtime path
|
||||
|
||||
#### Scenario: Legacy runtime remains available
|
||||
- **WHEN** the user runs `.venv/bin/python -m owner_voice_pet run-live`
|
||||
- **THEN** the legacy wake-word turn-based runtime SHALL remain available
|
||||
|
||||
### Requirement: AudioHub fanout
|
||||
The full-duplex runtime SHALL use a single AudioHub to own microphone input and distribute frames to independent consumers without frame stealing.
|
||||
|
||||
#### Scenario: Multiple consumers subscribe
|
||||
- **WHEN** VAD, streaming STT, interrupt detection, and diagnostics subscribe to processed capture audio
|
||||
- **THEN** each consumer SHALL receive the same ordered audio frame sequence from its own cursor
|
||||
|
||||
#### Scenario: Ring buffer overflows
|
||||
- **WHEN** capture or render ring buffer exceeds configured capacity
|
||||
- **THEN** the hub SHALL drop oldest safe frames, emit an overrun diagnostic, and continue without blocking audio callbacks
|
||||
|
||||
### Requirement: Required WebRTC APM in full-duplex mode
|
||||
The full-duplex runtime SHALL process microphone capture through WebRTC AEC, NS, and AGC using playback render reference before VAD, STT, or interrupt detection consume frames.
|
||||
|
||||
#### Scenario: APM is unavailable
|
||||
- **WHEN** `OWNER_AUDIO_APM_PROVIDER=webrtc`, `OWNER_AUDIO_APM_REQUIRED=1`, and no real WebRTC APM provider is available
|
||||
- **THEN** `run-agent-live` SHALL fail startup with a structured audio APM error
|
||||
- **AND** it SHALL NOT silently run the software barge-in fallback as complete full-duplex mode
|
||||
|
||||
#### Scenario: Fake APM is used in self-test
|
||||
- **WHEN** `agent-self-test` or unit tests request fake APM
|
||||
- **THEN** fake APM SHALL be allowed and SHALL be clearly reported as test-only
|
||||
|
||||
#### Scenario: Pure assistant echo occurs
|
||||
- **WHEN** microphone input is only assistant render echo
|
||||
- **THEN** APM-processed audio SHALL NOT trigger valid interruption or valid user transcript
|
||||
|
||||
### Requirement: Always-on interruption controller
|
||||
The full-duplex runtime SHALL run a continuous interruption controller that can cancel active response work from `thinking`, `speaking`, or cancellable `tool_running` states.
|
||||
|
||||
#### Scenario: User interrupts while speaking
|
||||
- **WHEN** the assistant is playing audio and processed capture contains valid user speech
|
||||
- **THEN** playback SHALL stop, active LLM/TTS work SHALL be cancelled, and the state SHALL transition through `interrupted` back to `listening`
|
||||
|
||||
#### Scenario: Interruption latency is measured
|
||||
- **WHEN** self-test injects user speech during playback
|
||||
- **THEN** the runtime SHALL report VAD-to-playback-stop latency and target P95 under 200 ms
|
||||
|
||||
#### Scenario: Interruption does not require STT partial
|
||||
- **WHEN** valid user speech is detected before streaming STT emits partial text
|
||||
- **THEN** playback cancellation SHALL still occur
|
||||
|
||||
### Requirement: Streaming STT and streaming TTS runtime
|
||||
The full-duplex runtime SHALL use streaming recognition and streaming speech output contracts for the agent path.
|
||||
|
||||
#### Scenario: User speaks during listening
|
||||
- **WHEN** processed capture contains user speech
|
||||
- **THEN** streaming STT SHALL emit partial or stable transcript events while speech continues and final transcript when the utterance ends
|
||||
|
||||
#### Scenario: LLM emits a sentence
|
||||
- **WHEN** the LLM stream produces a complete safe speech segment
|
||||
- **THEN** streaming TTS SHALL synthesize playable PCM chunks without waiting for the full assistant response
|
||||
|
||||
#### Scenario: Playback is interrupted
|
||||
- **WHEN** cancellation occurs during streaming TTS or playback
|
||||
- **THEN** unplayed audio chunks SHALL be discarded and unspoken assistant text SHALL NOT be committed to context
|
||||
|
||||
### Requirement: Full-duplex Conversation Manager
|
||||
The full-duplex runtime SHALL coordinate final user transcripts, session context, long-term memory retrieval, LLM streaming, tool calls, TTS playback, interruption, and recovery.
|
||||
|
||||
#### Scenario: Memory is enabled
|
||||
- **WHEN** a final user transcript is accepted and memory is enabled
|
||||
- **THEN** the Conversation Manager SHALL retrieve Top-K relevant memories and provide them as separate context to the LLM
|
||||
|
||||
#### Scenario: Tool call is requested
|
||||
- **WHEN** the LLM stream requests a supported tool call
|
||||
- **THEN** the Tool Router SHALL classify, approve, reject, or request confirmation before execution
|
||||
|
||||
#### Scenario: Tool is high risk
|
||||
- **WHEN** a tool request involves deletion, upload, payment, account changes, installation, or other high-risk action
|
||||
- **THEN** the runtime SHALL NOT execute it automatically
|
||||
|
||||
### Requirement: Full-duplex self-test commands
|
||||
The system SHALL provide deterministic self-test commands for the complete full-duplex agent runtime.
|
||||
|
||||
#### Scenario: Agent self-test runs
|
||||
- **WHEN** the developer runs `.venv/bin/python -m owner_voice_pet agent-self-test --profile full-duplex --turns 3`
|
||||
- **THEN** the command SHALL exercise STT, LLM, TTS, interruption, memory, and Tool Router paths and return JSON with `success=true` only when all checks pass
|
||||
|
||||
#### Scenario: Audio self-test runs
|
||||
- **WHEN** the developer runs `.venv/bin/python -m owner_voice_pet audio-self-test --duration 10 --check-echo`
|
||||
- **THEN** the command SHALL report device availability, APM provider health, echo suppression result, and interruption latency statistics
|
||||
@@ -0,0 +1,45 @@
|
||||
## 1. OpenSpec 与运行边界
|
||||
|
||||
- [x] 1.1 新建 `complete-full-duplex-agent-runtime` OpenSpec;前置条件:当前计划已确认;优先级:P0;验收标准:proposal/design/tasks/spec delta 存在;测试要点:`openspec validate complete-full-duplex-agent-runtime --strict`。
|
||||
- [x] 1.2 明确 `run-agent-live` 替换旧 `VoiceAssistantPipeline.run_agent_turn()`;前置条件:1.1;优先级:P0;验收标准:文档和任务说明旧路径为 legacy;测试要点:README 后续无冲突描述。
|
||||
- [x] 1.3 Phase 1 提交;前置条件:1.1-1.2;优先级:P0;验收标准:中文提交 `[全双工规格补齐]...`;测试要点:OpenSpec strict 和 git status。
|
||||
|
||||
## 2. AudioHub 与 WebRTC APM
|
||||
|
||||
- [ ] 2.1 实现 `AudioHub` 和独立订阅;前置条件:Phase 1;优先级:P0;验收标准:多消费者不抢帧;测试要点:两个订阅读取相同 frame 序列。
|
||||
- [ ] 2.2 实现 capture/render ring buffer 溢出诊断;前置条件:2.1;优先级:P0;验收标准:容量上限、丢旧帧、事件记录;测试要点:overflow fixture。
|
||||
- [ ] 2.3 接入 APM provider 到 AudioHub;前置条件:2.1;优先级:P0;验收标准:raw capture 经过 APM 后进入 processed ring;测试要点:fake APM echo suppression。
|
||||
- [ ] 2.4 `run-agent-live` 启动时检查真实 APM;前置条件:2.3;优先级:P0;验收标准:required 且不可用时结构化失败;测试要点:provider unavailable test。
|
||||
- [ ] 2.5 Phase 2 提交;前置条件:2.1-2.4;优先级:P0;验收标准:中文提交 `[音频底座重构]...`;测试要点:compileall、相关单测。
|
||||
|
||||
## 3. Continuous VAD/STT 与打断
|
||||
|
||||
- [ ] 3.1 实现 `InterruptController` 常驻检测;前置条件:AudioHub;优先级:P0;验收标准:speaking 中 VAD 命中触发 cancel;测试要点:latency fixture。
|
||||
- [ ] 3.2 接入 cancellation graph 到 LLM/TTS/playback;前置条件:3.1;优先级:P0;验收标准:打断取消所有 response 子任务;测试要点:幂等 cancel。
|
||||
- [ ] 3.3 实现 streaming STT worker;前置条件:AudioHub;优先级:P0;验收标准:partial/stable/final 事件;测试要点:final 进入 conversation,partial 不进上下文。
|
||||
- [ ] 3.4 打断后复用 buffered user audio;前置条件:3.1-3.3;优先级:P0;验收标准:不需要重新唤醒;测试要点:`speaking -> interrupted -> listening`。
|
||||
- [ ] 3.5 Phase 3 提交;前置条件:3.1-3.4;优先级:P0;验收标准:中文提交 `[全双工打断]...`;测试要点:相关单测和 self-test 子集。
|
||||
|
||||
## 4. Streaming LLM/TTS/Playback
|
||||
|
||||
- [ ] 4.1 默认启用 LLM streaming;前置条件:Phase 3;优先级:P0;验收标准:`run-agent-live --check-config` 显示 streaming true;测试要点:配置测试。
|
||||
- [ ] 4.2 实现 streaming TTS provider wrapper;前置条件:4.1;优先级:P0;验收标准:句子进入 TTS 后输出 PCM chunks;测试要点:fake/cosyvoice fallback。
|
||||
- [ ] 4.3 播放队列写 render reference;前置条件:4.2;优先级:P0;验收标准:播放 chunk 同步进入 APM reference;测试要点:render ring frame count。
|
||||
- [ ] 4.4 只提交已播 assistant 文本;前置条件:4.3;优先级:P0;验收标准:中途打断不写未播文本;测试要点:上下文断言。
|
||||
- [ ] 4.5 Phase 4 提交;前置条件:4.1-4.4;优先级:P0;验收标准:中文提交 `[流式回复播放]...`;测试要点:compileall、单测。
|
||||
|
||||
## 5. Memory 与 Tool Router
|
||||
|
||||
- [ ] 5.1 将 `AgentConversationManager` 接入 runtime;前置条件:Phase 4;优先级:P0;验收标准:memory context 注入 LLM;测试要点:相关记忆被召回。
|
||||
- [ ] 5.2 接入 FAISS/SQLite health;前置条件:5.1;优先级:P0;验收标准:memory enabled 时检查 index;测试要点:缺失/不一致错误。
|
||||
- [ ] 5.3 ToolRouter 接入 LLM tool calls;前置条件:5.1;优先级:P0;验收标准:memory.search tool result 回注入回复;测试要点:tool call integration。
|
||||
- [ ] 5.4 高风险工具确认/拒绝;前置条件:5.3;优先级:P0;验收标准:Open Interpreter/Playwright 默认不自动执行;测试要点:高风险 fixture。
|
||||
- [ ] 5.5 Phase 5 提交;前置条件:5.1-5.4;优先级:P0;验收标准:中文提交 `[Agent记忆工具]...`;测试要点:memory/tool/security 单测。
|
||||
|
||||
## 6. 自我测试、文档与最终验收
|
||||
|
||||
- [ ] 6.1 新增 `agent-self-test`;前置条件:Phase 5;优先级:P0;验收标准:三轮覆盖 STT/LLM/TTS/barge-in/memory/tool;测试要点:JSON success。
|
||||
- [ ] 6.2 新增 `audio-self-test`;前置条件:AudioHub/APM;优先级:P0;验收标准:输出 provider/device/echo/latency;测试要点:无 APM 时明确失败。
|
||||
- [ ] 6.3 更新 README 和 `.env.example`;前置条件:6.1-6.2;优先级:P0;验收标准:入口和配置无冲突;测试要点:命令示例准确。
|
||||
- [ ] 6.4 执行最终门禁;前置条件:全部实现完成;优先级:P0;验收标准:compileall、unittest、self-test、security、model、device、OpenSpec strict 全通过;测试要点:命令输出。
|
||||
- [ ] 6.5 Phase 6 提交;前置条件:6.1-6.4;优先级:P0;验收标准:中文提交 `[全双工自测]...`;测试要点:`git status --short` 为空。
|
||||
Reference in New Issue
Block a user