Owner/openspec/changes/separate-wake-and-realtime-transcript/design.md

# 独立本地唤醒与终端转写显示设计

## Context

真实运行日志显示，当前 `run-live` 会先截取语音段并使用 STT 判断是否包含“小杰小杰”。该实现把 wake detection 与 user utterance transcription 绑定在一起，导致唤醒慢、唤醒词污染正式问题、终端输出难以区分 STT 和 LLM 阶段。本设计把 wake word detection 升级为本地 KWS 模型路径，并把正式问题转写作为独立可见事件输出。

## Goals

1. 唤醒词“小杰小杰”由本地模型检测。
2. wake 阶段不调用云 ASR，也不调用正式 STT provider。
3. wake 命中后才开始正式问题录音和 STT。
4. 终端在 LLM 前显示正式问题转写文本。
5. 模型下载和检查覆盖 wake KWS、VAD、STT。
6. 自动化测试证明 wake/STT 分离、重复对话、上下文和错误恢复仍正常。

## Non-Goals

1. 不实现 GUI 桌宠窗口。
2. 不实现跨进程长期记忆。
3. 不在本阶段实现逐字 partial ASR 字幕；本阶段先保证正式问题 STT 完成后立即可见，且出现在 LLM 前。
4. 不把连续麦克风流上传到云端。

## Architecture

```text
run-live
  -> AppConfig(.env)
  -> VoiceAssistantPipeline
  -> PipelineEventBus
  -> TurnController
  -> SoundDeviceAudioTransport
  -> SherpaOnnxKeywordWakeWordProvider(local models/wake)
  -> AcknowledgeStage(local "我在")
  -> CaptureStage(raw mic frames)
  -> AudioPreprocessStage(local GTCRN denoise, capture only)
  -> PrimarySpeakerEndpoint/VAD(denoised frames)
  -> SherpaOnnxSttProvider(local CTC partial/final by default)
  -> DialogStage(session memory)
  -> ConversationContext
  -> OpenAICompatibleLlmProvider
  -> MacSayTtsProvider
  -> speaker playback
```

## Runtime Lifecycle

1. Load config.
2. Validate `OWNER_WAKE_PROVIDER=local_kws`.
3. Load KWS model from `OWNER_SPEECH_MODELS_DIR`.
4. Load VAD, STT, LLM, TTS.
5. Open microphone stream.
6. Wait for KWS wake event by feeding frames directly into wake provider.
7. On wake hit, reset wake stream and VAD recorder.
8. Record user utterance with VAD. The default provider is `hybrid`: project-local `sherpa-onnx` VAD remains the primary detector, and an energy threshold fallback prevents low microphone gain from being treated as no speech.
9. When `OWNER_ENDPOINT_MODE=primary_speaker`, build a temporary per-turn speaker profile from the first valid user speech frames and end the capture when the primary speaker is absent for `OWNER_SPEAKER_ABSENT_MS`.
10. When `OWNER_NOISE_FILTER_ENABLED=1`, pass formal user utterance frames through the local GTCRN denoiser before VAD, realtime STT, and final STT segment assembly.
11. Transcribe user utterance with the configured STT provider. The default is local CTC STT; cloud STT remains an explicit compatibility option.
12. Emit transcript event to terminal.
13. Append final user text and call LLM.
14. Synthesize/play reply with local TTS by default.
15. Append assistant reply and return to standby.

## Interfaces

### `SherpaOnnxKeywordWakeWordProvider`

```python
class SherpaOnnxKeywordWakeWordProvider:
    def __init__(self, models_dir, keyword, keywords_file=None, threshold=0.25, score=1.0, sherpa_module=None): ...
    def load(self) -> None: ...
    def detect(self, frame: AudioFrame) -> WakeEvent | None: ...
    def reset(self) -> None: ...
```

### `RuntimeReporter`

```python
class RuntimeReporter(Protocol):
    def status(self, state: str, message: str, *, turn_id: int | None = None) -> None: ...
    def transcript(self, text: str, *, final: bool, turn_id: int | None = None) -> None: ...
    def error(self, stage: str, code: str, message: str, *, turn_id: int | None = None) -> None: ...
```

### Pipeline events

```python
class PipelineEventBus:
    def subscribe(self, listener): ...
    def emit(self, event_type, *, turn_id=None, state=None, message="", payload=None): ...
```

Required event types are `pipeline_started`, `wake_listening`, `wake_detected`, `ack_started`, `capture_started`, `speech_started`, `speech_ended`, `stt_started`, `transcript_final`, `llm_started`, `tts_started`, `playback_finished`, `standby_resumed`, and `stage_error`.

### `AudioPreprocessor`

```python
class AudioPreprocessor:
    def load(self) -> None: ...
    def reset(self) -> None: ...
    def process_frame(self, frame: AudioFrame) -> AudioFrame: ...
    def flush(self) -> list[AudioFrame]: ...
```

`NoopAudioPreprocessor` preserves tests and disabled configurations. `SherpaOnnxDenoiserPreprocessor` uses `sherpa_onnx.OnlineSpeechDenoiser` with an `OfflineSpeechDenoiserGtcrnModelConfig` pointing to `models/denoise/gtcrn_simple.onnx`. It converts int16 PCM to float32, calls `run(samples, sample_rate)`, converts returned `DenoisedAudio.samples` back to int16 PCM, and adds diagnostic metadata without storing audio.

The first version applies preprocessing only in capture. Wake listening remains raw by default because wake KWS models can be sensitive to spectral changes introduced by denoising. `OWNER_WAKE_DENOISE_ENABLED=1` reserves an explicit future switch for wake preprocessing.

### Local CTC STT

`SherpaOnnxSttProvider` reads `providers.stt.type` from `models/manifest.json`:

1. `sherpa-onnx-streaming-transducer`: existing `tokens`/`encoder`/`decoder`/`joiner` files and `OnlineRecognizer.from_transducer`.
2. `sherpa-onnx-streaming-zipformer2-ctc`: new default `tokens`/`model` files and `OnlineRecognizer.from_zipformer2_ctc`.

Realtime partial and final transcript use the same local recognizer when `OWNER_SPEECH_PROVIDER=local`. Partial output is filtered before emitting `transcript_partial`: texts with fewer than two meaningful characters, duplicate texts, and short regressions from the last displayed text are ignored. The final transcript remains the only user text appended to `ConversationContext`.

### `TurnController`

```python
class TurnController:
    def run_turn(self, turn_id: int) -> TurnResult: ...
```

The controller owns one turn state machine and delegates work to provider-backed stages. It does not write terminal text directly; it emits events only.

### Primary speaker endpoint

The first implementation is a per-turn heuristic endpoint, not persistent voiceprint recognition. It extracts local PCM features from speech frames and compares future frames against the profile. If profile creation fails because the user speech is too short or too quiet, capture falls back to existing VAD silence endpoint.

## Low Latency Capture Revision

真人运行反馈显示，当前 capture 仍可能把用户第一句话开头吞掉或需要用户重复提问才能结束。本修正把低延迟 capture 作为 pipeline 内部约束：

1. `OWNER_POST_PLAYBACK_DRAIN_MS` 默认改为 `0`。ACK 播放结束后仅清掉播放期间积压在输入队列中的帧，不再主动等待并丢弃后续音频。
2. `SoundDeviceAudioTransport.read_frames()` 在拿到首帧后立即返回队列中所有可用帧，避免真实麦克风回调积压时 pipeline 逐帧追赶。
3. `OWNER_SPEAKER_PROFILE_MIN_MS` 控制临时主说话人画像最低就绪语音长度，默认 `120` ms，不再复用 `OWNER_VAD_MIN_DURATION_MS`。
4. 主说话人画像就绪后，`OWNER_SPEAKER_ABSENT_MS` 是结束正式问题采集的主条件；主说话人连续缺席达到该值后直接进入 STT，不再额外等待普通 VAD 最小时长。
5. 普通 VAD 静音仍作为画像不足或音色特征不可用时的兜底，最大录音时长仍作为最终保护。

## Realtime Transcript Revision

录音期间新增 partial transcript 通道，用于解决“说话时看不到文字结果”的体验问题：

1. Pipeline 事件新增 `transcript_partial`。终端 reporter 将其显示为 `实时转写：<文本>`，最终结果仍显示为 `转写结果：<文本>`。
2. partial transcript 只用于用户反馈，不写入 `ConversationContext`，不触发 LLM；LLM 仍只消费 final STT 结果。
3. 当 final ASR/TTS 走 cloud 时，partial transcript 使用本地 `sherpa-onnx` streaming STT，避免对云端 ASR 高频请求。
4. `VoiceAssistantPipeline` 在 `speech_started` 后把 capture 阶段已开始录音的帧 feed 给 realtime STT session；当 partial 文本变化时才 emit，避免刷屏。
5. `OWNER_REALTIME_TRANSCRIPT_ENABLED=1` 默认启用；设置为 `0` 可临时回退到只显示 final transcript。

## Realtime Transcript Idle Endpoint

真人 `run-live` 日志显示，用户已看到实时字幕完整推进后，capture 仍可能继续等待 VAD 或主说话人端点。新增 `OWNER_REALTIME_TRANSCRIPT_IDLE_TIMEOUT_MS` 作为文本级端点，默认 `1500` ms：

1. CaptureStage 只在已经输出过至少一次有效 `transcript_partial` 后启用该端点。
2. 每次 `transcript_partial` 真实发出时，记录该帧的音频时间戳为 `last_partial_ms`。
3. 后续已开始录音但没有新的 partial 文本输出时，如果当前帧时间戳与 `last_partial_ms` 的差值达到配置阈值，调用 `VadRecorder.finish("partial_transcript_idle")` 构建当前录音片段。
4. 构建出的 segment 继续走 final STT、LLM、TTS，不把 partial 文本写入上下文。
5. 配置为 `0` 时关闭该端点，保留只依赖主说话人/VAD 的旧行为。

该端点使用音频帧时间戳而不是 wall-clock，避免批量读帧或 CPU 抖动导致误判。它不会在没有 partial 的场景提前结束，避免 STT 尚未给出第一段文本时切掉用户开头。

## Model Files

```text
models/
  manifest.json
  wake/
    sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01-mobile/
      tokens.txt
      encoder-epoch-12-avg-2-chunk-16-left-64.int8.onnx
      decoder-epoch-12-avg-2-chunk-16-left-64.onnx
      joiner-epoch-12-avg-2-chunk-16-left-64.int8.onnx
    keywords.txt
  vad/
    silero_vad.onnx
  stt/
    sherpa-onnx-streaming-zipformer-ctc-zh-int8-2025-06-30/
      tokens.txt
      model.int8.onnx
  denoise/
    gtcrn_simple.onnx
```

## Error Handling

1. Missing KWS model: `WAKE_MODEL_MISSING`.
2. KWS load failure: `WAKE_MODEL_LOAD_FAILED`.
3. KWS runtime failure: `WAKE_MODEL_LOAD_FAILED` with retryable true.
4. Empty user STT: existing `STT_EMPTY_TRANSCRIPT`.
5. Invalid wake provider config: `CONFIG_MISSING_VALUE`.
6. Real microphone VAD miss: default `OWNER_VAD_PROVIDER=hybrid` SHALL accept speech when either the local model or the energy fallback detects speech.
7. Primary speaker endpoint profile failure: fallback to VAD silence endpoint.
8. Pipeline stage failure: emit `stage_error`, recover to standby, and keep the process alive unless startup dependencies are missing.
9. Playback drain misconfiguration: negative `OWNER_POST_PLAYBACK_DRAIN_MS` remains invalid; non-zero values are treated as explicit user tuning rather than default behavior.
10. Speaker profile threshold misconfiguration: non-positive `OWNER_SPEAKER_PROFILE_MIN_MS` fails config validation.
11. Realtime STT failure: startup model缺失按 `model-check` 暴露；capture 中 partial session 失败不得把 partial 文本写入上下文。
12. Denoiser missing model: `model-check` reports `denoise/gtcrn_simple.onnx` and live startup fails before microphone listening.
13. Denoiser runtime failure: the current turn emits `stage_error` and recovers to standby without invoking final STT or LLM.
14. CTC manifest mismatch: missing `model` or `tokens` files report `STT_MODEL_MISSING`; transducer manifests remain supported for existing local setups.
15. Partial text noise: single-character or transient partial output is ignored rather than displayed as terminal feedback.

## Testing Strategy

1. Unit test fake wake provider detects wake without STT calls.
2. Unit test repeated runtime runs two turns with exactly two STT calls.
3. Unit test terminal reporter records transcript before LLM stage.
4. Unit test LLM user content excludes wake keyword.
5. Unit test KWS provider missing model raises structured error.
6. Model-check test validates wake required files.
7. Pipeline event order test validates successful two-turn event sequence.
8. Primary speaker endpoint test validates that background noise or a later repeated utterance does not extend the current turn after the main speaker disappears.
9. First utterance preservation test validates that frames immediately after ACK are not discarded by post-playback drain.
10. Low-latency endpoint test validates that a short first question ends by primary speaker absence without waiting for a repeated second question.
11. Transport batching test validates that queued SoundDevice frames are returned together.
12. Partial transcript event test validates that realtime text appears after speech start and before final transcript.
13. Context isolation test validates partial transcript does not enter LLM messages.
14. Denoiser manifest/model-check test validates the required GTCRN file is present.
15. Fake denoiser test validates VAD, partial STT, and final STT receive denoised frames from the same processed stream.
16. Local speech provider test validates `OWNER_SPEECH_PROVIDER=local` does not construct cloud ASR/TTS providers.
17. Partial filter test validates single-character and short transient results are not emitted.
18. CTC STT loading test validates manifest type selects `from_zipformer2_ctc` and does not require transducer encoder/decoder/joiner files.
19. Simulated microphone live acceptance validates the current `VoiceAssistantPipeline` with generated wake frames, formal question frames, noisy short partials, background speech, two repeated turns, session context, TTS playback, and standby recovery.
20. Fixture replay test validates the generated simulated microphone JSONL can be written and replayed for deterministic debugging.

## Simulated Microphone Acceptance

`owner_voice_pet simulate-live` is a deterministic no-device acceptance runner. It does not replace `run-live` with a real microphone, but it closes the gap between small unit tests and human testing by feeding realistic ordered audio frames into the same live pipeline controller.

```text
simulate-live
  -> generated AudioFrame sequence
  -> BoundedMemoryAudioTransport(simulated microphone)
  -> VoiceAssistantPipeline
  -> KeywordWakeWordProvider(metadata wake)
  -> SimulatedNoiseFilter(metadata + PCM passthrough)
  -> PrimarySpeakerVadRecorder
  -> MetadataSttProvider(partial + final)
  -> MockLlmProvider
  -> SineTtsProvider
  -> JSON checks
```

The generated sequence intentionally contains:

1. one wake frame per turn;
2. six primary speaker frames per question, enough to establish the temporary profile;
3. short noisy partials `家` and `家确`, followed by stable question partials;
4. fifteen background speech frames after the primary speaker stops, enough to trigger the 300 ms primary-speaker absence endpoint;
5. a second wake/question sequence to prove recovery to standby and temporary context continuity.

The transport is bounded: if frames are exhausted unexpectedly, it raises a structured validation error instead of letting the pipeline loop forever. `--write-fixture` writes the exact simulated microphone frames as JSONL, and `--fixture` replays them for repeatable debugging.

## Real Provider Fixture Acceptance and Playback Drain

The simulated microphone command validates pipeline ordering with fake providers, but it does not prove that real local models and real playback can run together. The next acceptance layer uses generated 16 kHz PCM utterances as microphone input while keeping the provider chain real:

```text
owner_voice_pet real-live-check
  -> generated wake/question PCM via macOS say + afconvert
  -> MemoryAudioTransport(flush preserves prefilled fixture frames)
  -> SherpaOnnxKeywordWakeWordProvider(real KWS)
  -> HybridVadProvider(real sherpa VAD + energy fallback)
  -> SherpaOnnxDenoiserPreprocessor(real GTCRN)
  -> PrimarySpeakerVadRecorder
  -> SherpaOnnxSttProvider(real local CTC partial/final)
  -> OpenAICompatibleLlmProvider(real cloud LLM from .env)
  -> MacSayTtsProvider(real macOS TTS)
  -> SoundDeviceAudioTransport.play_pcm(real speaker smoke)
```

The CLI defaults to two generated turns: “我叫阿明，请你记住我的名字。” followed by “我叫什么名字？”. A recording wrapper around the real LLM provider stores the messages sent to each LLM call, so the command can verify that the second request contains first-turn process-local user/assistant history without exposing the API key. `--question` can override the generated prompts, `--voice` controls the macOS `say` voice, and `--no-playback` keeps TTS synthesis and playback events while skipping speaker output for quiet automated runs.

Playback drain semantics are intentionally split by transport type:

1. `SoundDeviceAudioTransport.flush_input()` clears the realtime callback queue that accumulated during ACK/TTS playback.
2. `MemoryAudioTransport` defaults to the same destructive flush behavior for direct unit tests.
3. Simulated and fixture-driven live tests set `flush_clears_input=False` because their queued frames represent future time-ordered microphone input, not already accumulated realtime echo.

`_drain_input_after_playback()` therefore always calls `flush_input()` once after actual playback. If `OWNER_POST_PLAYBACK_DRAIN_MS=0`, it returns immediately and does not perform any timed read/drop loop. If the value is positive, the positive window is treated as an explicit user tuning and the runtime reads and discards only that configured duration before a final flush. When `OWNER_WAKE_ACK_TEXT` is empty and no ACK playback occurs, the runtime skips playback drain entirely.

## Real Provider Timing Observability

`owner_voice_pet real-live-check` is also the repeatable latency diagnostic entry for the full chain. The command records timing at three levels:

1. Command lifecycle: `started_at`, `finished_at`, epoch millisecond timestamps, and total `duration_ms`.
2. Preparation phases: `prepare_config`, `generate_fixture_audio`, `build_pipeline`, and `pipeline_run`, each with start offset and duration.
3. Pipeline events: every emitted stage event is recorded with turn id, ISO timestamp, command-relative offset, and message. Derived `stage_timings` pair stable stage boundaries such as `wake_listening -> wake_detected`, `capture_started -> speech_started`, `speech_started -> speech_ended`, `stt_started -> transcript_final`, `llm_started -> tts_started`, `tts_started -> playback_finished`, and `wake_listening -> standby_resumed`.

The real LLM provider is wrapped by `RecordingLlmProvider`. For every cloud LLM call it records:

1. `sent_at` and `sent_offset_ms`, captured immediately before delegating to the OpenAI-compatible provider.
2. `first_delta_at` and `first_delta_ms`, captured when the first non-empty text delta is yielded.
3. `finished_at` and `duration_ms`, captured when the stream ends or unwinds through an error.
4. `message_count` and `last_user_preview`, enough to correlate the request with a turn without serializing credentials or full request payloads.

Timing output is diagnostic metadata only. It SHALL NOT include `OWNER_LLM_API_KEY`, raw audio, denoised audio, full provider request headers, or any other secret-bearing configuration.

## Automatic Continuous Dialog Decision

持续对话不再等同于“每次回复后固定打开 3 秒追问窗口”。新的回复后状态流为：

```text
llm_started
  -> tts_started/playback_finished
  -> continuation_decision_started
  -> continuation_decision_made(continue|standby)
  -> standby_resumed
     OR followup_listening -> capture_started -> transcript_final -> llm_started ...
```

`ContinuationDecisionStage` 使用 hybrid provider：

1. `RuleContinuationDecisionProvider` 先用可解释规则判断。明确问题、补充信息请求、选择题、等待用户下一步的表达判定为 `continue`；完成性陈述、播报结果、拒绝/报错、寒暄结束和不确定表达判定为 `standby`。
2. `LlmContinuationDecisionProvider` 只在规则返回 uncertain 时调用云端 LLM 小分类器。分类器 prompt 要求只输出 `continue` 或 `standby`，可附带 `confidence`；解析失败或低于 `OWNER_CONTINUATION_CONFIDENCE_THRESHOLD` 时按 `standby`。
3. `HybridContinuationDecisionProvider` 是默认 provider，由 `OWNER_CONTINUATION_DECISION_PROVIDER=hybrid` 启用；`rule` 可用于完全离线决策。

Follow-up 捕获复用现有 capture、实时字幕、final STT、LLM、TTS 流程，但跳过 wake 和 ACK。为了让“3 秒内无回答”真正快退，follow-up 捕获临时把 recorder no-speech timeout 限制为 `OWNER_FOLLOWUP_LISTEN_TIMEOUT_MS`；若返回 `VAD_TIMEOUT_NO_SPEECH`，事件层输出 `followup_timeout` 和 `continuous_session_ended`，然后恢复 standby。

## Barge-in Playback Interruption

第一版打断只解决 turn-based 播放中的用户抢话，不做电话式 full-duplex。实现边界如下：

1. `MacSayTtsProvider` 产物优先转成 16 kHz/16-bit/mono PCM，使播放阶段可以切成约 100 ms 小段，避免整段 `afplay` 阻塞。
2. `SpeakStage` 播放每个小段后检查 microphone queue。`OWNER_BARGE_IN_ECHO_GUARD_MS` 内只 flush/忽略，降低播报回声误触发。
3. echo guard 之后，若 VAD 累计有效语音达到 `OWNER_BARGE_IN_MIN_SPEECH_MS` 且 realtime STT partial 有效，发出 `barge_in_detected` 并停止剩余句子/剩余回复播放。
4. 已完整播放的句子文本才写入 process-local assistant context；未播出的句子和后续未播放 token 不写入上下文。
5. 打断后的用户输入进入免唤醒 follow-up 捕获路径，保留实时字幕和 final STT，最终仍只有 final transcript 进入 LLM。

## Async Barge-in Speaker Gate Revision

旧打断路径由播放 chunk 回调驱动，每写完一段音频后才读取麦克风，不满足“播放同时监听”的实时性要求，也无法稳定排除助手自己的回放。修正版边界如下：

1. `AsyncBargeInMonitor` 在可打断 TTS 回复开始时启动后台线程，独立调用 `AudioTransport.read_frames()`，默认每 20 ms 检查一次，不由播放循环触发麦克风处理。
2. `AudioTransport.play_pcm_chunks()` 只接收 `should_stop` 停止信号；播放线程在 20-40 ms chunk 边界检查该信号并尽快停止剩余播报。
3. `BargeInSpeakerGate` 从最近一次用户正式输入建立本次进程内临时用户音色画像，从当前 TTS PCM 建立助手回放音色画像。
4. 播放期间候选麦克风帧必须通过 VAD，且不能匹配助手回放画像；若用户画像已存在，还必须匹配用户画像；若用户画像不足，降级为“非助手音色 + 有效 realtime partial”。
5. 播放期间 realtime STT 只作为后台打断确认信号，不发 `transcript_partial` 事件，避免把助手回放显示成用户实时字幕。
6. 监听线程确认打断后把已确认用户帧写入 `_pending_capture_frames`，后续 follow-up capture 直接续接，减少用户抢话首字丢失。
7. 用户画像、助手画像、候选帧均只保存在进程内，不写入磁盘、不发送给 LLM、不跨进程复用。

## Migration

No database migration. Users should run:

```bash
python3.11 scripts/download_speech_models.py --dir models
.venv/bin/python -m owner_voice_pet model-check --models-dir models
```

Existing `.env` remains valid because new wake keys have defaults.
For the local voice-chain revision, users should also ensure:

```dotenv
OWNER_SPEECH_PROVIDER=local
OWNER_NOISE_FILTER_ENABLED=1
OWNER_WAKE_DENOISE_ENABLED=0
OWNER_NOISE_FILTER_PROVIDER=sherpa_onnx_gtcrn
```