[本地语音降噪]:完成本地语音链路和噪音过滤,包含降噪模型、本地ASR和实时字幕稳定策略

This commit is contained in:
mkbk
2026-06-17 22:55:33 +08:00
parent a77a172412
commit 8c75fc5baf
22 changed files with 803 additions and 64 deletions
@@ -31,12 +31,14 @@ run-live
-> SoundDeviceAudioTransport
-> SherpaOnnxKeywordWakeWordProvider(local models/wake)
-> AcknowledgeStage(local "我在")
-> CaptureStage(primary speaker endpoint by default)
-> CloudAsrSttProvider or SherpaOnnxSttProvider
-> CaptureStage(raw mic frames)
-> AudioPreprocessStage(local GTCRN denoise, capture only)
-> PrimarySpeakerEndpoint/VAD(denoised frames)
-> SherpaOnnxSttProvider(local CTC partial/final by default)
-> DialogStage(session memory)
-> ConversationContext
-> OpenAICompatibleLlmProvider
-> CloudTtsProvider or MacSayTtsProvider
-> MacSayTtsProvider
-> speaker playback
```
@@ -51,11 +53,12 @@ run-live
7. On wake hit, reset wake stream and VAD recorder.
8. Record user utterance with VAD. The default provider is `hybrid`: project-local `sherpa-onnx` VAD remains the primary detector, and an energy threshold fallback prevents low microphone gain from being treated as no speech.
9. When `OWNER_ENDPOINT_MODE=primary_speaker`, build a temporary per-turn speaker profile from the first valid user speech frames and end the capture when the primary speaker is absent for `OWNER_SPEAKER_ABSENT_MS`.
10. Transcribe user utterance.
11. Emit transcript event to terminal.
12. Append user text and call LLM.
13. Synthesize/play reply.
14. Append assistant reply and return to standby.
10. When `OWNER_NOISE_FILTER_ENABLED=1`, pass formal user utterance frames through the local GTCRN denoiser before VAD, realtime STT, and final STT segment assembly.
11. Transcribe user utterance with the configured STT provider. The default is local CTC STT; cloud STT remains an explicit compatibility option.
12. Emit transcript event to terminal.
13. Append final user text and call LLM.
14. Synthesize/play reply with local TTS by default.
15. Append assistant reply and return to standby.
## Interfaces
@@ -88,6 +91,29 @@ class PipelineEventBus:
Required event types are `pipeline_started`, `wake_listening`, `wake_detected`, `ack_started`, `capture_started`, `speech_started`, `speech_ended`, `stt_started`, `transcript_final`, `llm_started`, `tts_started`, `playback_finished`, `standby_resumed`, and `stage_error`.
### `AudioPreprocessor`
```python
class AudioPreprocessor:
def load(self) -> None: ...
def reset(self) -> None: ...
def process_frame(self, frame: AudioFrame) -> AudioFrame: ...
def flush(self) -> list[AudioFrame]: ...
```
`NoopAudioPreprocessor` preserves tests and disabled configurations. `SherpaOnnxDenoiserPreprocessor` uses `sherpa_onnx.OnlineSpeechDenoiser` with an `OfflineSpeechDenoiserGtcrnModelConfig` pointing to `models/denoise/gtcrn_simple.onnx`. It converts int16 PCM to float32, calls `run(samples, sample_rate)`, converts returned `DenoisedAudio.samples` back to int16 PCM, and adds diagnostic metadata without storing audio.
The first version applies preprocessing only in capture. Wake listening remains raw by default because wake KWS models can be sensitive to spectral changes introduced by denoising. `OWNER_WAKE_DENOISE_ENABLED=1` reserves an explicit future switch for wake preprocessing.
### Local CTC STT
`SherpaOnnxSttProvider` reads `providers.stt.type` from `models/manifest.json`:
1. `sherpa-onnx-streaming-transducer`: existing `tokens`/`encoder`/`decoder`/`joiner` files and `OnlineRecognizer.from_transducer`.
2. `sherpa-onnx-streaming-zipformer2-ctc`: new default `tokens`/`model` files and `OnlineRecognizer.from_zipformer2_ctc`.
Realtime partial and final transcript use the same local recognizer when `OWNER_SPEECH_PROVIDER=local`. Partial output is filtered before emitting `transcript_partial`: texts with fewer than two meaningful characters, duplicate texts, and short regressions from the last displayed text are ignored. The final transcript remains the only user text appended to `ConversationContext`.
### `TurnController`
```python
@@ -136,7 +162,11 @@ models/
vad/
silero_vad.onnx
stt/
sherpa-onnx-streaming-zipformer-zh-14M-2023-02-23/
sherpa-onnx-streaming-zipformer-ctc-zh-int8-2025-06-30/
tokens.txt
model.int8.onnx
denoise/
gtcrn_simple.onnx
```
## Error Handling
@@ -152,6 +182,10 @@ models/
9. Playback drain misconfiguration: negative `OWNER_POST_PLAYBACK_DRAIN_MS` remains invalid; non-zero values are treated as explicit user tuning rather than default behavior.
10. Speaker profile threshold misconfiguration: non-positive `OWNER_SPEAKER_PROFILE_MIN_MS` fails config validation.
11. Realtime STT failure: startup model缺失按 `model-check` 暴露;capture 中 partial session 失败不得把 partial 文本写入上下文。
12. Denoiser missing model: `model-check` reports `denoise/gtcrn_simple.onnx` and live startup fails before microphone listening.
13. Denoiser runtime failure: the current turn emits `stage_error` and recovers to standby without invoking final STT or LLM.
14. CTC manifest mismatch: missing `model` or `tokens` files report `STT_MODEL_MISSING`; transducer manifests remain supported for existing local setups.
15. Partial text noise: single-character or transient partial output is ignored rather than displayed as terminal feedback.
## Testing Strategy
@@ -168,6 +202,11 @@ models/
11. Transport batching test validates that queued SoundDevice frames are returned together.
12. Partial transcript event test validates that realtime text appears after speech start and before final transcript.
13. Context isolation test validates partial transcript does not enter LLM messages.
14. Denoiser manifest/model-check test validates the required GTCRN file is present.
15. Fake denoiser test validates VAD, partial STT, and final STT receive denoised frames from the same processed stream.
16. Local speech provider test validates `OWNER_SPEECH_PROVIDER=local` does not construct cloud ASR/TTS providers.
17. Partial filter test validates single-character and short transient results are not emitted.
18. CTC STT loading test validates manifest type selects `from_zipformer2_ctc` and does not require transducer encoder/decoder/joiner files.
## Migration
@@ -179,3 +218,11 @@ python3.11 scripts/download_speech_models.py --dir models
```
Existing `.env` remains valid because new wake keys have defaults.
For the local voice-chain revision, users should also ensure:
```dotenv
OWNER_SPEECH_PROVIDER=local
OWNER_NOISE_FILTER_ENABLED=1
OWNER_WAKE_DENOISE_ENABLED=0
OWNER_NOISE_FILTER_PROVIDER=sherpa_onnx_gtcrn
```