[本地语音降噪]:完成本地语音链路和噪音过滤,包含降噪模型、本地ASR和实时字幕稳定策略
This commit is contained in:
@@ -31,12 +31,14 @@ run-live
|
||||
-> SoundDeviceAudioTransport
|
||||
-> SherpaOnnxKeywordWakeWordProvider(local models/wake)
|
||||
-> AcknowledgeStage(local "我在")
|
||||
-> CaptureStage(primary speaker endpoint by default)
|
||||
-> CloudAsrSttProvider or SherpaOnnxSttProvider
|
||||
-> CaptureStage(raw mic frames)
|
||||
-> AudioPreprocessStage(local GTCRN denoise, capture only)
|
||||
-> PrimarySpeakerEndpoint/VAD(denoised frames)
|
||||
-> SherpaOnnxSttProvider(local CTC partial/final by default)
|
||||
-> DialogStage(session memory)
|
||||
-> ConversationContext
|
||||
-> OpenAICompatibleLlmProvider
|
||||
-> CloudTtsProvider or MacSayTtsProvider
|
||||
-> MacSayTtsProvider
|
||||
-> speaker playback
|
||||
```
|
||||
|
||||
@@ -51,11 +53,12 @@ run-live
|
||||
7. On wake hit, reset wake stream and VAD recorder.
|
||||
8. Record user utterance with VAD. The default provider is `hybrid`: project-local `sherpa-onnx` VAD remains the primary detector, and an energy threshold fallback prevents low microphone gain from being treated as no speech.
|
||||
9. When `OWNER_ENDPOINT_MODE=primary_speaker`, build a temporary per-turn speaker profile from the first valid user speech frames and end the capture when the primary speaker is absent for `OWNER_SPEAKER_ABSENT_MS`.
|
||||
10. Transcribe user utterance.
|
||||
11. Emit transcript event to terminal.
|
||||
12. Append user text and call LLM.
|
||||
13. Synthesize/play reply.
|
||||
14. Append assistant reply and return to standby.
|
||||
10. When `OWNER_NOISE_FILTER_ENABLED=1`, pass formal user utterance frames through the local GTCRN denoiser before VAD, realtime STT, and final STT segment assembly.
|
||||
11. Transcribe user utterance with the configured STT provider. The default is local CTC STT; cloud STT remains an explicit compatibility option.
|
||||
12. Emit transcript event to terminal.
|
||||
13. Append final user text and call LLM.
|
||||
14. Synthesize/play reply with local TTS by default.
|
||||
15. Append assistant reply and return to standby.
|
||||
|
||||
## Interfaces
|
||||
|
||||
@@ -88,6 +91,29 @@ class PipelineEventBus:
|
||||
|
||||
Required event types are `pipeline_started`, `wake_listening`, `wake_detected`, `ack_started`, `capture_started`, `speech_started`, `speech_ended`, `stt_started`, `transcript_final`, `llm_started`, `tts_started`, `playback_finished`, `standby_resumed`, and `stage_error`.
|
||||
|
||||
### `AudioPreprocessor`
|
||||
|
||||
```python
|
||||
class AudioPreprocessor:
|
||||
def load(self) -> None: ...
|
||||
def reset(self) -> None: ...
|
||||
def process_frame(self, frame: AudioFrame) -> AudioFrame: ...
|
||||
def flush(self) -> list[AudioFrame]: ...
|
||||
```
|
||||
|
||||
`NoopAudioPreprocessor` preserves tests and disabled configurations. `SherpaOnnxDenoiserPreprocessor` uses `sherpa_onnx.OnlineSpeechDenoiser` with an `OfflineSpeechDenoiserGtcrnModelConfig` pointing to `models/denoise/gtcrn_simple.onnx`. It converts int16 PCM to float32, calls `run(samples, sample_rate)`, converts returned `DenoisedAudio.samples` back to int16 PCM, and adds diagnostic metadata without storing audio.
|
||||
|
||||
The first version applies preprocessing only in capture. Wake listening remains raw by default because wake KWS models can be sensitive to spectral changes introduced by denoising. `OWNER_WAKE_DENOISE_ENABLED=1` reserves an explicit future switch for wake preprocessing.
|
||||
|
||||
### Local CTC STT
|
||||
|
||||
`SherpaOnnxSttProvider` reads `providers.stt.type` from `models/manifest.json`:
|
||||
|
||||
1. `sherpa-onnx-streaming-transducer`: existing `tokens`/`encoder`/`decoder`/`joiner` files and `OnlineRecognizer.from_transducer`.
|
||||
2. `sherpa-onnx-streaming-zipformer2-ctc`: new default `tokens`/`model` files and `OnlineRecognizer.from_zipformer2_ctc`.
|
||||
|
||||
Realtime partial and final transcript use the same local recognizer when `OWNER_SPEECH_PROVIDER=local`. Partial output is filtered before emitting `transcript_partial`: texts with fewer than two meaningful characters, duplicate texts, and short regressions from the last displayed text are ignored. The final transcript remains the only user text appended to `ConversationContext`.
|
||||
|
||||
### `TurnController`
|
||||
|
||||
```python
|
||||
@@ -136,7 +162,11 @@ models/
|
||||
vad/
|
||||
silero_vad.onnx
|
||||
stt/
|
||||
sherpa-onnx-streaming-zipformer-zh-14M-2023-02-23/
|
||||
sherpa-onnx-streaming-zipformer-ctc-zh-int8-2025-06-30/
|
||||
tokens.txt
|
||||
model.int8.onnx
|
||||
denoise/
|
||||
gtcrn_simple.onnx
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
@@ -152,6 +182,10 @@ models/
|
||||
9. Playback drain misconfiguration: negative `OWNER_POST_PLAYBACK_DRAIN_MS` remains invalid; non-zero values are treated as explicit user tuning rather than default behavior.
|
||||
10. Speaker profile threshold misconfiguration: non-positive `OWNER_SPEAKER_PROFILE_MIN_MS` fails config validation.
|
||||
11. Realtime STT failure: startup model缺失按 `model-check` 暴露;capture 中 partial session 失败不得把 partial 文本写入上下文。
|
||||
12. Denoiser missing model: `model-check` reports `denoise/gtcrn_simple.onnx` and live startup fails before microphone listening.
|
||||
13. Denoiser runtime failure: the current turn emits `stage_error` and recovers to standby without invoking final STT or LLM.
|
||||
14. CTC manifest mismatch: missing `model` or `tokens` files report `STT_MODEL_MISSING`; transducer manifests remain supported for existing local setups.
|
||||
15. Partial text noise: single-character or transient partial output is ignored rather than displayed as terminal feedback.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
@@ -168,6 +202,11 @@ models/
|
||||
11. Transport batching test validates that queued SoundDevice frames are returned together.
|
||||
12. Partial transcript event test validates that realtime text appears after speech start and before final transcript.
|
||||
13. Context isolation test validates partial transcript does not enter LLM messages.
|
||||
14. Denoiser manifest/model-check test validates the required GTCRN file is present.
|
||||
15. Fake denoiser test validates VAD, partial STT, and final STT receive denoised frames from the same processed stream.
|
||||
16. Local speech provider test validates `OWNER_SPEECH_PROVIDER=local` does not construct cloud ASR/TTS providers.
|
||||
17. Partial filter test validates single-character and short transient results are not emitted.
|
||||
18. CTC STT loading test validates manifest type selects `from_zipformer2_ctc` and does not require transducer encoder/decoder/joiner files.
|
||||
|
||||
## Migration
|
||||
|
||||
@@ -179,3 +218,11 @@ python3.11 scripts/download_speech_models.py --dir models
|
||||
```
|
||||
|
||||
Existing `.env` remains valid because new wake keys have defaults.
|
||||
For the local voice-chain revision, users should also ensure:
|
||||
|
||||
```dotenv
|
||||
OWNER_SPEECH_PROVIDER=local
|
||||
OWNER_NOISE_FILTER_ENABLED=1
|
||||
OWNER_WAKE_DENOISE_ENABLED=0
|
||||
OWNER_NOISE_FILTER_PROVIDER=sherpa_onnx_gtcrn
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user