[Wake/VAD/STT 与 Live runtime]:完成真实重复语音运行,包含云端ASR/TTS开关、run-live和临时上下文测试

This commit is contained in:
mkbk
2026-06-17 20:00:55 +08:00
parent ac97daa1e7
commit 4b7cd18a0f
20 changed files with 1043 additions and 68 deletions
@@ -1,7 +1,7 @@
## ADDED Requirements
### Requirement: Live repeat voice runtime
The system SHALL provide a `run-live` command that performs real repeated voice conversation with local microphone input, local speech processing, cloud LLM reply generation, local TTS, local speaker playback, and automatic return to standby.
The system SHALL provide a `run-live` command that performs real repeated voice conversation with local microphone input, configured speech recognition and speech synthesis providers, cloud LLM reply generation, local speaker playback, and automatic return to standby.
#### Scenario: Live runtime starts in standby
- **WHEN** the user runs `PYTHONPATH=src python3.11 -m owner_voice_pet run-live`
@@ -46,6 +46,21 @@ The system SHALL provide project-local speech model preparation and diagnostics
- **WHEN** `sherpa-onnx` is unavailable, a model file is missing, or a model cannot be loaded
- **THEN** `model-check` SHALL fail with a structured model error and SHALL NOT start live microphone listening
### Requirement: Configurable speech provider mode
The system SHALL read `OWNER_SPEECH_PROVIDER` from `.env` to choose between cloud speech providers and local speech providers for first-version live runtime.
#### Scenario: Cloud speech provider is selected
- **WHEN** `OWNER_SPEECH_PROVIDER=cloud`
- **THEN** live ASR SHALL use the configured cloud model from `OWNER_ASR_MODEL`, live TTS SHALL use the configured cloud model from `OWNER_TTS_MODEL`, and the implementation SHALL default those models to `mimo-v2.5-asr` and `mimo-v2.5-tts`
#### Scenario: Local speech provider is selected
- **WHEN** `OWNER_SPEECH_PROVIDER=local`
- **THEN** live ASR/VAD SHALL use project-local speech model assets and local TTS SHALL use a local playback-capable provider
#### Scenario: Speech provider is invalid
- **WHEN** `OWNER_SPEECH_PROVIDER` is neither `cloud` nor `local`
- **THEN** startup validation SHALL fail with a structured configuration error
### Requirement: Live audio device readiness
The system SHALL provide live audio device diagnostics and SHALL use `sounddevice` for first-version real microphone and speaker access.
@@ -123,11 +138,11 @@ The system SHALL use the local microphone as the first-version input Transport a
- **WHEN** the microphone or speaker is missing, denied, unsupported, or inaccessible through `sounddevice`
- **THEN** the system SHALL expose a recoverable Transport error with a stable error code and SHALL NOT silently fall back to fixture audio in live mode
### Requirement: Local STT transcription
The system SHALL transcribe captured user utterances through a local STT provider, with `sherpa-onnx` as the first live implementation candidate and project-local models under `models/`.
### Requirement: Configured STT transcription
The system SHALL transcribe captured user utterances through the configured STT provider; cloud mode SHALL use NewAPI-compatible ASR and local mode SHALL use project-local `sherpa-onnx` model assets under `models/`.
#### Scenario: STT succeeds
- **WHEN** local STT returns non-empty text for a captured live audio segment
- **WHEN** configured STT returns non-empty text for a captured live audio segment
- **THEN** the pipeline SHALL add the trimmed text as a user message to the current process conversation context
#### Scenario: STT returns empty text
@@ -135,11 +150,11 @@ The system SHALL transcribe captured user utterances through a local STT provide
- **THEN** the live runtime SHALL skip LLM invocation and return to standby with a recoverable status
#### Scenario: STT provider fails
- **WHEN** the STT provider raises an error or cannot load its local model
- **WHEN** the STT provider raises an error, cloud ASR fails, or local model loading fails
- **THEN** the system SHALL emit an STT or model error code and SHALL recover to a state where future wake attempts are possible if startup can continue safely
### Requirement: Local TTS synthesis and playback
The system SHALL synthesize assistant replies through a local TTS provider and SHALL play synthesized speech through the local system output path, with macOS `say/afplay` accepted as the first live implementation.
### Requirement: Configured TTS synthesis and playback
The system SHALL synthesize assistant replies through the configured TTS provider and SHALL play synthesized speech through the local system output path; cloud mode SHALL use NewAPI-compatible TTS and local mode SHALL use a local playback-capable TTS provider.
#### Scenario: Reply text is ready for speech
- **WHEN** the LLM produces a non-empty reply for a live turn