• Nicolas Patry's avatar
    Adding support for `microphone` streaming within pipeline. (#15046) · 623d8cb4
    Nicolas Patry authored
    
    
    * Adding support for `microphone` streaming within pipeline.
    
    - Uses `ffmpeg` to get microphone data.
    - Makes sure alignment is made to `size_of_sample`.
    - Works by sending `{"raw": ..data.., "stride": (n, left, right),
    "partial": bool}`
    directly to the pipeline enabling to stream partial results and still
    get inference.
    - Let's `partial` information flow through the pipeline to enable caller
      to get it back and choose to display text or not.
    
    - The striding reconstitution is bound to have errors since CTC does not
    keep previous state. Currently most of the errors are we don't know if
    there's a space or not between two chunks.
    Since we have some left striding info, we could use that during decoding
    to choose what to do with those spaces and even extra letters maybe (if
    the stride is long enough, it's bound to cover at least a few symbols)
    
    Fixing tests.
    
    Protecting with `require_torch`.
    
    `raw_ctc` support for nicer demo.
    
    Post rebase fixes.
    
    Revamp to split raw_mic_data from it's live chunking.
    
    - Requires a refactor to make everything a bit cleaner.
    
    Automatic resampling.
    
    Small fix.
    
    Small fix.
    
    * Post rebase fix (need to let super handle more logic, reorder args.)
    
    * Update docstrings
    
    * Docstring format.
    
    * Remove print.
    
    * Prevent flow of `input_values`.
    
    * Fixing `stride` too.
    
    * Fixing the PR by removing `raw_ctc`.
    
    * Better docstrings.
    
    * Fixing init.
    
    * Update src/transformers/pipelines/audio_utils.py
    Co-authored-by: default avatarAnton Lozhkov <aglozhkov@gmail.com>
    
    * Update tests/test_pipelines_automatic_speech_recognition.py
    Co-authored-by: default avatarAnton Lozhkov <aglozhkov@gmail.com>
    
    * Quality.
    Co-authored-by: default avatarAnton Lozhkov <aglozhkov@gmail.com>
    623d8cb4
test_pipelines_automatic_speech_recognition.py 24.3 KB