whisper.rn

React Native binding of whisper.cpp.

whisper.cpp: High-performance inference of OpenAI's Whisper automatic speech recognition (ASR) model

Screenshots

iOS: Tested on iPhone 13 Pro Max	Android: Tested on Pixel 6
(tiny.en, Core ML enabled, release mode + archive)	(tiny.en, armv8.2-a+fp16, release mode)

Installation

npm install whisper.rn

For iOS, please re-run npx pod-install again.

For Android, it's recommended to use ndkVersion = "24.0.8215888" (or above) in your root project build configuration for Apple Silicon Macs. Otherwise please follow this trobleshooting issue.

For Expo, you will need to prebuild the project before using it. See Expo guide for more details.

Add Microphone Permissions (Optional)

If you want to use realtime transcribe, you need to add the microphone permission to your app.

iOS

Add these lines to ios/[YOU_APP_NAME]/info.plist

<key>NSMicrophoneUsageDescription</key>
<string>This app requires microphone access in order to transcribe speech</string>

For tvOS, please note that the microphone is not supported.

Android

Add the following line to android/app/src/main/AndroidManifest.xml

<uses-permission android:name="android.permission.RECORD_AUDIO" />

Usage

import { initWhisper } from 'whisper.rn'

const whisperContext = await initWhisper({
  filePath: 'file://.../ggml-tiny.en.bin',
})

const sampleFilePath = 'file://.../sample.wav'
const options = { language: 'en' }
const { stop, promise } = whisperContext.transcribe(sampleFilePath, options)

const { result } = await promise
// result: (The inference text result from audio file)

Use realtime transcribe:

const { stop, subscribe } = await whisperContext.transcribeRealtime(options)

subscribe(evt => {
  const { isCapturing, data, processTime, recordingTime } = evt
  console.log(
    `Realtime transcribing: ${isCapturing ? 'ON' : 'OFF'}\n` +
      // The inference text result from audio record:
      `Result: ${data.result}\n\n` + 
      `Process time: ${processTime}ms\n` +
      `Recording time: ${recordingTime}ms`,
  )
  if (!isCapturing) console.log('Finished realtime transcribing')
})

In Android, you may need to request the microphone permission by PermissionAndroid.

Please visit the Documentation for more details.

Usage with assets

You can also use the model file / audio file from assets:

import { initWhisper } from 'whisper.rn'

const whisperContext = await initWhisper({
  filePath: require('../assets/ggml-tiny.en.bin'),
})

const { stop, promise } =
  whisperContext.transcribe(require('../assets/sample.wav'), options)

// ...

This requires editing the metro.config.js to support assets:

// ...
const defaultAssetExts = require('metro-config/src/defaults/defaults').assetExts

module.exports = {
  // ...
  resolver: {
    // ...
    assetExts: [
      ...defaultAssetExts,
      'bin', // whisper.rn: ggml model binary
      'mil', // whisper.rn: CoreML model asset
    ]
  },
}

Please note that it will significantly increase the size of the app in release mode.

Core ML support

Platform: iOS 15.0+, tvOS 15.0+

To use Core ML on iOS, you will need to have the Core ML model files.

The .mlmodelc model files is load depend on the ggml model file path. For example, if your ggml model path is ggml-tiny.en.bin, the Core ML model path will be ggml-tiny.en-encoder.mlmodelc. Please note that the ggml model is still needed as decoder or encoder fallback.

The Core ML models are hosted here: https://huggingface.co/ggerganov/whisper.cpp/tree/main

If you want to download model at runtime, during the host file is archive, you will need to unzip the file to get the .mlmodelc directory, you can use library like react-native-zip-archive, or host those individual files to download yourself.

The .mlmodelc is a directory, usually it includes 5 files (3 required):

[
  'model.mil',
  'coremldata.bin',
  'weights/weight.bin',
  // Not required:
  // 'metadata.json', 'analytics/coremldata.bin',
]

Or just use require to bundle that in your app, like the example app does, but this would increase the app size significantly.

const whisperContext = await initWhisper({
  filePath: require('../assets/ggml-tiny.en.bin')
  coreMLModelAsset:
    Platform.OS === 'ios'
      ? {
          filename: 'ggml-tiny.en-encoder.mlmodelc',
          assets: [
            require('../assets/ggml-tiny.en-encoder.mlmodelc/weights/weight.bin'),
            require('../assets/ggml-tiny.en-encoder.mlmodelc/model.mil'),
            require('../assets/ggml-tiny.en-encoder.mlmodelc/coremldata.bin'),
          ],
        }
      : undefined,
})

In real world, we recommended to split the asset imports into another platform specific file (e.g. context-opts.ios.js) to avoid these unused files in the bundle for Android.

Run with example

The example app provide a simple UI for testing the functions.

Used Whisper model: tiny.en in https://huggingface.co/ggerganov/whisper.cpp
Sample file: jfk.wav in https://github.com/ggerganov/whisper.cpp/tree/master/samples

Please follow the Development Workflow section of contributing guide to run the example app.

Mock whisper.rn

We have provided a mock version of whisper.rn for testing purpose you can use on Jest:

jest.mock('whisper.rn', () => require('whisper.rn/jest/mock'))

Contributing

See the contributing guide to learn how to contribute to the repository and the development workflow.

Troubleshooting

See the troubleshooting if you encounter any problem while using whisper.rn.

License

MIT

---

Made with create-react-native-library

---

Built and maintained by BRICKS.