feat(vertexai): Add flutter_soloud for sound output in Live API audio streaming example. #17305

khanhnwin · 2025-04-23T23:48:26Z

Description

Replace this paragraph with a description of what this PR is doing. If you're modifying existing behavior, describe the existing behavior, how this PR is changing it, and what motivated the change.

Related Issues

Replace this paragraph with a list of issues related to this PR from the issue database. Indicate, which of these issues are resolved or fixed by this PR. Note that you'll have to prefix the issue numbers with flutter/flutter#.

Checklist

Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]).
This will ensure a smooth and quick review process. Updating the pubspec.yaml and changelogs is not required.

I read the Contributor Guide and followed the process outlined there for submitting PRs.
My PR includes unit or integration tests for all changed/updated/fixed behaviors (See Contributor Guide).
All existing and new tests are passing.
I updated/added relevant documentation (doc comments with ///).
The analyzer (melos run analyze) does not report any problems on my PR.
I read and followed the Flutter Style Guide.
I signed the CLA.
I am willing to follow-up on review comments in a timely manner.

Breaking Change

Does your PR require plugin users to manually update their apps to accommodate your change?

Yes, this is a breaking change.
No, this is not a breaking change.

…lass!

MichaelVerdon · 2025-04-24T09:36:29Z

Hi there, thanks for the PR! I understand that this is a draft but can you please give me some more detail about this? I recently added a sound example to vertexai https://github.com/firebase/flutterfire/blob/main/packages/firebase_vertexai/firebase_vertexai/example/lib/pages/audio_page.dart unless yours is different?

khanhnwin · 2025-04-24T17:47:00Z

Hey @MichaelVerdon! From my understanding, the audio page that you linked to is for recording a it of audio and sending it to the Gemini API as opposed to streaming live audio, right?

I'm working with @cynthiajoan on refining the Live API example page, so in this case it's bidirectional audio streaming here => https://github.com/firebase/flutterfire/blob/main/packages/firebase_vertexai/firebase_vertexai/example/lib/pages/bidi_page.dart. Docs: https://firebase.google.com/docs/vertex-ai/live-api?platform=flutter

Edit: I realize my PR title was unclear, so I added more detail!

russellwheatley · 2025-05-13T10:13:06Z

@khanhnwin - is this still being worked on?

khanhnwin · 2025-05-19T21:43:34Z

@russellwheatley Yes! @cynthiajoan is going to clean it up and integrate it into the example app is my understanding.

cynthiajoan · 2025-05-19T23:04:15Z

@russellwheatley Yes! @cynthiajoan is going to clean it up and integrate it into the example app is my understanding.

Yeah I'm working on this and not planning to have this PR included in the I/O release.

kevsjh · 2025-05-24T16:03:54Z

The current implementation of Live AI API stream gets the AI to enter an endless loop of self replying, the audio of the speaker feeds back into the user audio stream due to the lack of echo cancelling.

Tested against both flutter_soloud and just_audio where both packages does not have AEC implementation so far.

What is the recommended flutter audio package for this use case as currently this approach is not usable for production

cynthiajoan · 2025-05-30T23:07:06Z

The current implementation of Live AI API stream gets the AI to enter an endless loop of self replying, the audio of the speaker feeds back into the user audio stream due to the lack of echo cancelling.

Tested against both flutter_soloud and just_audio where both packages does not have AEC implementation so far.

What is the recommended flutter audio package for this use case as currently this approach is not usable for production

In this PR we updated the audio_input.dart to add some attempts for AEC in the recorder config. Would you like to try to see if that improves the AI self hearing in your target platform?

natebosch · 2025-05-30T23:40:56Z

packages/firebase_vertexai/firebase_vertexai/example/lib/pages/bidi_page.dart

+  final AudioOutput audioOutput = AudioOutput();
+  final AudioInput audioInput = AudioInput();


Can these be kept private? I don't see any references outside this library.

Also consider omitting the repeated type name.

Suggested change

final AudioOutput audioOutput = AudioOutput();

final AudioInput audioInput = AudioInput();

final _audioOutput = AudioOutput();

final _audioInput = AudioInput();

natebosch · 2025-05-30T23:41:28Z

packages/firebase_ai/firebase_ai/example/lib/pages/bidi_page.dart

+    initAudioOutput();
+  }
+
+  Future<void> initAudioOutput() async {


Can this be private? Consider renaming _initAudio since it handles both input and output

natebosch · 2025-05-30T23:43:09Z

packages/firebase_ai/firebase_ai/example/lib/pages/bidi_page.dart

-      });
-      await _session.sendMediaStream(mediaChunkStream);
+      if (inputStream != null) {
+        Stream<InlineDataPart> inlineDataStream = inputStream.map((data) {


[nit] Consider omitting the variable type on the left.

Suggested change

Stream<InlineDataPart> inlineDataStream = inputStream.map((data) {

final inlineDataStream = inputStream.map((data) {

natebosch · 2025-05-30T23:43:54Z

packages/firebase_ai/firebase_ai/example/lib/pages/bidi_page.dart

    } catch (e) {
+      print(e);


Is this an intended change? Consider removing all print. Maybe use log instead?

natebosch · 2025-05-30T23:45:12Z

packages/firebase_ai/firebase_ai/example/lib/pages/function_calling_page.dart

-        throw UnimplementedError(
-          'Function not declared to the model: ${functionCall.name}',
-        );
+      for (final functionCall in functionCalls) {


This looks like a fine change to make - how does it relate to the rest of the PR? Consider comiting separately.

natebosch · 2025-05-30T23:46:05Z

packages/firebase_ai/firebase_ai/example/lib/utils/audio_input.dart

+    super.dispose();
+  }
+
+  Future<void> checkPermission() async {


Can this be private?

natebosch · 2025-05-30T23:46:45Z

packages/firebase_ai/firebase_ai/example/lib/utils/audio_output.dart

+  SoundHandle? handle;
+
+  Future<void> init() async {
+    /// Initialize the player.


[nit] use // for comments that are not member docs.

Suggested change

/// Initialize the player.

// Initialize the player.

natebosch · 2025-05-30T23:47:01Z

packages/firebase_ai/firebase_ai/example/lib/utils/audio_output.dart

+import 'package:flutter_soloud/flutter_soloud.dart';
+
+class AudioOutput {
+  AudioSource? stream; // Start playback


I don't understand this comment. Should it be expanded?

khanhnwin added 2 commits April 23, 2025 16:46

add soloud for sound output, but record is failing on iOS

3434e46

restarting my simulator fixed it... but incorporating my AudioInput c…

958d97b

…lass!

khanhnwin changed the title ~~Add flutter_soloud for sound output example.~~ Add flutter_soloud for sound output in Live API audio streaming example. Apr 24, 2025

add JS scripts for flutter_soloud on web

6188430

cynthiajoan changed the title ~~Add flutter_soloud for sound output in Live API audio streaming example.~~ feat(vertexai): Add flutter_soloud for sound output in Live API audio streaming example. Apr 24, 2025

cynthiajoan added 2 commits May 20, 2025 22:08

add gitignore to example

3ba57a2

Merge branch 'main' into khanh-flutter-soloud

e471203

cynthiajoan added 2 commits May 30, 2025 16:01

Clean up before review

340db1f

Merge branch 'main' into khanh-flutter-soloud

d9a2ebe

cynthiajoan marked this pull request as ready for review May 30, 2025 23:03

cynthiajoan requested review from natebosch and russellwheatley May 30, 2025 23:03

natebosch approved these changes May 30, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(vertexai): Add flutter_soloud for sound output in Live API audio streaming example. #17305

feat(vertexai): Add flutter_soloud for sound output in Live API audio streaming example. #17305

Uh oh!

khanhnwin commented Apr 23, 2025

Uh oh!

MichaelVerdon commented Apr 24, 2025 •

edited

Loading

Uh oh!

khanhnwin commented Apr 24, 2025 •

edited

Loading

Uh oh!

russellwheatley commented May 13, 2025

Uh oh!

khanhnwin commented May 19, 2025

Uh oh!

cynthiajoan commented May 19, 2025

Uh oh!

kevsjh commented May 24, 2025

Uh oh!

cynthiajoan commented May 30, 2025

Uh oh!

natebosch May 30, 2025

Uh oh!

natebosch May 30, 2025

Uh oh!

natebosch May 30, 2025

Uh oh!

natebosch May 30, 2025

Uh oh!

natebosch May 30, 2025

Uh oh!

natebosch May 30, 2025

Uh oh!

natebosch May 30, 2025

Uh oh!

natebosch May 30, 2025

Uh oh!

Uh oh!

		final AudioOutput audioOutput = AudioOutput();
		final AudioInput audioInput = AudioInput();

	Stream<InlineDataPart> inlineDataStream = inputStream.map((data) {
	final inlineDataStream = inputStream.map((data) {

feat(vertexai): Add flutter_soloud for sound output in Live API audio streaming example. #17305

Are you sure you want to change the base?

feat(vertexai): Add flutter_soloud for sound output in Live API audio streaming example. #17305

Uh oh!

Conversation

khanhnwin commented Apr 23, 2025

Description

Related Issues

Checklist

Breaking Change

Uh oh!

MichaelVerdon commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

khanhnwin commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

russellwheatley commented May 13, 2025

Uh oh!

khanhnwin commented May 19, 2025

Uh oh!

cynthiajoan commented May 19, 2025

Uh oh!

kevsjh commented May 24, 2025

Uh oh!

cynthiajoan commented May 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MichaelVerdon commented Apr 24, 2025 •

edited

Loading

khanhnwin commented Apr 24, 2025 •

edited

Loading