Echo cancel not working

[darkbasic]

I recently upgraded to Fedora 34 and since it switched to pipewire I can't set up pulseaudio to do echo cancellation anymore.
I decided to try PulseEffects' WebRTC filter to achieve the same goal, but it doesn't work: there is no way to select which source should I use to echo cancel and if I reproduce some audio from my computer speakers I can clearly see in the PulseEffect Source filtered microphone output. Am I doing something wrong?

[wwmm]

there is no way to select which source should I use to echo cancel

Unfortunately that is right. The webrtc plugin by default uses the sound card monitors as source for echo cancellation.

I have to think carefully about what to do with this plugin in the future. The plugin currently used is a built-in GStreamer plugin. But in the gtk4 port I am using native PipeWire filters instead of GStreamer. I will have to write myself a new plugin for echo cancellation. I am not sure how yet...

[darkbasic]

The webrtc plugin by default uses the sound card monitors as source for echo cancellation

That would be good enough for my needs, but unfortunately it doesn't work at all. Any idea why?

[wwmm]

Kill the current instance pulseeffects -q and restart it in debug mode G_MESSAGES_DEBUG=pulseeffects pulseeffects. THis way we will see what is being done under the hoods.

At least for me this webrtc plugin has always been a little inconsistent when it comes to echo cancellation. Sometimes it is really good but sometimes it is almost like it is not there. It was one of the hardest plugins to integrate exactly because of the need of a secondary GStreamer pipeline for the probe recording the monitors.

[wwmm]

This should not be a problem in EasyEffects 6.0.0

[darkbasic]

Unfortunately it still doesn't work with EasyEffects 7.0.0 and pipewire 0.3.61: https://youtu.be/h1910Z7ALGE

[darkbasic]

@wwmm do you want to re-open this issue or would you rather prefer me to create a new one?

[wwmm]

do you want to re-open this issue or would you rather prefer me to create a new one?

We can reopen this one but it is not clear to me what you expect to achieve in the video if OBS is the only stream in the mic tab and effects are disabled for it. The only reason why there is signal passing through the echo canceller plugin is pavucontrol. It's level meter streams trigger a not intended behavior I think it should not be happening anymore.

[darkbasic]

it is not clear to me what you expect to achieve in the video if OBS is the only stream in the mic tab and effects are disabled for it

I'm sorry, I saw the signal going through and I didn't notice it was disabled.
It's not just because of pavucontrol, it's the same even without it as you can see: https://youtu.be/AO7ZLnl55js
Sometimes switching to another tab and then going back to input effects fixes it, sometimes not.

Anyway what I want to achieve is the following: OBS records from the microphone while supertuxkart is playing some background music, which should get echo cancelled and thus nothing gets recorded (except maybe some noise, but the noise reduction plugin works very well).
In order to achieve so I have to enable OBS in the inputs (which I did in the second video) and maybe (I have no idea how the echo cancel plugin selects its source) enable the supertuxkart output as well. As you can see from the second video the output of the echo canceller is the same as the input, even with OBS enabled.

P.S.
I've forgot to actually unmute the mic in OBS, but it's the exact same thing if you do so.

[wwmm]

It's not just because of pavucontrol, it's the same even without it as you can see: https://youtu.be/AO7ZLnl55js
Sometimes switching to another tab and then going back to input effects fixes it, sometimes not.

Run pw-dot. Its output file can be viewed with xdot and will show the links between filters and devices as well as the link state. I want to see what is linked to EasyEffects virtual source.

OBS records from the microphone while supertuxkart is playing some background music, which should get echo cancelled and thus nothing gets recorded

I am not sure the speex echo canceller will go as far as completely removing the background music recorded by the mic. The most I have seen it doing is removing some minor echo in some situations. What you desire may be beyond its capabilities.

I have no idea how the echo cancel plugin selects its source

Besides the connections to the filters that are before and after it in the effects pipeline the echo canceller needs to link a probe to the soundcard device. At this moment the probes are automatically linked to the output device currently being used by EasyEffects output effects pipeline. IT should be the same you selected in EasyEffects PipeWire's Device Management tab.

[darkbasic]

I want to see what is linked to EasyEffects virtual source.

pw.zip

Shouldn't helvum achieve the same goal?

https://youtu.be/3LjqKTWa80k

At this moment the probes are automatically linked to the output device currently being used by EasyEffects output effects pipeline

As you can see from the previous video according to helvum these probes are not attached at all, but manually attaching them doesn't do anything as well.

I am not sure the speex echo canceller will go as far as completely removing the background music recorded by the mic

Pulseaudio echo cancellation (backed by webrtc-audio-processing) is so good that you won't be able to tell if there is anything playing back in the background. On the contrary EasyEffects echo cancellation is so bad that it actually worsen things. I'm wondering if this is because the probes aren't attached correctly or because it doesn't compensate the clock drift between the two different input and output devices (the input is an USB webcam and the output is an USB soundbar).

[wwmm]

Shouldn't helvum achieve the same goal?

I think pw-dot output is cleaner. And it shows me the sate of each link.

As you can see from the previous video according to helvum these probes are not attached at all, but manually attaching them doesn't do anything as well.

It seems a regression was introduced at some point. I am doing some tests and I can see this happening on my computer now.

I'm wondering if this is because the probes aren't attached correctly or because it doesn't compensate the clock drift between the two different input and output devices

Not having the probes connected definitely do not help. But Pulseaudio is not using the speex library to do echo cancellation. If I remember well it uses a custom version of the webrtc library that seems to only exist in Pulseaudio sources. I have never seen it available as library for general purpose use.

[darkbasic]

I have never seen it available as library for general purpose use.

https://gitlab.freedesktop.org/pulseaudio/webrtc-audio-processing

I think that pulseaudio used to do clock drift compensation, but pipewire (which uses the same library) doesn't: https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/master/spa/plugins/aec/aec-webrtc.cpp#L103

[wwmm]

I have updated our master branch fixing the regression about the probe not being linked. But I think that the other issues are just the speex library not being as good as webrtc. Or it needs to be hooked at the server side for it to actually be reliable...

I think that pulseaudio used to do clock drift compensation, but pipewire (which uses the same library) doesn't: >https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/master/spa/plugins/aec/aec-webrtc.cpp#L103

I will try to find some time to take a look at it. The library itself seems to have no documentation at all. So looking at how people have been using it seems to be the only way...

[darkbasic]

I have updated our master branch fixing the regression about the probe not being linked

Thanks, I can confirm it's working now: https://youtu.be/iZRdZ804ZM0

But I think that the other issues are just the speex library not being as good as webrtc.

That or the drift compensation. If you have a sound card with an input for a microphone can you check if it's better?

I will try to find some time to take a look at it.

Thanks, that's really appreciated!

[wwmm]

That or the drift compensation.

And if drift is the problem I am afraid that only on the server side something can be done about it.

If you have a sound card with an input for a microphone can you check if it's better?

At this moment I am testing with my usb microphone. I did not try to test with a mic in the soundcard jack. In Discord's mic test I can see that now that the probes are linked the echo canceller seems to help to remove some of the echo. But complete audio removal seems impossible. Or at least finding the correct parameter values for this seems an impossible mission...

[wwmm]

At this moment I am testing with my usb microphone

I tried with one in the soundcard jack and the performance is as disappointing as using a usb mic. The echo duration is shorter with the plugin. But this seems the most it can do in my current setup... Maybe we a clever selection of frame size and filter length things may get better. But it is not obvious how to customize them for a given environment.

[wwmm]

Pulseaudio echo cancellation (backed by webrtc-audio-processing) is so good that you won't be able to tell if there is anything playing back in the background.

@darkbasic I've noticed a silly mistake in a call to speex_preprocess_run. After the latest updates to our master branch now I actually see a considerable attenuation in the background music captured by my mic. It probably isn't as good as the server built-in webrtc plugin but at least on my side it has been improved by a lot. When I bypass the echo canceller the signal displayed by the spectrum plugin is clearly a lot more intense.

[wwmm]

If I add the Speech Processor plugin after the echo canceller there is almost no activity in the spectrum even if voice detection is disabled. So using them together is a good idea.

[darkbasic]

Uhm... it's definitely a little bit better but still not that good: https://youtu.be/MoVZJ6O4RJY
There are moments where it's decent and then a second later it suddenly jumps several decibels louder for no reason.
Speech Processor didn't seem to help a lot.

[wwmm]

Uhm... it's definitely a little bit better but still not that good: https://youtu.be/MoVZJ6O4RJY

Uhm... I see... I think I will have to do some tests with OBS to see if there is something different when using it. I was testing with Discord's mic test and Spotify. After starting the mic test I made Spotify playing some music in my speakers and after inserting the echo canceller the music captured by the microphone was almost completely gone from the Spectrum.

By the way it is strange that there is no spectrum activity in the video after enabling effects for OBS mic stream.

[wwmm]

I did a test with OBS. The result is in this video https://www.youtube.com/watch?v=apRiSlpNE5k. OBS was recording my desktop while I played music on Spotify. As far as I can see the plugin behaved as expected. The music was removed from the microphone signal processed by OBS. The video has audible music all the time because OBS is recording what is being sent to my speakers. But as far as the mic signal is concerned what was being captured from the speakers was removed. Are we testing different things?

[wwmm]

Are we testing different things?

The signal levels you have in your microphone pipeline are a lot higher than the ones I am getting from my microphone. So at least this is quite different.

[darkbasic]

The signal levels you have in your microphone pipeline are a lot higher than the ones I am getting from my microphone. So at least this is quite different.

I was going to say the same thing. My microphone gain has been set to maximize the SNR because when you talk over background music (or over someone else in a conference) the echo cancelled signal will be very weak before going through the auto gain and then the compressor. I don't think it is working better on your computer, I just think that you notice less issues because the overall signal is much weaker to start with.

At this point I fear that SpeexDSP is just not good enough for the task.

[wwmm]

I don't think it is working better on your computer, I just think that you notice less issues because the overall signal is much weaker to start with.

After putting before the echo canceller a plugin with the output gain set to the maximum the echo canceller was useless. So "too high" signal levels in its input are definitely a problem.

My microphone gain has been set to maximize the SNR because when you talk over background music (or over someone else in a conference) the echo cancelled signal will be very weak before going through the auto gain and then the compressor.

I see. I guess this will change a lot from one environment and hardware to another. Here in my room amplifying the echo canceller or the speech processor output was enough.

[darkbasic]

I patched webrtc-audio-processing to work on ppc64 and this is the result with pipewire's echo-cancel module: https://youtu.be/_oqo5tzXHMY

As you can see you cannot basically hear anything once the echo-cancel kicks in.

[darkbasic]

I've also discovered that drift compensation in the echo-canceler itself is not needed because PipeWire already does drift compensation so that all hardware looks like it is using the same clock.

[jeffshee]

Hi, I tried playing around with the libpipewire-module-echo-cancel module and managed to achieve a good result.
https://docs.pipewire.org/page_module_echo_cancel.html

I wonder if it's possible to incorporate this kind of configuration into easyeffects.

[wwmm]

I wonder if it's possible to incorporate this kind of configuration into easyeffects.

This kind of pipeline is already what happens when putting the echo canceller plugin in both pipelines. The main difference is probably the implementation of the pipewire plugin being better at handling strong signals in its input.

[jeffshee]

The main difference is probably the implementation of the pipewire plugin being better at handling strong signals in its input.

Yeah, probably it's the case. Meanwhile, I think it would be nice to share my configuration (libpipewire-module-echo-cancel + easyeffects) here, so anyone interested can try it. Tested with Fedora 38 and Fedora 39.

1. Prepare directory and config file for wireplumber.

mkdir -p ~/.config/wireplumber/
cp /usr/share/wireplumber/wireplumber.conf ~/.config/wireplumber

2. Edit ~/.config/wireplumber/wireplumber.conf. Add these lines into context.modules = [].

  # Echo Cancellation.
  {
    name = libpipewire-module-echo-cancel
    args = {
        # library.name  = aec/libspa-aec-webrtc
        # node.latency = 1024/48000
        # monitor.mode = false
        capture.props = {
          node.name         = "Echo Cancellation Capture"
          node.description  = "Echo Cancellation Capture"
          # Fake application.id to skip GNOME Shell's InputIndicator
          application.id    = org.gnome.VolumeControl
        }
        source.props = {
          node.name         = "Echo Cancellation Source"
          node.description  = "Echo Cancellation Source"
        }
        sink.props = {
          node.name         = "Echo Cancellation Sink"
          node.description  = "Echo Cancellation Sink"
        }
        playback.props = {
          node.name         = "Echo Cancellation Playback"
          node.description  = "Echo Cancellation Playback"
        }
      }
  }

3. Restart wireplumber service. If easyeffects is running at the background, re-login.

systemctl --user restart wireplumber.service

4. In easyeffects, configure like this:
   In Output, make sure the "Echo Cancellation Playback" is excluded.
   (You might want to save this setting into your presets if you are using those.)
   
   In Device Management, uncheck both "Use Default Input" and "Use Default Output".
   Instead, select "Echo Cancellation Source" and "Echo Cancellation Sink", respectively.
   
5. You should be able to use easyeffects normally with libpipewire-module-echo-cancel always enabled.
   You don't need to add "echo canceller" in easyeffects, just add "noise reduction" and it should be enough.

Note:
In your system settings/applications, always select the actual input/output devices, not the "Easy Effect" or "Echo Cancellation" source/sink!

Hope this helps! 😉

[alexmaras]

I had some issues with the method @jeffshee laid out recently - it seems that the wireplumber config needs to be moved into the pipewire config instead now. I'm not totally sure on why. There have been some mentions of config changes recently, and my audio system was broken when this happened and I needed to clear out my config and use the new one, so perhaps it's related to that.

I now instead have the exact same config in ~/.config/pipewire/pipewire.conf.d/echo-sink.conf and it works perfectly. No other notes really, it just works as-is if I shift the config into there instead.