Vertical bars in audio from model output

[KimberleyJensen]

Hi, i was checking the outputs of my Mel-Band Roformer model and i see lots of vertical lines in the audio that aren't there in the input. I am using this code for inference https://github.com/ZFTurbo/Music-Source-Separation-Training/blob/main/inference.py.

Here are my model settings

model:
dim: 192
depth: 8
stereo: true
num_stems: 1
time_transformer_depth: 1
freq_transformer_depth: 1
num_bands: 60
dim_head: 64
heads: 8
attn_dropout: 0
ff_dropout: 0
flash_attn: True
dim_freqs_in: 1025
sample_rate: 44100 # needed for mel filter bank from librosa
stft_n_fft: 2048
stft_hop_length: 512
stft_win_length: 2048
stft_normalized: False
mask_estimator_depth: 2
multi_stft_resolution_loss_weight: 1.0
multi_stft_resolutions_window_sizes: !!python/tuple

• 4096
• 2048
• 1024
• 512
• 256
  multi_stft_hop_size: 147
  multi_stft_normalized: False

I am training with a batch size of 12, 8 seconds of audio, learning rate 5.0e-05

Picture of the vertical bars

Has anyone else experienced this and managed to fix it? (My model has only trained for 24 hours with a single GPU so its possible as the model trains longer this problem goes away.)

[iver56]

Hey Kimberley Jensen! Love your work in the source separation space.

Please try again with a different window function for the (i)stft. I think the authors of the original paper used hann, which should be much more appropriate than what you have right now. The current default is all ones, which (I think) leads to those unwanted little clicks at a regular interval. The click interval matches the hop length. It would be nice to have this fixed upstream too.

Pytorch is in the process of switching their default window function to hann, but it takes time, so until then we have to set it explicitly.

[lucidrains]

@iver56 thank you for sharing your domain expertise! i followed your instructions and hann window should be the default windowing function in 0.3.7 (customizable)

[KimberleyJensen]

Thank you both @iver56 @lucidrains always so helpful i really appreciate it !!

[iver56]

Update: @lucidrains patched it in 0.3.7. Excellent response time 🤘😄

[ZFTurbo]

There is a problem now:

RuntimeError: stft input and window must be on the same device but got self on cuda:0 and window on cpu

We need to do something with device.

I fixed it in ugly way becasue self.device is incorrect during initialization:

window=default(stft_window, torch.hann_window(stft_win_length, device='cuda:0'))
...
freqs = torch.stft(torch.randn(1, 4096).to('cuda:0'), **self.stft_kwargs, return_complex=True).shape[1]
...
window=self.multi_stft_window_fn(window_size, device=self.device),

[lucidrains]

ohh didn't realize it returned a tensor

try 0.3.8?

[ZFTurbo]

Now it works. Thank you.