add noise and reverberance version for BinauralWSJ0Mix - #1502

@ycemsubakan

Hi @ycemsubakan , do you have time for checking this PR please?

Hi @huangzj421 , sorry for the delay. I will try to do it soon.

@huangzj421 , Could you add in the README the instructions on how to run the model with these noisy conditions? As before, the script should automatically download and create the corresponding dataset. I would like to try myself if everything works correctly.

For convtasnet-parallel-noisy.yaml I am getting this error. You can see the command I used also here.

And, I am observing the following error for convtasnet-parallel-reverb.yaml:

Please note that I tried convtasnet-parallel.yaml, and that seems to work fine.
Correction: Actually, I am also observing that the is stuck at 80, and it doesn't go down for `convtasnet-parallel.yaml'. In the previous PR, I tried and it the error was going down as expected.. I am not sure what's happening here..

For convtasnet-parallel-noisy.yaml I am getting this error. You can see the command I used also here.

I have corrected the data generation scripts and now these should work fine.

@huangzj421 , Could you add in the README the instructions on how to run the model with these noisy conditions? As before, the script should automatically download and create the corresponding dataset. I would like to try myself if everything works correctly.

The usage is the same as the previous PR which you should only set wsj_root and output_root and it will download and generate data into output_root.

And, I am observing the following error for convtasnet-parallel-reverb.yaml:

Please note that I tried convtasnet-parallel.yaml, and that seems to work fine. Correction: Actually, I am also observing that the is stuck at 80, and it doesn't go down for `convtasnet-parallel.yaml'. In the previous PR, I tried and it the error was going down as expected.. I am not sure what's happening here..

I am sorry that I don't understand the "Correction" part you said. Also the test check failed but I don't know why.

Actually, if the dataset is already created, we should not require wsj_root again, right?

What I mean with the 'correction' part is that, when I run the script, (with convtasnet_parallel.yaml, the error does not go down. It is stuck at 80.

Actually, if the dataset is already created, we should not require wsj_root again, right?

Yes, if the dataset is done and dynamic mixing don't need it in the hparams["base_folder_dm"].

OK, let me check it.

Yes, but we would also not need wsj_root if the dataset is already created.

For convtasnet-parallel-noisy.yaml I am getting this error. You can see the command I used also here.

I have corrected the data generation scripts and now these should work fine.

Ok, can you push your code after this correction in the data generation script?

For convtasnet-parallel-noisy.yaml I am getting this error. You can see the command I used also here.

I have corrected the data generation scripts and now these should work fine.

Ok, can you push your code after this correction in the data generation script?

Oh, it doesn't need to push cause I just corrected in my repo.

For convtasnet-parallel-noisy.yaml I am getting this error. You can see the command I used also here.

I have corrected the data generation scripts and now these should work fine.

Ok, can you push your code after this correction in the data generation script?

Oh, it doesn't need to push cause I just corrected in my repo.

I am now getting this error:

For convtasnet-parallel-noisy.yaml I am getting this error. You can see the command I used also here.

I have corrected the data generation scripts and now these should work fine.

Ok, can you push your code after this correction in the data generation script?

Oh, it doesn't need to push cause I just corrected in my repo.

I am now getting this error:

It seems that the download link is invalid. I am trying to search another one.

What I mean with the 'correction' part is that, when I run the script, (with convtasnet_parallel.yaml, the error does not go down. It is stuck at 80.

What is the unit of 80? 80%?

Hi @ycemsubakan , I have fixes some errors in train.py and the data generation scripts in my repo. After you remove the previous BinauralWSJ0Mix folder and run train.py again, it will download automatically and train smoothly.
But I still don't know the 80 stuck you said and pytest check failure. Thanks.

Hi @ycemsubakan , I have fixes some errors in train.py and the data generation scripts in my repo. After you remove the previous BinauralWSJ0Mix folder and run train.py again, it will download automatically and train smoothly. But I still don't know the 80 stuck you said and pytest check failure. Thanks.

• The test failures is a general problem at the moment due to an external dependency. (not because of this PR)
• The error is basically stuck for some reason at 80 dB negative SI-SNR and it doesn't go down. When you created the first PR, I trained the model for an epoch, and it was fine.

The noisy version seems to work after your last push. (the data generation)

But the loss is still kind of stuck:

The noisy version seems to work after your last push. (the data generation)

But the loss is still kind of stuck:

It was weired cause I just trained and the loss went down well:

My command was:
CUDA_VISIBLE_DEVICES=3,5,6,7 python -m torch.distributed.launch --nproc_per_node=4 train.py hparams/convtasnet-parallel.yaml --data_folder /mnt/test/ --wsj_root /mnt/wsj/ --distributed_launch --distributed_backend='nccl'
Your loss was too big, did you use the dynamic mixing?

The noisy version seems to work after your last push. (the data generation)
But the loss is still kind of stuck:

It was weired cause I just trained and the loss went down well: My command was: CUDA_VISIBLE_DEVICES=3,5,6,7 python -m torch.distributed.launch --nproc_per_node=4 train.py hparams/convtasnet-parallel.yaml --data_folder /mnt/test/ --wsj_root /mnt/wsj/ --distributed_launch --distributed_backend='nccl' Your loss was too big, did you use the dynamic mixing?

No, I am not using dynamic mixing. I am training on a single gpu. Can you try training on a single gpu?

The noisy version seems to work after your last push. (the data generation)
But the loss is still kind of stuck:

It was weired cause I just trained and the loss went down well: My command was: CUDA_VISIBLE_DEVICES=3,5,6,7 python -m torch.distributed.launch --nproc_per_node=4 train.py hparams/convtasnet-parallel.yaml --data_folder /mnt/test/ --wsj_root /mnt/wsj/ --distributed_launch --distributed_backend='nccl' Your loss was too big, did you use the dynamic mixing?

No, I am not using dynamic mixing. I am training on a single gpu. Can you try training on a single gpu?

It was not so big as well:

The noisy version seems to work after your last push. (the data generation)
But the loss is still kind of stuck:

It was weired cause I just trained and the loss went down well: My command was: CUDA_VISIBLE_DEVICES=3,5,6,7 python -m torch.distributed.launch --nproc_per_node=4 train.py hparams/convtasnet-parallel.yaml --data_folder /mnt/test/ --wsj_root /mnt/wsj/ --distributed_launch --distributed_backend='nccl' Your loss was too big, did you use the dynamic mixing?

No, I am not using dynamic mixing. I am training on a single gpu. Can you try training on a single gpu?

It was not so big as well:

Hm.. weird.. I am not sure why this is happening when I try. This doesn't happen with other recipes. I'll try to understand what's happening.

The noisy version seems to work after your last push. (the data generation)
But the loss is still kind of stuck:

It was weired cause I just trained and the loss went down well: My command was: CUDA_VISIBLE_DEVICES=3,5,6,7 python -m torch.distributed.launch --nproc_per_node=4 train.py hparams/convtasnet-parallel.yaml --data_folder /mnt/test/ --wsj_root /mnt/wsj/ --distributed_launch --distributed_backend='nccl' Your loss was too big, did you use the dynamic mixing?

No, I am not using dynamic mixing. I am training on a single gpu. Can you try training on a single gpu?

It was not so big as well:

Hm.. weird.. I am not sure why this is happening when I try. This doesn't happen with other recipes. I'll try to understand what's happening.

Maybe you could remove all the dataset and train it again?

The noisy version seems to work after your last push. (the data generation)
But the loss is still kind of stuck:

It was weired cause I just trained and the loss went down well: My command was: CUDA_VISIBLE_DEVICES=3,5,6,7 python -m torch.distributed.launch --nproc_per_node=4 train.py hparams/convtasnet-parallel.yaml --data_folder /mnt/test/ --wsj_root /mnt/wsj/ --distributed_launch --distributed_backend='nccl' Your loss was too big, did you use the dynamic mixing?

No, I am not using dynamic mixing. I am training on a single gpu. Can you try training on a single gpu?

It was not so big as well:

Hm.. weird.. I am not sure why this is happening when I try. This doesn't happen with other recipes. I'll try to understand what's happening.

Maybe you could remove all the dataset and train it again?

Yes, I already tried that. I generated the dataset from scratch.

Hi @ycemsubakan, do you have any updates?

@huangzj421 No, I will try to get to it this week.

@huangzj421 No, I will try to get to it this week.

Okay, thanks!

With convtasnet-parallel-noise.yaml, I observed the following error. Do you see it also?

With convtasnet-parallel-noise.yaml, I observed the following error. Do you see it also?

No, I guess you should update the automatic downloaded data generation scripts from https://github.com/huangzj421/Binaural-WSJ0Mix or you can remove the old version in your data_folder.

With convtasnet-parallel-noise.yaml, I observed the following error. Do you see it also?

No, I guess you should update the automatic downloaded data generation scripts from https://github.com/huangzj421/Binaural-WSJ0Mix or you can remove the old version in your data_folder.

I see.. It was working earlier. But, this could be an issue for people who use this recipe in the future. Can you automatically check if the folders are correct, and then if not, you can either automatically fix, or at least you need to give an error message so that people know that they should delete the folder, and create a new one.

@huangzj421 , all of the .yaml files for me now work. For some reason when I tried with my conda environment, as I mentioned the training was not going down. But with a standard virtual environment it works fine.

I think you can do the version check for the dataset in a separate PR, and if everything is fine with @mravanelli, we can merge.

@huangzj421 , all of the .yaml files for me now work. For some reason when I tried with my conda environment, as I mentioned the training was not going down. But with a standard virtual environment it works fine.

I think you can do the version check for the dataset in a separate PR, and if everything is fine with @mravanelli, we can merge.

That's great, thanks! 'Separate PR' means opening a new PR?

@huangzj421 , all of the .yaml files for me now work. For some reason when I tried with my conda environment, as I mentioned the training was not going down. But with a standard virtual environment it works fine.
I think you can do the version check for the dataset in a separate PR, and if everything is fine with @mravanelli, we can merge.

That's great, thanks! 'Separate PR' means opening a new PR?

Yes. If you already have a solution for dataset versioning, feel free to push it here. If you think it's going to take time, we can open a new PR, after we merge this one.

@huangzj421 , all of the .yaml files for me now work. For some reason when I tried with my conda environment, as I mentioned the training was not going down. But with a standard virtual environment it works fine.
I think you can do the version check for the dataset in a separate PR, and if everything is fine with @mravanelli, we can merge.

That's great, thanks! 'Separate PR' means opening a new PR?

Yes. If you already have a solution for dataset versioning, feel free to push it here. If you think it's going to take time, we can open a new PR, after we merge this one.

Okay, got it. But some checks of PR failed, how can we merge it?

@huangzj421 , all of the .yaml files for me now work. For some reason when I tried with my conda environment, as I mentioned the training was not going down. But with a standard virtual environment it works fine.
I think you can do the version check for the dataset in a separate PR, and if everything is fine with @mravanelli, we can merge.

That's great, thanks! 'Separate PR' means opening a new PR?

Yes. If you already have a solution for dataset versioning, feel free to push it here. If you think it's going to take time, we can open a new PR, after we merge this one.

Okay, got it. But some checks of PR failed, how can we merge it?

The checks are failing because of a general problem. It will be fixed soon.

@huangzj421 , all of the .yaml files for me now work. For some reason when I tried with my conda environment, as I mentioned the training was not going down. But with a standard virtual environment it works fine.
I think you can do the version check for the dataset in a separate PR, and if everything is fine with @mravanelli, we can merge.

That's great, thanks! 'Separate PR' means opening a new PR?

Yes. If you already have a solution for dataset versioning, feel free to push it here. If you think it's going to take time, we can open a new PR, after we merge this one.

Okay, got it. But some checks of PR failed, how can we merge it?

The checks are failing because of a general problem. It will be fixed soon.

Okay, thank you!

Hi @mravanelli , do you have time to check this PR? Cem has reviewed it and everything is fine. Thanks.

Everything seems fine to me. I ran the code one other time. I did some updates to README. I will merge once the tests pass.