-
Notifications
You must be signed in to change notification settings - Fork 346
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ValueError with NaN tensors in gimVI training #2215
Comments
Hi, thanks for bringing up this up and sorry you're running into this issue. One possible source for these NaN errors are the exponential activations in the model - we've had similar issues before with other models depending on the dataset used as well as any non-default arguments that are passed during model initialization. Unfortunately, there's no straightforward way to modify these at the moment, and we cannot change them on our end because of reproducibility with the original manuscript. Would you be able to share the dataset as well as a reproducible notebook? |
Thanks for the fast response. Here is a link to a minimal notebook and data files for reproducing the errors (when I tested it today, gimVI seems to fail on the 32 gene data and succeeds on the 28 gene data, so maybe there are some variable behaviors there): https://www.dropbox.com/sh/5smbltpudpntmh4/AABrxiGL8jlNxdq3dvHOx3bga?dl=0 |
Thanks! I'll try to get this running and get back to you |
Hello, I am also running into the same issue. Is there a way steps to modifying could be publicly posted? |
any news on this? I am also experiencing the same error (mine is when running |
PS I tried with the same code and same object using an old environment (with scvi 0.2.3 installed) and it ran. the env where I was getting this error had 1.0.3 installed |
with the environment which previously gave this issue activated, I ran: pip uninstall scvi-tools
pip install scvi-tools==0.20.3 then restarted the kernel and the notebook ran without error. |
With |
One possible cause of this issue is invalid data input, I guess the input needs to be a non-negative matrix. BTW, it would be reasonable to add an assert line at the outer modules, avoiding misdirections like this. |
Has there been any updates on this? I have tried running different versions (0.19.0, 0.20.3, and most recently 1.0.4) but none of them consistently work for the test data and notebook. Using scanpy==1.9.6 and scvi-tools==1.0.4, I did notice that if I restart the kernel a few times, sometimes it is able to complete the model.train(200) call successfully so perhaps it is due to some stochastic part of the model? |
Hi, I would also like to hear if someone solved the problem, in my case model.train() function runs with 2 epochs but everything more than that and I am getting the same value error: |
Same story here - downgraded from 1.0.4 to 0.20.3 and MultiVI ran fine. Dropping a comment mostly to be notified if something budges on this front. |
When training the gimVI model, I am running into a ValueError in the first epoch for some datasets and not others. In all cases, the inputs are AnnData objects with raw counts for both the RNAseq and for the spatial data (dtype=float64). I have tried filtering out cells with zero counts and/or normalizing the RNAseq data but am still running into the same error. Strangely, gimVI seems to be able to train successfully on one dataset but when I remove 4 genes from the spatial data (32 -> 28 genes), it fails on that dataset. Happy to share data if that would be helpful and if there are suggestions for doing so (screenshots of basic data info below).
Versions:
The text was updated successfully, but these errors were encountered: