Spontanious validation error

[Poly2it]

After running for some time, this program may run into fatal errors. The rest of my system was running in a faultless state the whole time.

thread 'main' panicked at /home/bach/.cargo/git/checkouts/wgpu-f9afa33caa1e84c9/ffb4852/wgpu/src/backend/wgpu_core.rs:23
Error in Queue::submit: Validation Error

Caused by:
  Parent device is lost

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[ArthurBrussee]

Unfortunately that error is basically what happens if anything goes wrong, so it's reallly hard to say :/

What platforms is this on? Is this during training or rendering? How many steps had you run? Did it work ok before that?

[Poly2it]

Unfortunately that error is basically what happens if anything goes wrong, so it's reallly hard to say :/

What platforms is this on? Is this during training or rendering? How many steps had you run? Did it work ok before that?

During training, on Linux. In my experience it is independent of previous iterations, and seems not to be an issue of a lack of system resources. Maybe there should be some printouts?

[ArthurBrussee]

Does this by any chance happen at the same time as #28 ?

[Poly2it]

Does this by any chance happen at the same time as #28 ?

No, this leaves a trace, #28 is a hard termination.

[ArthurBrussee]

Hiya, I'm wondering if this still happens after some recent stability updates. I have a suspicion of what was causing it which is fixed now.

[Poly2it]

Hiya, I'm wondering if this still happens after some recent stability updates. I have a suspicion of what was causing it which is fixed now.

I have not experienced it as of recently. I could see if I can recreate it before/after the fix commit if you'd like to link it.