-
Notifications
You must be signed in to change notification settings - Fork 109
Question about RGB Activation Padding #4
Comments
Hey, good questions. The only reason for the sigmoid "padding" exists it that helped optimization in some corner cases, like when the learning rate is very large. This is because sigmoids saturate for very large or very small inputs, which causes the gradient to go to zero, which can cause optimization to catastrophically fail. Padding means that the model can emit 0 or 1 without saturating the sigmoid and therefore killing the gradients. Shrinking the range of the sigmoid is probably going to cause problems in some circumstances, because it means that absolute white and absolute black will both be inexpressible for the model. I'd expect this to drive rgb values to the tails of this sigmoid, which will cause dead gradients. It makes sense to me that playing with this parameter will affect how "clean" the |
Thanks for the thorough answer! It makes sense that shrunken sigmoid saturates and that will cause dead gradients for color, but my main thinking is that having it be padded allows an equivalence between |
Hi, this is super interesting work that seems to solve a key problem with NeRF.
One thing I noticed when reading through your paper was that you used a modified RGB activation function.
I tried using this padded sigmoid with normal NeRF, and I noticed that it tends to cause background pixels to have non-zero density because they can saturate all the way to black or white, and was wondering if you encountered the same thing? I was looking at the
acc
image returned from volumetric integration of the weights. I'm not sure if it's significant, but I was wondering whether you compared normal sigmoid to the widened one?I was also wondering if you experimented with shrinking the range of sigmoid? I tried lowering the range of sigmoid, and that seems to produce much cleaner
acc
s, at the cost of less RGB range, but to a negligible extent.Thanks!
The text was updated successfully, but these errors were encountered: