Question about pixel_coords.

[whyygug]

What's the meaning of "make sure that no point in warped image is a combinaison of im and gray" ?
It seems that points outside the image boundaries will still be set to 0, if don't use the code above.
Can you explain in detail what this code does? Thank you！

[suvigy]

I would also be interested in it. :)
As I can see you first tried also without it (in inverse_warp and cam2pixel) but did you have some artifacts on depth and that's why you switched to the solution mentioned by @whyygug? What can one expect without setting this out of pixels to 2? (if we just create the valid mask based on the image-inside coordinates)

[mhariat]

Good point.

When you set align_corners to False (default setting), you’re actually considering pixels as squares with the middle of the square as the pixel reference. So you could end-up with out-of-bounds in this setting that are still valid points with respect to the valid_mask. The fact that the grid_sample function and the valid_mask are not aligned on their interpretation of the out-of-bounds could create artifacts on the border (although I haven't experimented any with SC-Sfm). That’s why in the code above, the author is actually detaching the gradient from those points to avoid any unexpected behaviors.

It’s interesting to note that when align_corners is set to True, the valid_mask and the out-of-bounds interpretation of grid_sample are consistent and it doesn’t make any difference to have padding_mode set to “zeros” or to “border” because the valid_mask will remove those points afterward.