matrix c2w in loc_inference.py

[kongbia]

In loc_inference.py, $R,t$ generated by PnP represents the world-to-camera transformation in my understanding. Why assign it to the $c2w$ matrix, which seems to represent the camera-to-world transform in compute_warping_loss

c2w = torch.eye(4, 4, device='cuda') 
c2w[:3, :3] = torch.from_numpy(R).float()
c2w[:3, 3] = torch.from_numpy(t[:, 0]).float()

[haksorus]

Sorry for the late reply.

It's actually the world-to-camera matrix.
Thus, predicted/optimized poses as well as GT poses from 3DGS cameras are presented in w2c format. In this case, the matrix was given the wrong name, I will fix it soon.

Thanks for noticing!

[kongbia]

Sorry for the late reply.

It's actually the world-to-camera matrix. Thus, predicted/optimized poses as well as GT poses from 3DGS cameras are presented in w2c format. In this case, the matrix was given the wrong name, I will fix it soon.

Thanks for noticing!

Thanks for your reply, I still have a minor question.
In compute_warping_loss function

def compute_warping_loss(vr, qr, quat_opt, t_opt, pose, K, depth):
    warp = pose @ from_cam_tensor_to_w2c(torch.cat([quat_opt, t_opt], dim=0)).inverse()
    warped_image = differentiable_warp(vr.unsqueeze(0), depth.unsqueeze(0), warp.unsqueeze(0), K.unsqueeze(0))
    loss = F.mse_loss(warped_image, qr.unsqueeze(0))

    return loss

warp first implements the c2w back-projection via initial pose pose, and then implements w2c projection via optimized pose quat_opt, t_opt. Howver, the pose in the code represents w2c, while the from_cam_tensor_to_w2c(torch.cat([quat_opt, t_opt], dim=0)).inverse() represents c2w, which seems not to be consistent with the principle.

[haksorus]

Sorry for the late reply.
It's actually the world-to-camera matrix. Thus, predicted/optimized poses as well as GT poses from 3DGS cameras are presented in w2c format. In this case, the matrix was given the wrong name, I will fix it soon.
Thanks for noticing!

Thanks for your reply, I still have a minor question. In compute_warping_loss function

def compute_warping_loss(vr, qr, quat_opt, t_opt, pose, K, depth):
    warp = pose @ from_cam_tensor_to_w2c(torch.cat([quat_opt, t_opt], dim=0)).inverse()
    warped_image = differentiable_warp(vr.unsqueeze(0), depth.unsqueeze(0), warp.unsqueeze(0), K.unsqueeze(0))
    loss = F.mse_loss(warped_image, qr.unsqueeze(0))

    return loss

warp first implements the c2w back-projection via initial pose pose, and then implements w2c projection via optimized pose quat_opt, t_opt. Howver, the pose in the code represents w2c, while the from_cam_tensor_to_w2c(torch.cat([quat_opt, t_opt], dim=0)).inverse() represents c2w, which seems not to be consistent with the principle.

Here the warp is defined as warp = w2c @ c2w
So in the differentiable_warp function the part

    cam_points = warp @ world_points # (B, 4, H*W)

is equal to

    world_points = c2w @ world_points
    cam_points = w2c @ world_points  # (B, 4, H*W)

I hope this helps.

[kongbia]

Sorry to bother you again.
Since warp = w2c @ c2w, then

w2c = pose 
c2w = from_cam_tensor_to_w2c(torch.cat([quat_opt, t_opt], dim=0)).inverse()

But according to the paper it should perform a c2w back-projection with the initial pose pose.inverse() and a w2c projection with the optimized pose quat_opt, t_opt.
However, when I usewarp = from_cam_tensor_to_w2c(torch.cat([quat_opt, t_opt], dim=0)) @ pose.inverse(), it seems that the result is not correct. Could you please help me?

[haksorus]

However, when I usewarp = from_cam_tensor_to_w2c(torch.cat([quat_opt, t_opt], dim=0)) @ pose.inverse(), it seems that the result is not correct. Could you please help me?

You just need to take the inverse of warp function in this definition.

warp = from_cam_tensor_to_w2c(torch.cat([quat_opt, t_opt], dim=0)) @ pose.inverse()
warp = warp.inverse()

And the result should be correct.