You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Thank you for sharing such an insightful work and the detailed deployment instructions! While preparing for a real-world deployment of ReKep, I encountered challenges in updating the keypoint_movable_mask without the concept of object in the real-world environment. Currently, in the simulation environment, keypoint_movable_mask is tightly coupled with object-level information and is updated based on whether the corresponding object of a keypoint is grasped. Specifically, the function _update_keypoint_movable_mask depends on methods like self.env.get_object_by_keypoint() and self.env.is_grasping().
Problem Description:
In real-world environments:
There is no explicit "object" abstraction as in the simulation.
Keypoints may be defined on deformable or extended objects (e.g., one end of a stick or multiple parts of clothing). Using proximity to the end-effector or simple visual clues to decide movable states can lead to inconsistencies:
Example Issue: If a gripper holds one end of a long stick, keypoints on the opposite end of the stick may not be marked as movable, even though they should be, because there is no global "object" abstraction to infer that the keypoints belong to the same rigid body.
It would be helpful if you give offer a specific recommendation or example implementation for keypoint_movable_mask updates in the absence of object concepts, as this is critical for closed-loop replanning.
Thank you for your time and for providing such a powerful framework. I look forward to hearing your insights on this matter!
Best regards,
The text was updated successfully, but these errors were encountered:
Hi @hhmr123@huangwl18 , I encountered the same problem. Have you solved this problem or maybe some possible solutions, like directly delete the keypoint_movable_mask?
Hi authors,
Thank you for sharing such an insightful work and the detailed deployment instructions! While preparing for a real-world deployment of ReKep, I encountered challenges in updating the keypoint_movable_mask without the concept of object in the real-world environment. Currently, in the simulation environment, keypoint_movable_mask is tightly coupled with object-level information and is updated based on whether the corresponding object of a keypoint is grasped. Specifically, the function _update_keypoint_movable_mask depends on methods like self.env.get_object_by_keypoint() and self.env.is_grasping().
Problem Description:
In real-world environments:
There is no explicit "object" abstraction as in the simulation.
Keypoints may be defined on deformable or extended objects (e.g., one end of a stick or multiple parts of clothing). Using proximity to the end-effector or simple visual clues to decide movable states can lead to inconsistencies:
Example Issue: If a gripper holds one end of a long stick, keypoints on the opposite end of the stick may not be marked as movable, even though they should be, because there is no global "object" abstraction to infer that the keypoints belong to the same rigid body.
It would be helpful if you give offer a specific recommendation or example implementation for keypoint_movable_mask updates in the absence of object concepts, as this is critical for closed-loop replanning.
Thank you for your time and for providing such a powerful framework. I look forward to hearing your insights on this matter!
Best regards,
The text was updated successfully, but these errors were encountered: