Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Questions for Handling keypoint_movable_mask in Real-World Deployment Without Object Concepts #32

Open
hhmr123 opened this issue Dec 19, 2024 · 1 comment

Comments

@hhmr123
Copy link

hhmr123 commented Dec 19, 2024

Hi authors,

Thank you for sharing such an insightful work and the detailed deployment instructions! While preparing for a real-world deployment of ReKep, I encountered challenges in updating the keypoint_movable_mask without the concept of object in the real-world environment. Currently, in the simulation environment, keypoint_movable_mask is tightly coupled with object-level information and is updated based on whether the corresponding object of a keypoint is grasped. Specifically, the function _update_keypoint_movable_mask depends on methods like self.env.get_object_by_keypoint() and self.env.is_grasping().

Problem Description:

In real-world environments:

There is no explicit "object" abstraction as in the simulation.
Keypoints may be defined on deformable or extended objects (e.g., one end of a stick or multiple parts of clothing). Using proximity to the end-effector or simple visual clues to decide movable states can lead to inconsistencies:
Example Issue: If a gripper holds one end of a long stick, keypoints on the opposite end of the stick may not be marked as movable, even though they should be, because there is no global "object" abstraction to infer that the keypoints belong to the same rigid body.

It would be helpful if you give offer a specific recommendation or example implementation for keypoint_movable_mask updates in the absence of object concepts, as this is critical for closed-loop replanning.

Thank you for your time and for providing such a powerful framework. I look forward to hearing your insights on this matter!

Best regards,

@ge79puv
Copy link

ge79puv commented Dec 30, 2024

Hi @hhmr123 @huangwl18 , I encountered the same problem. Have you solved this problem or maybe some possible solutions, like directly delete the keypoint_movable_mask?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants