Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: ceastld/sam2
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: main
Choose a base ref
...
head repository: facebookresearch/sam2
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref
Checking mergeability… Don’t worry, you can still create the pull request.
  • 3 commits
  • 27 files changed
  • 1 contributor

Commits on Dec 11, 2024

  1. SAM 2 Update 12/11/2024 -- full model compilation for a major VOS spe…

    …edup and a new SAM2VideoPredictor to better handle multi-object tracking (facebookresearch#486)
    
    This PR provides new features and updates for SAM 2:
    
    - We now support `torch.compile` of the entire SAM 2 model on videos, which can be turned on by setting `vos_optimized=True` in `build_sam2_video_predictor` (it uses the new `SAM2VideoPredictorVOS` predictor class in `sam2/sam2_video_predictor.py`).
      * Compared to the previous setting (which only compiles the image encoder backbone), the new full model compilation gives a major speedup in inference FPS.
      * In the VOS prediction script `tools/vos_inference.py`, you can specify this option in `tools/vos_inference.py` via the `--use_vos_optimized_video_predictor` flag.
      * Note that turning on this flag might introduce a small variance in the predictions due to numerical differences caused by `torch.compile` of the full model.
      * **PyTorch 2.5.1 is the minimum version for full support of this feature**. (Earlier PyTorch versions might run into compilation errors in some cases.) Therefore, we have updated the minimum PyTorch version to 2.5.1 accordingly in the installation scripts.
    - We also update the implementation of the `SAM2VideoPredictor` class for the SAM 2 video prediction in `sam2/sam2_video_predictor.py`, which allows for independent per-object inference. Specifically, in the new `SAM2VideoPredictor`:
      * Now **we handle the inference of each object independently** (as if we are opening a separate session for each object) while sharing their backbone features.
      * This change allows us to relax the assumption of prompting for multi-object tracking. Previously (due to the batching behavior in inference), if a video frame receives clicks for only a subset of objects, the rest of the (non-prompted) objects are assumed to be non-existent in this frame (i.e., in such frames, the user is telling SAM 2 that the rest of the objects don't appear). Now, if a frame receives clicks for only a subset of objects, we do not make any assumptions about the remaining (non-prompted) objects (i.e., now each object is handled independently and is not affected by how other objects are prompted). As a result, **we allow adding new objects after tracking starts** after this change (which was previously a restriction on usage).
      * We believe that the new version is a more natural inference behavior and therefore switched to it as the default behavior. The previous implementation of `SAM2VideoPredictor` is backed up to in `sam2/sam2_video_predictor_legacy.py`. All the VOS inference results using `tools/vos_inference.py` should remain the same after this change to the `SAM2VideoPredictor` class.
    ronghanghu authored Dec 11, 2024
    Configuration menu
    Copy the full SHA
    393ae33 View commit details
    Browse the repository at this point in the history

Commits on Dec 12, 2024

  1. patch for the case of offload_state_to_cpu=True in the new `SAM2Vid…

    …eoPredictor` (facebookresearch#490)
    
    This PR adds a pathc for the case of `offload_state_to_cpu=True` where `pred_masks` might have been offload to CPU device (close facebookresearch#489)
    ronghanghu authored Dec 12, 2024
    Configuration menu
    Copy the full SHA
    722d1d1 View commit details
    Browse the repository at this point in the history

Commits on Dec 16, 2024

  1. remove .pin_memory() in obj_pos of SAM2Base to resolve and erro…

    …r in MPS (facebookresearch#495)
    
    In this PR, we remove `.pin_memory()` in `obj_pos` of `SAM2Base` to resolve and error in MPS. Investigations show that `.pin_memory()` causes an error of `Attempted to set the storage of a tensor on device "cpu" to a storage on different device "mps:0"`, as originally reported in facebookresearch#487.
    
    (close facebookresearch#487)
    ronghanghu authored Dec 16, 2024
    Configuration menu
    Copy the full SHA
    2b90b9f View commit details
    Browse the repository at this point in the history
Loading