Computer Vision for Surgical Applications Project

Overview

This project implements an advanced image segmentation system for surgical instruments using synthetic data generation, segmentation model development, and domain adaptation techniques. It simulates a real-world scenario where annotated medical data is unavailable due to the high cost of expert annotation.

The project is divided into three phases:

1. Synthetic Data Generation: Creating a synthetic dataset of surgical instrument images and segmentation masks.
2. Segmentation Model Development: Training and fine-tuning a DeepLabV3+ model on the synthetic dataset.
3. Domain Adaptation: Enhancing the model's performance on real surgical images by bridging the domain gap.

Additionally, the project provides two utility scripts:

• predict.py: For running segmentation on single images.
• video.py: For applying segmentation on video frames.

A link to the final model weights: DeepLab v3 model

---

Project Structure

project/
│
├── data_generation/          # Phase 1: Synthetic Data Generation
├── model_development/        # Phase 2: Segmentation Model Development
├── domain_adaptation/        # Phase 3: Domain Adaptation
├── predict.py                # Utility script for single-image predictions
├── video.py                  # Utility script for video segmentation
└── requirements/             # Dependency requirements for each phase

Key Scripts

predict.py

This script predicts segmentation masks for a single image using a trained DeepLabV3+ model.

Key Arguments

1. -i or --image_idx (optional):
   Specify the index of an image from the default path ./data_generation/output/jpg_format/.
   Example: -i 0 for the image 0_color.jpg.
   Cannot be used with --image_path.

2. -p or --image_path (optional):
   Provide the full path to the image file.
   Example: -p ./path/to/image.jpg.

3. -m or --model_path:
   Path to the trained model. Default: ./model_development/deeplabv3_model.pth.

4. -o or --output_dir:
   Directory to save the results. Must be specified unless --dev or --test is used.

5. --dev: Use the development directory (./model_development/predictions/) as the output location.

6. --test: Use the test directory (./domain_adaptation/predictions/) as the output location.

Rules and Examples

• Provide either --image_idx or --image_path (not both).
• Specify one output option: -o, --dev, or --test.

Example:

python predict.py -i 0 --dev
python predict.py -p ./image.jpg -o ./results/

video.py Arguments

This script processes a video and applies segmentation frame-by-frame.

Key Arguments

1. -v or --input_video_path (required):
   Path to the input video. Default: /datashare/project/vids_test/4_2_24_A_1.mp4.

2. -m or --model_path:
   Path to the trained model. Default: ./model_development/deeplabv3_model.pth.

3. -o or --output_dir:
   Directory to save the segmented video. Must be specified unless --dev or --test is used.

4. --dev: Use the development directory (./model_development/predictions/) as the output location.

5. --test: Use the test directory (./domain_adaptation/predictions/) as the output location.

Rules and Examples

• Specify one output option: -o, --dev, or --test.

Example:

python video.py -v ./input.mp4 --test
python video.py -o ./output/

---

Installation

1. Clone the repository:

   git clone https://github.com/AmitShirazi1/CV_for_Surgical_Applications_Project.git
   cd CV_for_Surgical_Applications_Project/

2. Install basic dependencies:

   pip install -r requirements/basic.txt

3. Install dependencies for the desired phase:

   pip install -r requirements/<phase>.txt

   Replace <phase> with data_generation, model_development, or domain_adaptation.

4. Install Blender for Phase 1 (Synthetic Data Generation):
   Download and Install Blender.

---

How to Run

Phase 1: Synthetic Data Generation

1. Generate synthetic images and segmentation masks:

   blenderproc run ./data_generation/synthetic_data_generator.py -b -n 1000

   Replace 1000 with the desired number of images.

2. Review and inspect the generated dataset: Open investigate_hdf5_output.ipynb in Jupyter Notebook to visualize or convert the dataset.

Refer to the Phase 1 README for details.

---

Phase 2: Segmentation Model Development

1. Train the model using synthetic data:

   python ./model_development/train.py --images_path <train_data> --val_images_path <val_data>

   Replace <train_data> and <val_data> with the paths to training and validation datasets.

2. Fine-tune the trained model:

   python ./model_development/finetuning.py --images_path <train_data>

Refer to the Phase 2 README for details.

---

Phase 3: Domain Adaptation

1. Prepare and organize datasets:

   python ./domain_adaptation/adapting_with_cyclegan.py

2. Train a CycleGAN model for translating synthetic images into real-style images:

   python ./pytorch-CycleGAN-and-pix2pix/train.py --dataroot ./domain_adaptation/data --name synthetic2real

3. Fine-tune the segmentation model using the domain-adapted dataset:

   python ./domain_adaptation/finetuning.py -i <images_dir> -m <masks_dir>

Refer to the Phase 3 README for details.

---

Outputs

1. Phase 1: Synthetic dataset in HDF5 and JPEG formats.
2. Phase 2: Trained segmentation model (deeplabv3_model.pth) and fine-tuned model (deeplabv3_model_tuned.pth).
3. Phase 3: Domain-adapted model (deeplabv3_model_adapted.pth).

Predicted results are saved in the specified output_dir during inference.

---

Notes and Recommendations

• Use a GPU for training and fine-tuning to improve performance.
• Phase-specific READMEs provide additional implementation details, insights, and challenges encountered.
• Further refinements could include experimenting with additional domain adaptation techniques or expanding the synthetic dataset's diversity.