A Generative AI Extension for the OHIF Viewer

This Repository contains the OHIF Viewer with an Generative AI extension, that enables the user to input a text and ganerates a CT scan of the chest. The OHIF Viewer is a zero-footprint medical image viewer provided by the Open Health Imaging Foundation (OHIF). It is a configurable and extensible progressive web application with out-of-the-box support for image archives which support DICOMweb. This extension requires an backend server to run the generative AI model (MedSyn) to convert text input into 3D CT scans.

---

Screenshot of Generative AI extension. Left: Findings and Impressions of original CT scan. Right: Enter prompt to generate CT and server Status, below already generated images.

Developing

Requirements

• Yarn 1.17.3+
• Node 18+
• Docker
• Yarn Workspaces should be enabled on your machine:
  • yarn config set workspaces-experimental true
• To make inference with model GPU with 40GB RAM required

Getting Started

Run Viewer

1. Clone this repository
   • git clone https://github.com/TomWartm/Viewers.git
2. Navigate to the cloned project's directory
3. yarn install to restore dependencies and link projects

Run Backend

1. Clone the backend repository (on a machine with large GPU RAM)
   • git clone https://github.com/TomWartm/MedsynBackend
2. Navigate to the cloned project's directory
3. Install required python packages conda env create --file environment.yml
4. Actiave environment conda activate medsyn-3-8
5. Navigate to src folder
6. Run flask server flask run

Add dummy Data

Add NIfTI files to the folder data/nifti (some are available on our google drive) and use the notebook in backend/nifti_to_orthan.ipynb to converti files into DICOM and upload to the Orthanc server.

Commands

These commands are available from the root directory.

Yarn Commands	Description
Develop
yarn orthanc:up	Starts Orthanc server in docker
yarn dev:orthanc	Runs the viewer with Orthnac backend

Project

The OHIF Medical Image Viewing Platform is maintained as a monorepo. This means that this repository, instead of containing a single project, contains many projects. If you explore our project structure, you'll see the following:

.
├── extensions               #
│   ├── _example             # Skeleton of example extension
│   ├── default              # basic set of useful functionalities (datasources, panels, etc)
│   ├── cornerstone       # image rendering and tools w/ Cornerstone3D
│   ├── cornerstone-dicom-sr # DICOM Structured Report rendering and export
│   ├── cornerstone-dicom-sr # DICOM Structured Report rendering and export
│   ├── cornerstone-dicom-seg # DICOM Segmentation rendering and export
│   ├── cornerstone-dicom-rt # DICOM RTSTRUCT rendering
│   ├── cornerstone-microscopy # Whole Slide Microscopy rendering
│   ├── dicom-pdf # PDF rendering
│   ├── dicom-video # DICOM RESTful Services
│   ├── measurement-tracking # Longitudinal measurement tracking
|   ├── text-input-extension # generative ML model promting
│   ├── tmtv # Total Metabolic Tumor Volume (TMTV) calculation
|

│
├── modes                    #
│   ├── _example             # Skeleton of example mode
│   ├── generative-ai       # generative ML model promting
│   ├── basic-dev-mode       # Basic development mode
│   ├── longitudinal         # Longitudinal mode (measurement tracking)
│   ├── tmtv       # Total Metabolic Tumor Volume (TMTV) calculation mode
│   └── microscopy          # Whole Slide Microscopy mode
│
├── platform                 #
│   ├── core                 # Business Logic
│   ├── i18n                 # Internationalization Support
│   ├── ui                   # React component library
│   ├── docs                 # Documentation
│   └── viewer               # Connects platform and extension projects
│
├── ...                      # misc. shared configuration
├── lerna.json               # MonoRepo (Lerna) settings
├── package.json             # Shared devDependencies and commands
└── README.md                # This file

License