preprocessing is a python library designed for the purpose of preprocessing MRI data at QTIM. It currently supports reorganization of dicom files, subsequent nifti conversion, and preprocessing for brain data. Its outputs are intended to follow the BIDS organizational scheme.
If you are working on a Martinos machine and want to use preprocessing directly, you will already have access to it through the corresponding pyenv environment (visit this guide for more information on using pyenv at Martinos). Simply run:
pyenv activate preprocessingand you will be able to use all of the functionality provided through the CLI.
If however, you are working on another machine or want to use the preprocessing library as part of a larger project, you should install it into a python virtual environment alongside some external software.
To install the preprocessing library in a python virtual environment, you may use either pip or poetry, like so:
pip install git+ssh://[email protected]/QTIM-Lab/preprocessing.gitpoetry add git+ssh://[email protected]/QTIM-Lab/preprocessing.gitAside from python dependencies, this repo also requires external software to run some of the commands. On Martinos Machines, everything will be sourced automatically and will require no additional work on your part. If you want to use preprocessing on your own machine, you will have to install ANTS (>= 2.3.5), Slicer (>= 5.2.2), fsl (>= 6.0.6), freesurfer (dev), and CUDA 11.8 and source them within your shell. For user-specific or otherwise non-system installations, it is recommended to add analogous lines to the following directly to your .bashrc, .zshrc, etc.:
export PATH=/usr/local/freesurfer/dev/bin:${PATH}
export PATH=/usr/pubsw/packages/fsl/6.0.6/bin:${PATH}
export PATH=/usr/pubsw/packages/slicer/Slicer-5.2.2-linux-amd64/:${PATH}
export PATH=/usr/pubsw/packages/ANTS/2.3.5/bin:${PATH}
export PATH=/usr/pubsw/packages/CUDA/11.8/bin:${PATH}
export LD_LIBRARY_PATH=/usr/pubsw/packages/CUDA/11.8/lib64:{LD_LIBRARY_PATH}
export FREESURFER_HOME=/usr/local/freesurfer/dev
source ${FREESURFER_HOME}/SetUpFreeSurfer.sh
export ANTSPATH=/usr/pubsw/packages/ANTS/2.3.5/bin
export FSLDIR=/usr/pubsw/packages/fsl/6.0.6/
source ${FSLDIR}/etc/fslconf/fsl.shWhile you can import this code directly into your own python scripts, the most convenient way to run common functions is through the CLI. Once installed to your project's environment using pip or poetry, you will be able to access the most important functionality through preprocessing's various subcommands. At any time, use preprocessing --help for an overview of the available commands and a brief description of each. For more in depth descriptions of each command, use preprocessing <command> --help.
This library utilizes csv files for the majority of its functions. Specific columns may be required for each to run properly. If you are using the CLI, use --help to view the columns required for a given command. Alternatively, reference the docstrings if you wish to use preprocessing directly within python. For an example of what formatting to expect, check here.
preprocessing old-project-anon-keys <input-dir> <output-dir>This command creates anonymization keys for anonymous PatientID and StudyID from the previous QTIM organizational scheme. This command is compatible with data following a following <Patient_ID>/<Study_ID> directory hierarchy.
preprocessing reorganize-dicoms <original-dicom-dir> <new-dicom-dir> \
[--anon-csv=None] [-c | --cpus=1]This command reorganizes DICOMs to follow the BIDS convention. Any DICOMs found recursively within this directory will be reorganized (at least one level of subdirectories is assumed). Anonomyzation keys for PatientIDs and StudyIDs are provided within a csv.
preprocessing dataset-to-nifti <nifti-dir> <csv> [--overwrite=False] \
[-c | --cpus=1] [-h | --help]This command converts DICOMs to NIfTI file format. A csv is required to map a DICOM series to the resulting .nii.gz file and to provide the context for filenames. The outputs will comply with the BIDS conventions.
preprocessing predict-series <csv> [--ruleset="brain"] \
[--description-key=default_key] [-c | --cpus=1] \
[-h | --help]This command predicts the sequence type for every series in your dataset. A csv is required to indicate the location of the corresponding DICOMs. Predictions are made using the mr-series-selection repo's analysis of the DICOM header. A json can be provided to combine multiple NormalizedDescriptions into a single category. Description keys should map a NormalizedDescription to a standard name and a SeriesType (BIDS modality) like so:
default_key = {
"T1Pre": [["iso3D AX T1 NonContrast", "iso3D AX T1 NonContrast RFMT"], "anat"],
"T1Post": [["iso3D AX T1 WithContrast", "iso3D AX T1 WithContrast RFMT"], "anat"],
}
preprocessing brain-preprocessing <preprocessed-dir> <csv> \
[--pipeline-key="preprocessed"] [--registration-key="T1Post"] \
[--longitudinal-registration=False] [-m | --model="affine"] \
[--orientation="RAS"] [--spacing="1,1,1"] [--no-skullstrip=False] \
[-c | --cpus=1] [-g | --gpu=False] [-v | --verbose=False] [-h | --help]This command preprocesses NIfTI files for deep learning. A csv is required to indicate the location of source files and to procide the context for filenames. The outputs will comply with BIDS conventions.
preprocessing deug-preprocessing <preprocessed-dir> <csv> \
[--patients=None] [--pipeline-key="debug"] \
[--registration-key="T1Post"] [--longitudinal-registration=False] \
[-m | --model="affine"] [--orientation="RAS"] [--spacing="1,1,1"] \
[--no-skullstrip=False] [-c | --cpus=1] [-g | --gpu=False] \
[-v | --verbose=False] [-h | --help]This command has the same functionality as the brain-preprocessing command, with the exception of saving intermediate results under different filenames. It is recommended to indicate the specific patients that want to be analyzed.