Post-Training Optimization Tool

Introduction

Post-training Optimization Tool (POT) is designed to accelerate the inference of deep learning models by applying special methods without model retraining or fine-tuning, for example, post-training 8-bit quantization. Therefore, the tool does not require a training dataset or a pipeline. To apply post-training algorithms from the POT, you need:

• A floating-point precision model, FP32 or FP16, converted into the OpenVINO™ Intermediate Representation (IR) format and run on CPU with the OpenVINO™.
• A representative calibration dataset representing a use case scenario, for example, 300 samples.

Figure below shows the optimization workflow:

To get started with POT tool refer to the corresponding OpenVINO™ documentation.

Installation

From PyPI

POT is distributed as a part of OpenVINO™ Development Tools package. For installation instruction please refer to this document.

From GitHub

As prerequisites, you should install OpenVINO™ Runtime and other dependencies such as Model Optimizer and Accuracy Checker.

To install POT from source:

• Clone OpenVINO repository

  git clone --recusive https://github.com/openvinotoolkit/openvino.git

• Navigate to openvino/tools/pot/ folder
• Install POT package:

  python3 setup.py install

After installation POT is available as a Python library under openvino.tools.pot.* and in the command line by the pot alias. To verify it, run pot -h.

Examples

OpenVINO provides several examples to demonstrate the POT optimization workflow:

• Command-line example:
  • Quantization of Image Classification model
• API tutorials:
  • Quantization of Image Classification model
  • Quantization of Object Detection model from Model Zoo
  • Quantization of Segmentation model for medical data
  • Quantization of BERT for Text Classification
• API examples:
  • Quantization of 3D segmentation model
  • Quantization of Face Detection model
  • Quantization of Object Detection model with controable accuracy
  • Quantizatin of speech model for GNA device

See Also

• Performance Benchmarks
• INT8 Quantization by Using Web-Based Interface of the DL Workbench