Skip to content

Commit

Permalink
DOCS shift to rst - Embedding Preprocessing Computation (openvinotool…
Browse files Browse the repository at this point in the history
  • Loading branch information
sgolebiewski-intel authored Mar 30, 2023
1 parent b3503c8 commit ab96cc9
Show file tree
Hide file tree
Showing 3 changed files with 881 additions and 473 deletions.
184 changes: 119 additions & 65 deletions docs/MO_DG/prepare_model/Additional_Optimizations.md
Original file line number Diff line number Diff line change
@@ -1,102 +1,156 @@
# Embedding Preprocessing Computation {#openvino_docs_MO_DG_Additional_Optimization_Use_Cases}

Input data for inference can be different from the training dataset and requires additional preprocessing before inference.
To accelerate the whole pipeline including preprocessing and inference, Model Optimizer provides special parameters such as `--mean_values`,
@sphinxdirective

`--scale_values`, `--reverse_input_channels`, and `--layout`. Based on these parameters, Model Optimizer generates OpenVINO IR with additionally
inserted sub-graphs to perform the defined preprocessing. This preprocessing block can perform mean-scale normalization of input data,
reverting data along channel dimension, and changing the data layout.
See the following sections for details on the parameters, or the [Overview of Preprocessing API](../../OV_Runtime_UG/preprocessing_overview.md) for the same functionality in OpenVINO Runtime.
Input data for inference can be different from the training dataset and requires
additional preprocessing before inference. To accelerate the whole pipeline including
preprocessing and inference, Model Optimizer provides special parameters such as ``--mean_values``,

## Specifying Layout
``--scale_values``, ``--reverse_input_channels``, and ``--layout``. Based on these
parameters, Model Optimizer generates OpenVINO IR with additionally inserted sub-graphs
to perform the defined preprocessing. This preprocessing block can perform mean-scale
normalization of input data, reverting data along channel dimension, and changing
the data layout. See the following sections for details on the parameters, or the
:doc:`Overview of Preprocessing API <openvino_docs_OV_UG_Preprocessing_Overview>`
for the same functionality in OpenVINO Runtime.

You may need to set input layouts, as it is required by some preprocessing, for example, setting a batch, applying mean or scales, and reversing input channels (BGR<->RGB).
Specifying Layout
#################

Layout defines the meaning of dimensions in shape and can be specified for both inputs and outputs. Some preprocessing requires to set input layouts, for example, setting a batch, applying mean or scales, and reversing input channels (BGR<->RGB).
You may need to set input layouts, as it is required by some preprocessing, for
example, setting a batch, applying mean or scales, and reversing input channels (BGR<->RGB).

For the layout syntax, check the [Layout API overview](../../OV_Runtime_UG/layout_overview.md).
To specify the layout, you can use the `--layout` option followed by the layout value.
Layout defines the meaning of dimensions in shape and can be specified for both
inputs and outputs. Some preprocessing requires to set input layouts, for example,
setting a batch, applying mean or scales, and reversing input channels (BGR<->RGB).

For example, the following command specifies the `NHWC` layout for a Tensorflow `nasnet_large` model that was exported to the ONNX format:
For the layout syntax, check the :doc:`Layout API overview <openvino_docs_OV_UG_Layout_Overview>`.
To specify the layout, you can use the ``--layout`` option followed by the layout value.

```
mo --input_model tf_nasnet_large.onnx --layout nhwc
```
For example, the following command specifies the ``NHWC`` layout for a Tensorflow
``nasnet_large`` model that was exported to the ONNX format:

Additionally, if a model has more than one input or needs both input and output layouts specified, you need to provide the name of each input or output to apply the layout.
.. code-block:: sh

For example, the following command specifies the layout for an ONNX `Yolo v3 Tiny` model with its first input `input_1` in `NCHW` layout and second input `image_shape` having two dimensions: batch and size of the image expressed as the `N?` layout:
mo --input_model tf_nasnet_large.onnx --layout nhwc

```
mo --input_model yolov3-tiny.onnx --layout input_1(nchw),image_shape(n?)
```

## Changing Model Layout
Additionally, if a model has more than one input or needs both input and output
layouts specified, you need to provide the name of each input or output to apply the layout.

For example, the following command specifies the layout for an ONNX ``Yolo v3 Tiny``
model with its first input ``input_1`` in ``NCHW`` layout and second input ``image_shape``
having two dimensions: batch and size of the image expressed as the ``N?`` layout:

.. code-block:: sh

mo --input_model yolov3-tiny.onnx --layout input_1(nchw),image_shape(n?)


Changing Model Layout
#####################

Changing the model layout may be necessary if it differs from the one presented by input data.
Use either `--layout` or `--source_layout` with `--target_layout` to change the layout.
Use either ``--layout`` or ``--source_layout`` with ``--target_layout`` to change the layout.

For example, for the same ``nasnet_large`` model mentioned previously, you can use
the following commands to provide data in the ``NCHW`` layout:

For example, for the same `nasnet_large` model mentioned previously, you can use the following commands to provide data in the `NCHW` layout:
.. code-block:: sh

```
mo --input_model tf_nasnet_large.onnx --source_layout nhwc --target_layout nchw
mo --input_model tf_nasnet_large.onnx --layout "nhwc->nchw"
```
mo --input_model tf_nasnet_large.onnx --source_layout nhwc --target_layout nchw
mo --input_model tf_nasnet_large.onnx --layout "nhwc->nchw"

Again, if a model has more than one input or needs both input and output layouts specified, you need to provide the name of each input or output to apply the layout.

For example, to provide data in the `NHWC` layout for the `Yolo v3 Tiny` model mentioned earlier, use the following commands:
Again, if a model has more than one input or needs both input and output layouts
specified, you need to provide the name of each input or output to apply the layout.

```
mo --input_model yolov3-tiny.onnx --source_layout "input_1(nchw),image_shape(n?)" --target_layout "input_1(nhwc)"
mo --input_model yolov3-tiny.onnx --layout "input_1(nchw->nhwc),image_shape(n?)"
```
For example, to provide data in the ``NHWC`` layout for the `Yolo v3 Tiny` model
mentioned earlier, use the following commands:

## Specifying Mean and Scale Values
Neural network models are usually trained with the normalized input data. This means that the input data values are converted to be in a specific range,
for example, `[0, 1]` or `[-1, 1]`. Sometimes, the mean values (mean images) are subtracted from the input data values as part of the preprocessing.
.. code-block:: sh

mo --input_model yolov3-tiny.onnx --source_layout "input_1(nchw),image_shape(n?)" --target_layout "input_1(nhwc)"
mo --input_model yolov3-tiny.onnx --layout "input_1(nchw->nhwc),image_shape(n?)"


Specifying Mean and Scale Values
################################

Neural network models are usually trained with the normalized input data. This
means that the input data values are converted to be in a specific range, for example,
``[0, 1]`` or ``[-1, 1]``. Sometimes, the mean values (mean images) are subtracted
from the input data values as part of the preprocessing.

There are two cases of how the input data preprocessing is implemented.
* The input preprocessing operations are a part of a model.

In this case, the application does not perform a separate preprocessing step: everything is embedded into the model itself. Model Optimizer will generate the OpenVINO IR format with required preprocessing operations, and no `mean` and `scale` parameters are required.
* The input preprocessing operations are not a part of a model and the preprocessing is performed within the application which feeds the model with input data.
* The input preprocessing operations are a part of a model.

In this case, the application does not perform a separate preprocessing step:
everything is embedded into the model itself. Model Optimizer will generate the
OpenVINO IR format with required preprocessing operations, and no ``mean`` and
``scale`` parameters are required.
* The input preprocessing operations are not a part of a model and the preprocessing
is performed within the application which feeds the model with input data.

In this case, information about mean/scale values should be provided to Model Optimizer to embed it to the generated OpenVINO IR format.
In this case, information about mean/scale values should be provided to Model
Optimizer to embed it to the generated OpenVINO IR format.

Model Optimizer provides command-line parameters to specify the values: `--mean_values`, `--scale_values`, `--scale`.
Using these parameters, Model Optimizer embeds the corresponding preprocessing block for mean-value normalization of the input data
Model Optimizer provides command-line parameters to specify the values: ``--mean_values``,
``--scale_values``, ``--scale``. Using these parameters, Model Optimizer embeds the
corresponding preprocessing block for mean-value normalization of the input data
and optimizes this block so that the preprocessing takes negligible time for inference.

For example, the following command runs Model Optimizer for the PaddlePaddle UNet model and applies mean-scale normalization to the input data:
For example, the following command runs Model Optimizer for the PaddlePaddle UNet
model and applies mean-scale normalization to the input data:

.. code-block:: sh

mo --input_model unet.pdmodel --mean_values [123,117,104] --scale 255


Reversing Input Channels
########################

Sometimes, input images for your application can be of the RGB (or BGR) format
and the model is trained on images of the BGR (or RGB) format, which is in the
opposite order of color channels. In this case, it is important to preprocess the
input images by reverting the color channels before inference.

To embed this preprocessing step into OpenVINO IR, Model Optimizer provides the
``--reverse_input_channels`` command-line parameter to shuffle the color channels.

The ``--reverse_input_channels`` parameter can be used to preprocess the model
input in the following cases:

* Only one dimension in the input shape has a size equal to ``3``.
* One dimension has an undefined size and is marked as ``C`` channel using ``layout`` parameters.

Using the ``--reverse_input_channels`` parameter, Model Optimizer embeds the corresponding
preprocessing block for reverting the input data along channel dimension and optimizes
this block so that the preprocessing takes only negligible time for inference.

For example, the following command launches Model Optimizer for the TensorFlow AlexNet
model and embeds the ``reverse_input_channel`` preprocessing block into OpenVINO IR:

```sh
mo --input_model unet.pdmodel --mean_values [123,117,104] --scale 255
```
.. code-block:: sh

## Reversing Input Channels <a name="when_to_reverse_input_channels"></a>
Sometimes, input images for your application can be of the RGB (or BGR) format and the model is trained on images of the BGR (or RGB) format,
which is in the opposite order of color channels. In this case, it is important to preprocess the input images by reverting the color channels before inference.
mo --input_model alexnet.pb --reverse_input_channels

To embed this preprocessing step into OpenVINO IR, Model Optimizer provides the `--reverse_input_channels` command-line parameter to shuffle the color channels.

The `--reverse_input_channels` parameter can be used to preprocess the model input in the following cases:
* Only one dimension in the input shape has a size equal to 3.
* One dimension has an undefined size and is marked as `C` channel using `layout` parameters.
.. note::

Using the `--reverse_input_channels` parameter, Model Optimizer embeds the corresponding preprocessing block for reverting
the input data along channel dimension and optimizes this block so that the preprocessing takes only negligible time for inference.
If both mean and scale values are specified, the mean is subtracted first and
then the scale is applied regardless of the order of options in the command-line.
Input values are *divided* by the scale value(s). If the ``--reverse_input_channels``
option is also used, ``reverse_input_channels`` will be applied first, then ``mean``
and after that ``scale``. The data flow in the model looks as follows:
``Parameter -> ReverseInputChannels -> Mean apply-> Scale apply -> the original body of the model``.

For example, the following command launches Model Optimizer for the TensorFlow AlexNet model and embeds the `reverse_input_channel` preprocessing block into OpenVINO IR:
Additional Resources
####################

```sh
mo --input_model alexnet.pb --reverse_input_channels
```
* :doc:`Overview of Preprocessing API <openvino_docs_OV_UG_Preprocessing_Overview>`

> **NOTE**: If both mean and scale values are specified, the mean is subtracted first and then the scale is applied regardless of the order of options
in the command-line. Input values are *divided* by the scale value(s). If the `--reverse_input_channels` option is also used, `reverse_input_channels`
will be applied first, then `mean` and after that `scale`. The data flow in the model looks as follows:
`Parameter -> ReverseInputChannels -> Mean apply-> Scale apply -> the original body of the model`.
@endsphinxdirective

## Additional Resources
* [Overview of Preprocessing API](../../OV_Runtime_UG/preprocessing_overview.md)
35 changes: 24 additions & 11 deletions docs/MO_DG/prepare_model/FP16_Compression.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,29 @@
# Compressing a Model to FP16 {#openvino_docs_MO_DG_FP16_Compression}

Model Optimizer by default converts all floating-point weights to `FP16` data type. The resulting IR is called
compressed `FP16` model. The resulting model will occupy about twice as less space in the file system,
but it may have some accuracy drop. For most models, the accuracy drop is negligible.
But in case if accuracy drop is significant user can disable compression explicitly.
@sphinxdirective

By default, models are compressed to `FP16`, but you can disable compression by specifying `--compress_to_fp16=False`:
```
mo --input_model INPUT_MODEL --compress_to_fp16=False
```
Model Optimizer by default converts all floating-point weights to ``FP16`` data type.
The resulting IR is called compressed ``FP16`` model. The resulting model will occupy
about twice as less space in the file system, but it may have some accuracy drop.
For most models, the accuracy drop is negligible. But in case if accuracy drop is
significant user can disable compression explicitly.

For details on how plugins handle compressed `FP16` models, see [Working with devices](../../OV_Runtime_UG/supported_plugins/Device_Plugins.md).
By default, models are compressed to ``FP16``, but you can disable compression by
specifying ``--compress_to_fp16=False``:

> **NOTE**: `FP16` compression is sometimes used as the initial step for `INT8` quantization.
> Refer to the [Post-training optimization](../../../tools/pot/docs/Introduction.md) guide for more information about that.
.. code-block:: sh

mo --input_model INPUT_MODEL --compress_to_fp16=False


For details on how plugins handle compressed ``FP16`` models, see
:doc:`Working with devices <openvino_docs_OV_UG_Working_with_devices>`.

.. note::

``FP16`` compression is sometimes used as the initial step for ``INT8`` quantization.
Refer to the :doc:`Post-training optimization <pot_introduction>` guide for more
information about that.


@endsphinxdirective
Loading

0 comments on commit ab96cc9

Please sign in to comment.