forked from openvinotoolkit/openvino
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
DOCS shift to rst - Embedding Preprocessing Computation (openvinotool…
- Loading branch information
1 parent
b3503c8
commit ab96cc9
Showing
3 changed files
with
881 additions
and
473 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,102 +1,156 @@ | ||
# Embedding Preprocessing Computation {#openvino_docs_MO_DG_Additional_Optimization_Use_Cases} | ||
|
||
Input data for inference can be different from the training dataset and requires additional preprocessing before inference. | ||
To accelerate the whole pipeline including preprocessing and inference, Model Optimizer provides special parameters such as `--mean_values`, | ||
@sphinxdirective | ||
|
||
`--scale_values`, `--reverse_input_channels`, and `--layout`. Based on these parameters, Model Optimizer generates OpenVINO IR with additionally | ||
inserted sub-graphs to perform the defined preprocessing. This preprocessing block can perform mean-scale normalization of input data, | ||
reverting data along channel dimension, and changing the data layout. | ||
See the following sections for details on the parameters, or the [Overview of Preprocessing API](../../OV_Runtime_UG/preprocessing_overview.md) for the same functionality in OpenVINO Runtime. | ||
Input data for inference can be different from the training dataset and requires | ||
additional preprocessing before inference. To accelerate the whole pipeline including | ||
preprocessing and inference, Model Optimizer provides special parameters such as ``--mean_values``, | ||
|
||
## Specifying Layout | ||
``--scale_values``, ``--reverse_input_channels``, and ``--layout``. Based on these | ||
parameters, Model Optimizer generates OpenVINO IR with additionally inserted sub-graphs | ||
to perform the defined preprocessing. This preprocessing block can perform mean-scale | ||
normalization of input data, reverting data along channel dimension, and changing | ||
the data layout. See the following sections for details on the parameters, or the | ||
:doc:`Overview of Preprocessing API <openvino_docs_OV_UG_Preprocessing_Overview>` | ||
for the same functionality in OpenVINO Runtime. | ||
|
||
You may need to set input layouts, as it is required by some preprocessing, for example, setting a batch, applying mean or scales, and reversing input channels (BGR<->RGB). | ||
Specifying Layout | ||
################# | ||
|
||
Layout defines the meaning of dimensions in shape and can be specified for both inputs and outputs. Some preprocessing requires to set input layouts, for example, setting a batch, applying mean or scales, and reversing input channels (BGR<->RGB). | ||
You may need to set input layouts, as it is required by some preprocessing, for | ||
example, setting a batch, applying mean or scales, and reversing input channels (BGR<->RGB). | ||
|
||
For the layout syntax, check the [Layout API overview](../../OV_Runtime_UG/layout_overview.md). | ||
To specify the layout, you can use the `--layout` option followed by the layout value. | ||
Layout defines the meaning of dimensions in shape and can be specified for both | ||
inputs and outputs. Some preprocessing requires to set input layouts, for example, | ||
setting a batch, applying mean or scales, and reversing input channels (BGR<->RGB). | ||
|
||
For example, the following command specifies the `NHWC` layout for a Tensorflow `nasnet_large` model that was exported to the ONNX format: | ||
For the layout syntax, check the :doc:`Layout API overview <openvino_docs_OV_UG_Layout_Overview>`. | ||
To specify the layout, you can use the ``--layout`` option followed by the layout value. | ||
|
||
``` | ||
mo --input_model tf_nasnet_large.onnx --layout nhwc | ||
``` | ||
For example, the following command specifies the ``NHWC`` layout for a Tensorflow | ||
``nasnet_large`` model that was exported to the ONNX format: | ||
|
||
Additionally, if a model has more than one input or needs both input and output layouts specified, you need to provide the name of each input or output to apply the layout. | ||
.. code-block:: sh | ||
|
||
For example, the following command specifies the layout for an ONNX `Yolo v3 Tiny` model with its first input `input_1` in `NCHW` layout and second input `image_shape` having two dimensions: batch and size of the image expressed as the `N?` layout: | ||
mo --input_model tf_nasnet_large.onnx --layout nhwc | ||
|
||
``` | ||
mo --input_model yolov3-tiny.onnx --layout input_1(nchw),image_shape(n?) | ||
``` | ||
|
||
## Changing Model Layout | ||
Additionally, if a model has more than one input or needs both input and output | ||
layouts specified, you need to provide the name of each input or output to apply the layout. | ||
|
||
For example, the following command specifies the layout for an ONNX ``Yolo v3 Tiny`` | ||
model with its first input ``input_1`` in ``NCHW`` layout and second input ``image_shape`` | ||
having two dimensions: batch and size of the image expressed as the ``N?`` layout: | ||
|
||
.. code-block:: sh | ||
|
||
mo --input_model yolov3-tiny.onnx --layout input_1(nchw),image_shape(n?) | ||
|
||
|
||
Changing Model Layout | ||
##################### | ||
|
||
Changing the model layout may be necessary if it differs from the one presented by input data. | ||
Use either `--layout` or `--source_layout` with `--target_layout` to change the layout. | ||
Use either ``--layout`` or ``--source_layout`` with ``--target_layout`` to change the layout. | ||
|
||
For example, for the same ``nasnet_large`` model mentioned previously, you can use | ||
the following commands to provide data in the ``NCHW`` layout: | ||
|
||
For example, for the same `nasnet_large` model mentioned previously, you can use the following commands to provide data in the `NCHW` layout: | ||
.. code-block:: sh | ||
|
||
``` | ||
mo --input_model tf_nasnet_large.onnx --source_layout nhwc --target_layout nchw | ||
mo --input_model tf_nasnet_large.onnx --layout "nhwc->nchw" | ||
``` | ||
mo --input_model tf_nasnet_large.onnx --source_layout nhwc --target_layout nchw | ||
mo --input_model tf_nasnet_large.onnx --layout "nhwc->nchw" | ||
|
||
Again, if a model has more than one input or needs both input and output layouts specified, you need to provide the name of each input or output to apply the layout. | ||
|
||
For example, to provide data in the `NHWC` layout for the `Yolo v3 Tiny` model mentioned earlier, use the following commands: | ||
Again, if a model has more than one input or needs both input and output layouts | ||
specified, you need to provide the name of each input or output to apply the layout. | ||
|
||
``` | ||
mo --input_model yolov3-tiny.onnx --source_layout "input_1(nchw),image_shape(n?)" --target_layout "input_1(nhwc)" | ||
mo --input_model yolov3-tiny.onnx --layout "input_1(nchw->nhwc),image_shape(n?)" | ||
``` | ||
For example, to provide data in the ``NHWC`` layout for the `Yolo v3 Tiny` model | ||
mentioned earlier, use the following commands: | ||
|
||
## Specifying Mean and Scale Values | ||
Neural network models are usually trained with the normalized input data. This means that the input data values are converted to be in a specific range, | ||
for example, `[0, 1]` or `[-1, 1]`. Sometimes, the mean values (mean images) are subtracted from the input data values as part of the preprocessing. | ||
.. code-block:: sh | ||
|
||
mo --input_model yolov3-tiny.onnx --source_layout "input_1(nchw),image_shape(n?)" --target_layout "input_1(nhwc)" | ||
mo --input_model yolov3-tiny.onnx --layout "input_1(nchw->nhwc),image_shape(n?)" | ||
|
||
|
||
Specifying Mean and Scale Values | ||
################################ | ||
|
||
Neural network models are usually trained with the normalized input data. This | ||
means that the input data values are converted to be in a specific range, for example, | ||
``[0, 1]`` or ``[-1, 1]``. Sometimes, the mean values (mean images) are subtracted | ||
from the input data values as part of the preprocessing. | ||
|
||
There are two cases of how the input data preprocessing is implemented. | ||
* The input preprocessing operations are a part of a model. | ||
|
||
In this case, the application does not perform a separate preprocessing step: everything is embedded into the model itself. Model Optimizer will generate the OpenVINO IR format with required preprocessing operations, and no `mean` and `scale` parameters are required. | ||
* The input preprocessing operations are not a part of a model and the preprocessing is performed within the application which feeds the model with input data. | ||
* The input preprocessing operations are a part of a model. | ||
|
||
In this case, the application does not perform a separate preprocessing step: | ||
everything is embedded into the model itself. Model Optimizer will generate the | ||
OpenVINO IR format with required preprocessing operations, and no ``mean`` and | ||
``scale`` parameters are required. | ||
* The input preprocessing operations are not a part of a model and the preprocessing | ||
is performed within the application which feeds the model with input data. | ||
|
||
In this case, information about mean/scale values should be provided to Model Optimizer to embed it to the generated OpenVINO IR format. | ||
In this case, information about mean/scale values should be provided to Model | ||
Optimizer to embed it to the generated OpenVINO IR format. | ||
|
||
Model Optimizer provides command-line parameters to specify the values: `--mean_values`, `--scale_values`, `--scale`. | ||
Using these parameters, Model Optimizer embeds the corresponding preprocessing block for mean-value normalization of the input data | ||
Model Optimizer provides command-line parameters to specify the values: ``--mean_values``, | ||
``--scale_values``, ``--scale``. Using these parameters, Model Optimizer embeds the | ||
corresponding preprocessing block for mean-value normalization of the input data | ||
and optimizes this block so that the preprocessing takes negligible time for inference. | ||
|
||
For example, the following command runs Model Optimizer for the PaddlePaddle UNet model and applies mean-scale normalization to the input data: | ||
For example, the following command runs Model Optimizer for the PaddlePaddle UNet | ||
model and applies mean-scale normalization to the input data: | ||
|
||
.. code-block:: sh | ||
|
||
mo --input_model unet.pdmodel --mean_values [123,117,104] --scale 255 | ||
|
||
|
||
Reversing Input Channels | ||
######################## | ||
|
||
Sometimes, input images for your application can be of the RGB (or BGR) format | ||
and the model is trained on images of the BGR (or RGB) format, which is in the | ||
opposite order of color channels. In this case, it is important to preprocess the | ||
input images by reverting the color channels before inference. | ||
|
||
To embed this preprocessing step into OpenVINO IR, Model Optimizer provides the | ||
``--reverse_input_channels`` command-line parameter to shuffle the color channels. | ||
|
||
The ``--reverse_input_channels`` parameter can be used to preprocess the model | ||
input in the following cases: | ||
|
||
* Only one dimension in the input shape has a size equal to ``3``. | ||
* One dimension has an undefined size and is marked as ``C`` channel using ``layout`` parameters. | ||
|
||
Using the ``--reverse_input_channels`` parameter, Model Optimizer embeds the corresponding | ||
preprocessing block for reverting the input data along channel dimension and optimizes | ||
this block so that the preprocessing takes only negligible time for inference. | ||
|
||
For example, the following command launches Model Optimizer for the TensorFlow AlexNet | ||
model and embeds the ``reverse_input_channel`` preprocessing block into OpenVINO IR: | ||
|
||
```sh | ||
mo --input_model unet.pdmodel --mean_values [123,117,104] --scale 255 | ||
``` | ||
.. code-block:: sh | ||
|
||
## Reversing Input Channels <a name="when_to_reverse_input_channels"></a> | ||
Sometimes, input images for your application can be of the RGB (or BGR) format and the model is trained on images of the BGR (or RGB) format, | ||
which is in the opposite order of color channels. In this case, it is important to preprocess the input images by reverting the color channels before inference. | ||
mo --input_model alexnet.pb --reverse_input_channels | ||
|
||
To embed this preprocessing step into OpenVINO IR, Model Optimizer provides the `--reverse_input_channels` command-line parameter to shuffle the color channels. | ||
|
||
The `--reverse_input_channels` parameter can be used to preprocess the model input in the following cases: | ||
* Only one dimension in the input shape has a size equal to 3. | ||
* One dimension has an undefined size and is marked as `C` channel using `layout` parameters. | ||
.. note:: | ||
|
||
Using the `--reverse_input_channels` parameter, Model Optimizer embeds the corresponding preprocessing block for reverting | ||
the input data along channel dimension and optimizes this block so that the preprocessing takes only negligible time for inference. | ||
If both mean and scale values are specified, the mean is subtracted first and | ||
then the scale is applied regardless of the order of options in the command-line. | ||
Input values are *divided* by the scale value(s). If the ``--reverse_input_channels`` | ||
option is also used, ``reverse_input_channels`` will be applied first, then ``mean`` | ||
and after that ``scale``. The data flow in the model looks as follows: | ||
``Parameter -> ReverseInputChannels -> Mean apply-> Scale apply -> the original body of the model``. | ||
|
||
For example, the following command launches Model Optimizer for the TensorFlow AlexNet model and embeds the `reverse_input_channel` preprocessing block into OpenVINO IR: | ||
Additional Resources | ||
#################### | ||
|
||
```sh | ||
mo --input_model alexnet.pb --reverse_input_channels | ||
``` | ||
* :doc:`Overview of Preprocessing API <openvino_docs_OV_UG_Preprocessing_Overview>` | ||
|
||
> **NOTE**: If both mean and scale values are specified, the mean is subtracted first and then the scale is applied regardless of the order of options | ||
in the command-line. Input values are *divided* by the scale value(s). If the `--reverse_input_channels` option is also used, `reverse_input_channels` | ||
will be applied first, then `mean` and after that `scale`. The data flow in the model looks as follows: | ||
`Parameter -> ReverseInputChannels -> Mean apply-> Scale apply -> the original body of the model`. | ||
@endsphinxdirective | ||
|
||
## Additional Resources | ||
* [Overview of Preprocessing API](../../OV_Runtime_UG/preprocessing_overview.md) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,16 +1,29 @@ | ||
# Compressing a Model to FP16 {#openvino_docs_MO_DG_FP16_Compression} | ||
|
||
Model Optimizer by default converts all floating-point weights to `FP16` data type. The resulting IR is called | ||
compressed `FP16` model. The resulting model will occupy about twice as less space in the file system, | ||
but it may have some accuracy drop. For most models, the accuracy drop is negligible. | ||
But in case if accuracy drop is significant user can disable compression explicitly. | ||
@sphinxdirective | ||
|
||
By default, models are compressed to `FP16`, but you can disable compression by specifying `--compress_to_fp16=False`: | ||
``` | ||
mo --input_model INPUT_MODEL --compress_to_fp16=False | ||
``` | ||
Model Optimizer by default converts all floating-point weights to ``FP16`` data type. | ||
The resulting IR is called compressed ``FP16`` model. The resulting model will occupy | ||
about twice as less space in the file system, but it may have some accuracy drop. | ||
For most models, the accuracy drop is negligible. But in case if accuracy drop is | ||
significant user can disable compression explicitly. | ||
|
||
For details on how plugins handle compressed `FP16` models, see [Working with devices](../../OV_Runtime_UG/supported_plugins/Device_Plugins.md). | ||
By default, models are compressed to ``FP16``, but you can disable compression by | ||
specifying ``--compress_to_fp16=False``: | ||
|
||
> **NOTE**: `FP16` compression is sometimes used as the initial step for `INT8` quantization. | ||
> Refer to the [Post-training optimization](../../../tools/pot/docs/Introduction.md) guide for more information about that. | ||
.. code-block:: sh | ||
|
||
mo --input_model INPUT_MODEL --compress_to_fp16=False | ||
|
||
|
||
For details on how plugins handle compressed ``FP16`` models, see | ||
:doc:`Working with devices <openvino_docs_OV_UG_Working_with_devices>`. | ||
|
||
.. note:: | ||
|
||
``FP16`` compression is sometimes used as the initial step for ``INT8`` quantization. | ||
Refer to the :doc:`Post-training optimization <pot_introduction>` guide for more | ||
information about that. | ||
|
||
|
||
@endsphinxdirective |
Oops, something went wrong.