Merge pull request meituan#405 from czla/main

Support OpenCV depolyment with Python/C++

README.md

@@ -103,6 +103,7 @@ Your can also specify a checkpoint path to `--resume` parameter by
 ### Deployment
 
 *  [ONNX](./deploy/ONNX)
+*  [OpenCV Python/C++](./deploy/ONNX/OpenCV)
 *  [OpenVINO](./deploy/OpenVINO)
 *  [Partial Quantization](./tools/partial_quantization)
 

deploy/ONNX/OpenCV/README.md

@@ -0,0 +1,92 @@
+#  Object Detection using YOLOv5/YOLOv6/YOLOX and OpenCV DNN (Python/C++)
+
+## 0. Install Dependancies
+```
+OpenCV >= 4.5.4
+```
+Only **OpenCV >= 4.5.4** can read onnx model file by dnn module.
+
+## 1. Usage
+Change work directory to `/path/to/YOLOv6/deploy/ONNX/OpenCV`
+### 1.1 Python
+
+- YOLOv5&YOLOv6:
+```Python
+python yolo.py --model /path/to/onnx/yolov5n.onnx --img /path/to/sample.jpg --classesFile /path/to/coco.names
+                                     yolov5s.onnx
+                                     yolov5m.onnx
+                                     yolov6n.onnx
+                                     yolov6s.onnx
+                                     yolov6t.onnx
+```
+- YOLOX:
+```Python
+python yolox.py --model /path/to/onnx/yolox_nano.onnx --img /path/to/sample.jpg --classesFile /path/to/coco.names
+                                      yolox_tiny.onnx
+                                      yolox_s.onnx
+                                      yolox_m.onnx
+```
+
+### 1.2 CMake C++ Linux YOLOv5
+```C++ Linux
+cd yolov5    // modify CMakeLists.txt
+mkdir build
+cd build
+cmake ..
+make
+./yolov5 /path/to/onnx/yolov5n.onnx /path/to/sample.jpg /path/to/coco.names
+                       yolov5s.onnx
+                       yolov5m.onnx
+```
+
+### 1.3 CMake C++ Linux YOLOv6
+```C++ Linux
+cd yolov6    // modify CMakeLists.txt
+mkdir build
+cd build
+cmake ..
+make
+./yolov6 /path/to/onnx/yolov6n.onnx /path/to/sample.jpg /path/to/coco.names
+                       yolov6t.onnx
+                       yolov6s.onnx
+```
+
+### 1.4 CMake C++ Linux YOLOX
+```C++ Linux
+cd yolox    // modify CMakeLists.txt
+mkdir build
+cd build
+cmake ..
+make
+./yolox /path/to/onnx/yolox_nano.onnx /path/to/sample.jpg /path/to/coco.names
+                      yolox_tiny.onnx
+                      yolox_s.onnx
+                      yolox_m.onnx
+```
+
+## 2. Result
+| Model | Speed CPU b1(ms) Python | Speed CPU b1(ms) C++ | mAP<sup>val 0.5:0.95</sup> | params(M) | FLOPs(G) |
+| :-- | :-: | :-: | :-: | :-: | :-: |
+| **YOLOv5n** | 116.47 | 118.89 | 28.0 | 1.9 | 4.5 |
+| **YOLOv5s** | 200.53 | 202.22 | 37.4 | 7.2 | 16.5 |
+| **YOLOv5m** | 294.98 | 291.86 | 45.4 | 21.2 | 49.0 |
+|  |  |  |  |  |  |
+| **YOLOv6-n** | 66.88 | 69.96 | 35.0 | 4.3 | 4.7 |
+| **YOLOv6-tiny** | 133.15 | 137.59 | 41.3 | 15.0 | 36.7 |
+| **YOLOv6-s** | 164.44 | 163.38 | 43.1 | 17.2 | 44.2 |
+|  |  |  |  |  |  |
+| **YOLOX-Nano** | 81.06 | 86.75 | 25.8@416 | 0.91 | 1.08@416 |
+| **YOLOX-tiny** | 129.72 | 144.19 | 32.8@416 | 5.06 | 6.45@416 |
+| **YOLOX-s** | 180.86 | 169.96 | 40.5 | 9.0 | 26.8 |
+| **YOLOX-m** | 336.34 | 357.91 | 47.2 | 25.3 | 73.8 |
+
+**Note**:
+- All onnx models are converted from official github([Google Drive](https://drive.google.com/drive/folders/1Nw6M_Y6XLASyB0RxhSI2z_QRtt70Picl?usp=sharing)).
+- Speed is test by [dnn::Net::getPerfProfile](https://docs.opencv.org/4.5.5/db/d30/classcv_1_1dnn_1_1Net.html), we report the average inference time of 300 runs on the same environment.
+- The mAP/params/FLOPs are from official github.
+- Test environment: MacOS 11.4 with 2.6 GHz 6-core Intel Core i7, 16GB Memory.
+
+### Visualization
+<div align="left"> <img src="../../../assets/yolov5s.jpg" width="1000"></div>
+<div align="left"> <img src="../../../assets/yolov6s.jpg" width="1000"></div>
+<div align="left"> <img src="../../../assets/yoloxs.jpg" width="1000"></div>

deploy/ONNX/OpenCV/coco.names

@@ -0,0 +1,80 @@
+person
+bicycle
+car
+motorbike
+aeroplane
+bus
+train
+truck
+boat
+traffic light
+fire hydrant
+stop sign
+parking meter
+bench
+bird
+cat
+dog
+horse
+sheep
+cow
+elephant
+bear
+zebra
+giraffe
+backpack
+umbrella
+handbag
+tie
+suitcase
+frisbee
+skis
+snowboard
+sports ball
+kite
+baseball bat
+baseball glove
+skateboard
+surfboard
+tennis racket
+bottle
+wine glass
+cup
+fork
+knife
+spoon
+bowl
+banana
+apple
+sandwich
+orange
+broccoli
+carrot
+hot dog
+pizza
+donut
+cake
+chair
+sofa
+pottedplant
+bed
+diningtable
+toilet
+tvmonitor
+laptop
+mouse
+remote
+keyboard
+cell phone
+microwave
+oven
+toaster
+sink
+refrigerator
+book
+clock
+vase
+scissors
+teddy bear
+hair drier
+toothbrush

deploy/ONNX/OpenCV/yolo.py

@@ -0,0 +1,149 @@
+import cv2
+import numpy as np
+import os
+import argparse
+
+
+# Constants.
+INPUT_WIDTH = 640
+INPUT_HEIGHT = 640
+SCORE_THRESHOLD = 0.5			# cls score
+NMS_THRESHOLD = 0.45
+CONFIDENCE_THRESHOLD = 0.45		# obj confidence
+
+# Text parameters.
+FONT_FACE = cv2.FONT_HERSHEY_SIMPLEX
+FONT_SCALE = 0.7
+THICKNESS = 1
+
+# Colors
+BLACK  = (0,0,0)
+BLUE   = (255,178,50)
+YELLOW = (0,255,255)
+RED = (0,0,255)
+
+
+def draw_label(input_image, label, left, top):
+    """Draw text onto image at location."""
+
+    # Get text size.
+    text_size = cv2.getTextSize(label, FONT_FACE, FONT_SCALE, THICKNESS)
+    dim, baseline = text_size[0], text_size[1]
+    # Use text size to create a BLACK rectangle. 
+    cv2.rectangle(input_image, (left, top), (left + dim[0], top + dim[1] + baseline), BLACK, cv2.FILLED)
+    # Display text inside the rectangle.
+    cv2.putText(input_image, label, (left, top + dim[1]), FONT_FACE, FONT_SCALE, YELLOW, THICKNESS, cv2.LINE_AA)
+
+
+def pre_process(input_image, net):
+	# Create a 4D blob from a frame.
+	blob = cv2.dnn.blobFromImage(input_image, 1/255, (INPUT_WIDTH, INPUT_HEIGHT), [0,0,0], 1, crop=False)
+
+	# Sets the input to the network.
+	net.setInput(blob)
+
+	# Runs the forward pass to get output of the output layers.
+	output_layers = net.getUnconnectedOutLayersNames()
+	outputs = net.forward(output_layers)
+	# print(outputs[0].shape)
+
+	return outputs
+
+
+def post_process(input_image, outputs):
+	# Lists to hold respective values while unwrapping.
+	class_ids = []
+	confidences = []
+	boxes = []
+
+	# Rows.
+	rows = outputs[0].shape[1]
+
+	image_height, image_width = input_image.shape[:2]
+
+	# Resizing factor.
+	x_factor = image_width / INPUT_WIDTH
+	y_factor =  image_height / INPUT_HEIGHT
+
+	# Iterate through 25200 detections.
+	for r in range(rows):
+		row = outputs[0][0][r]
+		confidence = row[4]
+
+		# Discard bad detections and continue.
+		if confidence >= CONFIDENCE_THRESHOLD:
+			classes_scores = row[5:]
+
+			# Get the index of max class score.
+			class_id = np.argmax(classes_scores)
+
+			#  Continue if the class score is above threshold.
+			if (classes_scores[class_id] > SCORE_THRESHOLD):
+				confidences.append(confidence)
+				class_ids.append(class_id)
+
+				cx, cy, w, h = row[0], row[1], row[2], row[3]
+
+				left = int((cx - w/2) * x_factor)
+				top = int((cy - h/2) * y_factor)
+				width = int(w * x_factor)
+				height = int(h * y_factor)
+
+				box = np.array([left, top, width, height])
+				boxes.append(box)
+
+	# Perform non maximum suppression to eliminate redundant overlapping boxes with
+	# lower confidences.
+	indices = cv2.dnn.NMSBoxes(boxes, confidences, CONFIDENCE_THRESHOLD, NMS_THRESHOLD)
+	for i in indices:
+		box = boxes[i]
+		left = box[0]
+		top = box[1]
+		width = box[2]
+		height = box[3]
+		cv2.rectangle(input_image, (left, top), (left + width, top + height), BLUE, 3*THICKNESS)
+		label = "{}:{:.2f}".format(classes[class_ids[i]], confidences[i])
+		draw_label(input_image, label, left, top)
+
+	return input_image
+
+
+if __name__ == '__main__':
+	parser = argparse.ArgumentParser()
+	parser.add_argument('--model', default='models/yolov6n.onnx', help="Input your onnx model.")
+	parser.add_argument('--img', default='sample.jpg', help="Path to your input image.")
+	parser.add_argument('--classesFile', default='coco.names', help="Path to your classesFile.")
+	args = parser.parse_args()
+
+	# Load class names.
+	model_path, img_path, classesFile = args.model, args.img, args.classesFile
+	window_name = os.path.splitext(os.path.basename(model_path))[0]
+	classes = None
+	with open(classesFile, 'rt') as f:
+		classes = f.read().rstrip('\n').split('\n')
+
+	# Load image.
+	frame = cv2.imread(img_path)
+	input = frame.copy()
+
+	# Give the weight files to the model and load the network using them.
+	net = cv2.dnn.readNet(model_path)
+
+	# Put efficiency information. The function getPerfProfile returns the overall time for inference(t) and the 
+	# timings for each of the layers(in layersTimes)
+	# Process image.
+	cycles = 300
+	total_time = 0
+	for i in range(cycles):
+		detections = pre_process(input.copy(), net)
+		img = post_process(frame.copy(), detections)
+		t, _ = net.getPerfProfile()
+		total_time += t
+		print(f'Cycle [{i + 1}]:\t{t * 1000.0 / cv2.getTickFrequency():.2f}\tms')
+
+	avg_time = total_time / cycles
+	label = 'Average Inference time: %.2f ms' % (avg_time * 1000.0 / cv2.getTickFrequency())
+	print(f'Model: {window_name}\n{label}')
+	cv2.putText(img, label, (20, 40), FONT_FACE, FONT_SCALE, RED, THICKNESS, cv2.LINE_AA)
+	cv2.imshow(window_name, img)
+	cv2.waitKey(0)

deploy/ONNX/OpenCV/yolov5/CMakeLists.txt

@@ -0,0 +1,33 @@
+# CMakeLists.txt
+
+# Older versions of CMake are likely to work just fine but, since
+# I don't know where to cut off I just use the version I'm using
+cmake_minimum_required(VERSION "3.17")
+
+# name of this example project
+project(simple-demo)
+
+# set OpenCV_DIR variable equal to the path to the cmake
+# files within the previously installed opencv program
+# path like /xxx/yyy/opencv/install/lib/cmake/opencv4
+set(OpenCV_DIR ${OpenCV_DIR})
+
+# Tell compiler to use C++ 14 features which is needed because
+# Clang version is often behind in the XCode installation
+set(CMAKE_CXX_STANDARD 14)
+
+# configure the necessary common CMake environment variables
+# needed to include and link the OpenCV program into this
+# demo project, namely OpenCV_INCLUDE_DIRS and OpenCV_LIBS
+find_package( OpenCV REQUIRED )
+
+# tell the build to include the headers from OpenCV
+include_directories( ${OpenCV_INCLUDE_DIRS} )
+
+# specify the executable target to be built
+# path like /xxx/yyy/opencv/install/include
+add_executable(yolov5 yolov5.cpp)
+
+# tell it to link the executable target against OpenCV
+# path like /xxx/yyy/opencv/install/lib
+target_link_libraries(yolov5 ${OpenCV_LIBS} )