Working Code: Yolo detection , Kalman tracking

Author: AvnishGupta143

.gitignore

@@ -0,0 +1,4 @@
+.vscode/
+build/
+videos/project_track_and_detect.avi
+model/yolov3.weights

CMakeLists.txt

@@ -0,0 +1,31 @@
+cmake_minimum_required(VERSION 2.6)
+set(CMAKE_CXX_STANDARD 14)
+project(ObjectDetectionAndTracking)
+
+set(CXX_FLAGS "-Wall")
+set(CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS} "-std=c++14 -pthread -g")
+
+include_directories(include 
+                    ${PROJECT_SOURCE_DIR}/include 
+                    ${PROJECT_SOURCE_DIR}/include/Detector 
+                    ${PROJECT_SOURCE_DIR}/include/Tracker
+                    )
+
+find_package(OpenCV 4.1 REQUIRED)
+find_package (Eigen3 REQUIRED)
+include_directories(${OpenCV_INCLUDE_DIRS} ${EIGEN3_INCLUDE_DIRS})
+link_directories(${OpenCV_LIBRARY_DIRS})
+add_definitions(${OpenCV_DEFINITIONS})
+
+add_executable(detect_and_track src/main.cpp 
+                                src/frame_grabber.cpp 
+                                src/yolo_object_detector.cpp 
+                                src/frame_writer.cpp
+                                src/hungarian.cpp
+                                src/kalman_filter.cpp
+                                src/kalman_track.cpp
+                                src/tracker.cpp
+                                src/object_tracker.cpp
+                                )
+
+target_link_libraries(detect_and_track ${OpenCV_LIBRARIES} ${EIGEN3_DIR})

README.md

@@ -1 +1,168 @@
-# MultiThreaded-Object-Detection-and-Tracking-cpp
+# MultiThreaded Object Detection and Tracking
+
+## Description
+
+This repository contains an implementation of a multithreaded application for detecting objects and tracking objects in a user-specified video.  Example output of running the application on the input video (`videos/input_video.mp4`) is the resulting video (`videos/project_track_and_detect.avi`).
+
+The whole detection and tracking application runs on four asynchronous tasks. First task start reading frames using Frame Grabber object (`frame_grabber.h`) and pushes them them into a message queue (`message_queue.h`) which is thread safe implementation using conditional variables. The second tasks detects the frames by pulling the frames from message queue as they become available using Object Detector (`object_detector.h`). The class (`object_detector.h`) is a abtract class. After detection of each frame, the detector puts detected frame along with the required input data for tracking into another queue having a special message is type of tracking messages (`tracking_msg.h`). This two tasks run in parallel.
+Once they are completed another task is started, which pulls the tracking messages from another queue and does the tracking using Object Tracker (`object_tracker.h`). The Object Tracker uses the Tracker (`tracker.h`) which provides functionality to update the tracks. The tracks are implements as abstract class (`track.h`). Each frame after tracking is put in the output queue.
+After the completion of tracking task, the final task is started using Frame Writer object (`frame_writer.h`) which pulls the messages from the output queue and writes them to the user-specified output video file.
+
+### Red dot on the objects shows the tracking and the green dot shows the position measured by detecting the object and calculating center of bounding box
+
+<img src="data/output.gif"/>
+
+## Detectors:
+
+This project provides the abstact implementation of object detector, hence any custom object detector can be added. Currently, YOLO3-Object-Detector is implemented(`yolo_object_detector.h`).
+
+### YOLO3-Object-Detector
+
+This object detector is trained on coco dataset and can recognize upto 80 different classes of objects in the moving frame.
+
+## Trackers: Kalman
+
+The project provides abstract implementation of the tracker in form of tracks, hence any custom object tracker can be added. Currently, kalman tracker (`kalman_track.h`) is implemented using Kalman-Filter (`kalman_filter.h`). An assignment problem is used to associate the objects detected by detectors and tracked by trackers. Here it is solved using Hungarian Algorithm.
+
+Tracking consist of two major steps:
+1. Prediction: Predict the object locations in the next frame.
+2. Data association: Use the predicted locations to associate detections across frames to form tracks.
+
+## Dependencies for Running Locally
+* cmake >= 3.7
+  * All OSes: [click here for installation instructions](https://cmake.org/install/)
+* make >= 4.1 (Linux, Mac), 3.81 (Windows)
+  * Linux: make is installed by default on most Linux distros
+  * Mac: [install Xcode command line tools to get make](https://developer.apple.com/xcode/features/)
+  * Windows: [Click here for installation instructions](http://gnuwin32.sourceforge.net/packages/make.htm)
+* gcc/g++ >= 5.4
+  * Linux: gcc / g++ is installed by default on most Linux distros
+  * Mac: same deal as make - [install Xcode command line tools](https://developer.apple.com/xcode/features/)
+  * Windows: recommend using [MinGW](http://www.mingw.org/)
+* OpenCV >= 4.1
+  * The OpenCV 4.1.0 source code can be found [here](https://github.com/opencv/opencv/tree/4.1.0)
+* Eigen3
+  * Install using "sudo apt-get install libeigen3"
+
+## Basic Build Instructions
+
+1. Clone this repo.
+
+2. Run the following commands to download object detection models from [Darknet](https://pjreddie.com/darknet/)
+
+```
+cd CppND-Capstone
+mkdir model && cd model
+wget https://pjreddie.com/media/files/yolov3.weights
+wget https://github.com/pjreddie/darknet/blob/master/cfg/yolov3.cfg?raw=true -O ./yolov3.cfg
+```
+
+3. Make a build directory in the top level directory: `mkdir build && cd build`
+
+4. Compile: `cmake .. && make`
+
+5. Run it: `./detect_and_track` (**Runtime can vary depending on the number of frames in the video**)
+
+6. Progress can tracked while the program is running
+```
+Model Loaded Successfully!
+Loaded 80 Class Names
+Initialised the tracker
+Read 10 frames from total 1260 frames
+Read 20 frames from total 1260 frames
+Read 30 frames from total 1260 frames
+Read 40 frames from total 1260 frames
+Read 50 frames from total 1260 frames
+....
+....
+Detected 10 frames
+Read 520 frames from total 1260 frames
+Detected 20 frames
+Read 530 frames from total 1260 frames
+Detected 30 frames
+Read 540 frames from total 1260 frames
+....
+....
+Tracked 1240 frames
+Tracked 1250 frames
+Written 1253 frames
+-------  Done !!  -------
+
+7. The video can be played from (`videos/project_track_and_detect.avi`) after the program terminates
+
+## Code Instructions
+
+1. Change the model_config file path and model_weight_path in main.cpp
+2. Change the input and output video file path in main.cpp
+2. Turn the track flag to false in main.cpp if you want to run detection only
+
+## Satisfied Rubric Points
+
+### The application reads data from a file and process the data.
+
+* The project reads frames from a video file using the `ObjectDetector` class (`line 46 include/Detector/object_detector.h`).
+
+### The application implements the abstract classes and pure virtual functions.
+
+* The object detector class (`include/Detector/object_detector.h`) and the tracks class (`include/Tracker/track.h`).
+
+### Use of Object Oriented Programming techniques.
+
+* The classes `FrameGrabber` (`src/frame_grabber.cpp` , `include/frame_grabber.h`), `FrameWriter` (`src/frame_writer.cpp` , `include/frame_writer.h`), `MessageQueue` (`include/message_queue.h`), `YOLODetector` (`include/Detector/yolo_object_detector.h`, `src/yolo_object_detector.cpp`), `KalmanFilter` (`include/Tracker/kalman_filter.h` , `src/kalman_filter.cpp`), `Tracker` (`include/Tracker/tracker.h` , `src/tracker.cpp`) and `KalmanTrack` (`include/Tracker/kalman_track.h` , `src/kalman_track.cpp`).
+
+### Use of Inheritence techniques.
+
+* The class `YOLODetector` is inherited from parent class `ObjectDetector` (`line 15 include/Detector/yolo_object_detector.h`).
+* The class `KalmanTrack` is inherited from parent class `Track` (`line 8 include/Tracker/yolo_object_detector.h`).
+
+### Classes use appropriate access specifiers for class members.
+
+* Example of class-specific access specifiers in `Tracker` class definition (`include/Tracker/tracker.h`).
+
+### Use of Overloaded Functions.
+
+* The function getTracks() is overloaded in `Tracker` class (`line 47,49 include/Tracker/tracker.h`).
+
+### Templates generalize functions.
+
+* The `MessageQueue` is a templated class (`include/message_queue.h`).
+
+### Use of references in function declarations.
+
+* Example of method that uses reference in function declaration is the implementation of the `YOLODetector` constructor (`line 4 src/yolo_object_detector.cpp`).
+
+### Use of scope / Resource Acquisition Is Initialization (RAII) where appropriate.
+
+* Example use of RAII can be seen acquisition and release of locks in `MessageQueue` (`lines 19,55,62 include/message_queue.h`).
+
+### Use of move semantics to move data, instead of copying it, where possible.
+
+* Example use of move semantics is the pushing and removing of items  in `MessageQueue` (`line 41 include/message_queue.h`).
+
+### Use of smart pointers.
+
+* Example use of the smart pointers (std::unique_ptr) is (`line 7 src/yolo_object_detector.cpp`), (`line 15 src/tracker.cpp`).
+
+### Use of multithreading.
+
+* The project uses two asynchronous tasks (std::async) (`lines 54,57,63,68 src/main.cpp`).
+
+### Use of condition variable.
+
+* A condition variable is used in the project in the implementation of message queue (`lines 38,28 include/message_queue.h`).
+
+## References
+
+* The video used in this repository was taken from the repository [udacity/CarND-Vehicle-Detection](udacity/CarND-Vehicle-Detection).
+
+* OpenCV YOLO Object Detection (https://github.com/opencv/opencv/blob/master/samples/dnn/object_detection.cpp)
+
+* Kalman Filter - [Artificial Intelligence for Robotics](https://www.udacity.com/course/artificial-intelligence-for-robotics--cs373#) Udacity Course
+
+* Hungarian Algorithm - [here](http://www.mathworks.com/matlabcentral/fileexchange/6543-functions-for-the-rectangular-assignment-problem)
+
+* Motion-Based Multiple Object Tracking - [here](https://in.mathworks.com/help/vision/examples/motion-based-multiple-object-tracking.html)
+
+* Multiple Object Tracking - [here](https://in.mathworks.com/help/vision/ug/multiple-object-tracking.html)
+
+* Computer Vision for tracking - [here](https://towardsdatascience.com/computer-vision-for-tracking-8220759eee85)

content

@@ -0,0 +1,43 @@
+# Description
+
+	Obstacle detection or tracking moving objects is one of the most interesting topics in computer vision. 
+	This problem could be solved in two steps:
+  
+	1, Detecting moving objects in each frame
+	2, Tracking historical objects with some tracking algorithms
+  
+	An assignment problem is used to associate the objects detected by detectors and tracked by trackers.
+  
+	We can found some introduction of this framework here,
+	https://towardsdatascience.com/computer-vision-for-tracking-8220759eee85
+  
+	Another example in more detail with matlab code (detecors and trackers may different),
+	https://www.mathworks.com/help/vision/examples/motion-based-multiple-object-tracking.html
+
+	Here I implemented a highly efficient and scalable C++ framework to combine the state of art 
+	deep-learning based detectors (Yolo3 demoed here) and correlation filters based trackers 
+	(KCF, Kalman Filters also implemented). The assignment problem is solved by hungarian algorithm.
+  
+# Detectors: Yolo3 
+
+	Yolo3 is trained for detecting bottles, cans and hands in this demo. It is trained with Keras 
+	and compiled with tensorflow C++ into a DLL. (YOLO3.DLL under bin folder). CUDA 9.2 is used to
+	compile the tensorflow C++ library. 
+
+	YOLO3.DLL can be compiled with the code under folder detectors and tensorflow C++ library if 
+	you have tensorflow C++ library compiled.
+
+# Trackers: Kalman Filter and KCF
+
+	Kalman filter is fast but less accurate. KCF is accurate but much slower. 
+	They are implemnted with exactly same interface, so we can easily switch from one to another 
+	in the project.
+
+# Live Camera Capture: OpenCV
+
+	OpenCV is used to capture live video frames and used for image preprocessing.
+
+# Misc
+
+	YOLO3.DLL and the model file are too big. They can be downloaded from following link:
+	https://pan.baidu.com/s/1CPYU2o59vutoq-OJewObRw

data/output.gif

@@ -0,0 +1,95 @@
+#ifndef OBJECT_DETECTOR_H
+#define OBJECT_DETECTOR_H
+
+#include <fstream>
+#include <string>
+#include <iostream>
+#include <opencv2/opencv.hpp>
+#include <opencv2/highgui.hpp>
+#include <opencv2/dnn.hpp>
+#include "message_queue.h"
+#include "point2D.h"
+#include "tracking_msg.h"
+
+// Abstract class which provides functionality to detect object in the frames and have some functions to write the video file
+class ObjectDetector
+{
+  public:
+
+    // Constructor / Destructor
+    ObjectDetector(){};
+    ~ObjectDetector(){};
+  
+    // Method to detect object in the given frame
+    virtual void detectObject(cv::Mat &frame, std::vector<cv::Mat> &info) = 0;
+
+    // Method to detect object in the queue having frames
+    virtual void detectObjectInQueue(MessageQueue<cv::Mat> &msgq, MessageQueue<cv::Mat> &outq) = 0;
+
+    // Method to detect object in the msgq having frames and put the detected frame and detected points in trackq
+    virtual void detectObjectInQueueAndTrack(MessageQueue<cv::Mat> &msgq, MessageQueue<TrackingMsg> &trackq) = 0;
+
+    // Method to find the bounding boxes over the confident predictions
+    virtual void postProcessDetectedObjectFrame(cv::Mat &frame, const std::vector<cv::Mat> &info) = 0;
+
+    // Function to add inference time to the detected object frame
+    virtual void addDetectionTimeToFrame(cv::Mat &frame) = 0;
+
+    // Draws detected object bounding boxes on supplied video frame
+    inline void drawBoundingBoxToFrame(int left, int top, int right, int bottom, cv::Mat& frame)
+    {
+      //Draw a rectangle displaying the bounding box
+      cv::rectangle(frame, cv::Point(left, top), cv::Point(right, bottom), cv::Scalar(0, 0, 255));
+    }
+
+    // Function to add object class to the detected object frame
+    inline void addObjectClassToFrame(int classId, float conf, int left, int top, cv::Mat& frame)
+    {
+      std::string label = std::to_string(conf);
+      if (!_classes.empty())
+	    { 
+      	label = _classes[classId] + ":" + label;
+    	}
+      else
+      {
+        return;
+      }
+      
+      //Add the label at the top of the bounding box
+      int baseLine;
+      cv::Size labelSize = getTextSize(label, cv::FONT_HERSHEY_SIMPLEX, 0.6, 1, &baseLine);
+      top = std::max(top, labelSize.height);
+      cv::rectangle(frame, cv::Point(left, top), cv::Point(left+labelSize.width, top - std::min(top, labelSize.height)), cv::Scalar(255, 0, 255), -1);
+      cv::putText(frame, label, cv::Point(left, top), cv::FONT_HERSHEY_SIMPLEX, 0.6, cv::Scalar(255,255,255));
+    }
+
+    // Function to load the class names in a vector
+    inline void loadClasses(std::string classesFilename)
+    {
+      std::fstream file(classesFilename);
+      std::string line;
+      while(std::getline(file, line))
+      {
+        _classes.push_back(line);
+      }
+      std::cout << "Loaded " << _classes.size() <<" Class Names" << std::endl;
+    }
+
+    // Confidence threshold for class
+    float _confidenceThresh;
+
+    // Non-max Suppression threshold to remove the access bounding boxes
+    float _nonMaxSuppressionThresh;
+
+    // Unique Pointer holding the Object Dectector
+    std::unique_ptr<cv::dnn::Net> _detector;
+
+    // Vector containing class names
+    std::vector<std::string> _classes;
+
+    // Frames Detected
+    int _numFramesDetected{0};
+
+};
+
+#endif

include/Detector/object_detector.h

@@ -0,0 +1,49 @@
+#ifndef YOLO_DETECTOR_H
+#define YOLO_DETECTOR_H
+
+#include <vector>
+#include <string>
+#include <iostream>
+#include <opencv2/opencv.hpp>
+#include <opencv2/highgui.hpp>
+#include <opencv2/dnn.hpp>
+#include "object_detector.h"
+#include "point2D.h"
+#include "tracking_msg.h"
+
+// class which provides functionality to detect object using YOLO algorithm
+class YOLODetector:  public ObjectDetector
+{
+  public:
+
+    // Constructor and Destructor
+    YOLODetector(std::string modelWeights, std::string modelConfig, std::string classesFilename, float confThresh, float nmsThresh);
+    ~YOLODetector();
+
+    // Method to detect object in the msgq having frames and put the detected frames in the outq
+    void detectObjectInQueue(MessageQueue<cv::Mat> &msgq, MessageQueue<cv::Mat> &outq);
+
+    // Method to detect object in the msgq having frames and put the detected frame and detected points in trackq
+    void detectObjectInQueueAndTrack(MessageQueue<cv::Mat> &msgq, MessageQueue<TrackingMsg> &trackq);
+
+    // Method to detect object in the given frame
+    void detectObject(cv::Mat &frame, std::vector<cv::Mat> &info);
+
+  protected:
+
+    // Method to the bounding boxes over the confident predictions
+    void postProcessDetectedObjectFrame(cv::Mat &frame, const std::vector<cv::Mat> &info);
+
+    // Method to add inference time to detected object frame
+    void addDetectionTimeToFrame(cv::Mat &frame);
+
+  private:
+    
+    // Input width and height of the image to the network
+    int _inputWidth;
+    int _inputHeight;
+    float _frameDetectionTime;
+    std::vector<Point2D> _objectCenters;
+};
+
+#endif