Sanity checks for flir camera with 3-d CMU-Perceptual-Computing-Lab#1254

doc/demo_overview.md

@@ -210,7 +210,7 @@ Each flag is divided into flag name, default value, and description.
 - DEFINE_bool(disable_blending,           false,          "If enabled, it will render the results (keypoint skeletons or heatmaps) on a black background, instead of being rendered into the original image. Related: `part_to_show`, `alpha_pose`, and `alpha_pose`.");
 
 11. OpenPose Rendering Pose
-- DEFINE_double(render_threshold,         0.05,           "Only estimated keypoints whose score confidences are higher than this threshold will be rendered. Generally, a high threshold (> 0.5) will only render very clear body parts; while small thresholds (~0.1) will also output guessed and occluded keypoints, but also more false positives (i.e., wrong detections).");
+- DEFINE_double(render_threshold,         0.05,           "Only estimated keypoints whose score confidences are higher than this threshold will be rendered. Note: Rendered refers only to visual display in the OpenPose basic GUI, not in the saved results. Generally, a high threshold (> 0.5) will only render very clear body parts; while small thresholds (~0.1) will also output guessed and occluded keypoints, but also more false positives (i.e., wrong detections).");
 - DEFINE_int32(render_pose,               -1,             "Set to 0 for no rendering, 1 for CPU rendering (slightly faster), and 2 for GPU rendering (slower but greater functionality, e.g., `alpha_X` flags). If -1, it will pick CPU if CPU_ONLY is enabled, or GPU if CUDA is enabled. If rendering is enabled, it will render both `outputData` and `cvOutputData` with the original image and desired body part to be shown (i.e., keypoints, heat maps or PAFs).");
 - DEFINE_double(alpha_pose,               0.6,            "Blending factor (range 0-1) for the body part rendering. 1 will show it completely, 0 will hide it. Only valid for GPU rendering.");
 - DEFINE_double(alpha_heatmap,            0.7,            "Blending factor (range 0-1) between heatmap and original frame. 1 will only show the heatmap, 0 will only show the frame. Only valid for GPU rendering.");

doc/faq.md

@@ -18,6 +18,7 @@ OpenPose - Frequently Asked Question (FAQ)
         12. [CMake-GUI Error While Getting Default Caffe](#cmake-gui-error-while-getting-default-caffe)
         13. [Libgomp Out of Memory Error](#libgomp-out-of-memory-error)
         14. [Runtime Error with Turing GPU (Tesla T4) or Volta GPU][#runtime-error-with-turing-gpu-teslat4-or-volta-gpu)
+        15. [Obscure CMake Error about Caffe or Pybind](#obscure-cmake-error-about-caffe-or-pybind).
     2. [Speed Performance Issues](#speed-performance-issues)
         1. [Speed Up, Memory Reduction, and Benchmark](#speed-up-memory-reduction-and-benchmark)
         2. [How to Measure the Latency Time?](#how-to-measure-the-latency-time)
@@ -171,6 +172,14 @@ git submodle update
 
 
 
+#### Obscure CMake Error about Caffe or Pybind
+**Q:** There appear some weird and obscure errors on CMake about Caffe and/or Pybind.
+
+**A**: Check [doc/prerequisites.md#general-tips](./prerequisites.md#general-tips) to run the `git submodule` command.
+
+
+
+
 
 ### Speed Performance Issues
 #### Speed Up, Memory Reduction, and Benchmark

doc/installation.md

@@ -330,9 +330,10 @@ Also as a side note, if the default installation fails (i.e., the one explained
     3. `Caffe_LIBS` set to `/usr/local/lib/libcaffe.dylib`.
     4. Run `Configure` and `Generate` from CMake GUI.
 
-You may also have to apply the following patch if you have the latest OSX 10.14. It can be done as follows:
-
-`cd 3rdparty/caffe; git apply ../../scripts/osx/mac_opencl_patch.txt`
+In addition, if you face an OpenCV error during compiling time similar to `fatal error: 'opencv2/highgui/highgui.hpp' file not found`, please apply the following patch (this error has been reported in the latest OSX 10.14):
+```
+cd 3rdparty/caffe; git apply ../../scripts/osx/mac_opencl_patch.txt
+```
 
 
 #### 3D Reconstruction Module

doc/modules/3d_reconstruction_module.md

@@ -100,7 +100,7 @@ The visual GUI should show 3 screens.
 It should be similar to the following image.
 
 <p align="center">
-    <img src="media/openpose3d.png">
+    <img src="../media/openpose3d.gif">
 </p>
 
 

include/openpose/flags.hpp

@@ -184,9 +184,10 @@ DEFINE_bool(disable_blending,           false,          "If enabled, it will ren
                                                         " `alpha_pose`, and `alpha_pose`.");
 // OpenPose Rendering Pose
 DEFINE_double(render_threshold,         0.05,           "Only estimated keypoints whose score confidences are higher than this threshold will be"
-                                                        " rendered. Generally, a high threshold (> 0.5) will only render very clear body parts;"
-                                                        " while small thresholds (~0.1) will also output guessed and occluded keypoints, but also"
-                                                        " more false positives (i.e., wrong detections).");
+                                                        " rendered. Note: Rendered refers only to visual display in the OpenPose basic GUI, not in"
+                                                        " the saved results. Generally, a high threshold (> 0.5) will only render very clear body"
+                                                        " parts; while small thresholds (~0.1) will also output guessed and occluded keypoints,"
+                                                        " but also more false positives (i.e., wrong detections).");
 DEFINE_int32(render_pose,               -1,             "Set to 0 for no rendering, 1 for CPU rendering (slightly faster), and 2 for GPU rendering"
                                                         " (slower but greater functionality, e.g., `alpha_X` flags). If -1, it will pick CPU if"
                                                         " CPU_ONLY is enabled, or GPU if CUDA is enabled. If rendering is enabled, it will render"

include/openpose/producer/spinnakerWrapper.hpp

@@ -23,6 +23,10 @@ namespace op
 
         std::vector<cv::Mat> getRawFrames();
 
+        /**
+         * Note: The camera parameters are only read if undistortImage is true. This should be changed to add a
+         * new bool flag in the constructor, e.g., readCameraParameters
+         */
         std::vector<cv::Mat> getCameraMatrices() const;
 
         std::vector<cv::Mat> getCameraExtrinsics() const;

include/openpose/utilities/keypoint.hpp

@@ -70,8 +70,13 @@ namespace op
     float getKeypointsRoi(const Array<T>& keypoints, const int personA, const int personB, const T threshold);
 
     template <typename T>
-    float getKeypointsRoi(const Array<T>& keypointsA, const int personA, const Array<T>& keypointsB, const int personB,
-                          const T threshold);
+    float getKeypointsRoi(
+        const Array<T>& keypointsA, const int personA, const Array<T>& keypointsB, const int personB,
+        const T threshold);
+
+    template <typename T>
+    float getKeypointsRoi(
+        const Rectangle<T>& rectangleA, const Rectangle<T>& rectangleB);
 }
 
 #endif // OPENPOSE_UTILITIES_KEYPOINT_HPP

include/openpose/wrapper/wrapperStructFace.hpp

@@ -54,7 +54,8 @@ namespace op
 
         /**
          * Rendering threshold. Only estimated keypoints whose score confidences are higher than this value will be
-         * rendered. Generally, a high threshold (> 0.5) will only render very clear body parts; while small thresholds
+         * rendered. Note: Rendered refers only to visual display in the OpenPose basic GUI, not in the saved results.
+         * Generally, a high threshold (> 0.5) will only render very clear body parts; while small thresholds
          * (~0.1) will also output guessed and occluded keypoints, but also more false positives (i.e., wrong
          * detections).
          */

include/openpose/wrapper/wrapperStructHand.hpp

@@ -73,7 +73,8 @@ namespace op
 
         /**
          * Rendering threshold. Only estimated keypoints whose score confidences are higher than this value will be
-         * rendered. Generally, a high threshold (> 0.5) will only render very clear body parts; while small thresholds
+         * rendered. Note: Rendered refers only to visual display in the OpenPose basic GUI, not in the saved results.
+         * Generally, a high threshold (> 0.5) will only render very clear body parts; while small thresholds
          * (~0.1) will also output guessed and occluded keypoints, but also more false positives (i.e., wrong
          * detections).
          */

include/openpose/wrapper/wrapperStructPose.hpp

@@ -143,7 +143,8 @@ namespace op
 
         /**
          * Rendering threshold. Only estimated keypoints whose score confidences are higher than this value will be
-         * rendered. Generally, a high threshold (> 0.5) will only render very clear body parts; while small thresholds
+         * rendered. Note: Rendered refers only to visual display in the OpenPose basic GUI, not in the saved results.
+         * Generally, a high threshold (> 0.5) will only render very clear body parts; while small thresholds
          * (~0.1) will also output guessed and occluded keypoints, but also more false positives (i.e., wrong
          * detections).
          */

src/openpose/3d/poseTriangulation.cpp

@@ -505,8 +505,8 @@ namespace op
     {
         try
         {
-            return reconstructArray(std::vector<std::vector<Array<float>>>{keypointsVector},
-                                    cameraMatrices, imageSizes).at(0);
+            return reconstructArray(
+                std::vector<std::vector<Array<float>>>{keypointsVector}, cameraMatrices, imageSizes).at(0);
         }
         catch (const std::exception& e)
         {
@@ -515,13 +515,30 @@ namespace op
         }
     }
 
+    const std::string sFlirErrorMessage{
+        " If you are simultaneously using FLIR cameras (`--flir_camera`) and the 3-D reconstruction module"
+        " (`--3d), you should also enable `--frame_undistort` so their camera parameters are read."};
     std::vector<Array<float>> PoseTriangulation::reconstructArray(
         const std::vector<std::vector<Array<float>>>& keypointsVectors,
         const std::vector<cv::Mat>& cameraMatrices,
         const std::vector<Point<int>>& imageSizes) const
     {
         try
         {
+            // Sanity checks
+            if (cameraMatrices.size() < 2)
+                error("3-D reconstruction (`--3d`) requires at least 2 camera views, only found "
+                    + std::to_string(cameraMatrices.size()) + "camera parameter matrices." + sFlirErrorMessage,
+                    __LINE__, __FUNCTION__, __FILE__);
+            for (const auto& cameraMatrix : cameraMatrices)
+                if (cameraMatrix.empty())
+                    error("Camera matrix was found empty during 3-D reconstruction (`--3d`)." + sFlirErrorMessage,
+                        __LINE__, __FUNCTION__, __FILE__);
+            if (cameraMatrices.size() != imageSizes.size())
+                error("The camera parameters and number of images must be the same ("
+                    + std::to_string(cameraMatrices.size()) + " vs. " + std::to_string(imageSizes.size()) + ").",
+                    __LINE__, __FUNCTION__, __FILE__);
+            // Run 3-D reconstruction
             bool keypointsReconstructed = false;
             std::vector<Array<float>> keypoints3Ds(keypointsVectors.size());
             // std::vector<std::thread> threads(keypointsVectors.size()-1);

src/openpose/face/faceExtractorNet.cpp

@@ -27,7 +27,7 @@ namespace op
                    __LINE__, __FUNCTION__, __FILE__);
             // Warnings
             if (!mHeatMapTypes.empty())
-                log("Note only keypoint heatmaps are available with face heatmaps (no background nor PAFs).",
+                log("Note that only the keypoint heatmaps are available with face heatmaps (no background nor PAFs).",
                     Priority::High);
         }
         catch (const std::exception& e)

src/openpose/hand/handExtractorNet.cpp

@@ -29,7 +29,7 @@ namespace op
                    __LINE__, __FUNCTION__, __FILE__);
             // Warnings
             if (!mHeatMapTypes.empty())
-                log("Note only keypoint heatmaps are available with hand heatmaps (no background nor PAFs).",
+                log("Note that only the keypoint heatmaps are available with hand heatmaps (no background nor PAFs).",
                     Priority::High);
         }
         catch (const std::exception& e)

src/openpose/net/bodyPartConnectorBase.cpp

@@ -83,20 +83,20 @@ namespace op
     void getRoiDiameterAndBounds(
         Rectangle<int>& roi, int& diameter, int& partFirstNon0, int& partLastNon0,
         const std::vector<int>& personVector, const T* const peaksPtr,
-        const int partInit, const int partEnd)
+        const int partInit, const int partEnd, const float margin)
     {
         try
         {
             // Find ROI, partFirstNon0, and partLastNon0
-            roi = Rectangle<int>{0,0,0,0};
+            roi = Rectangle<int>{std::numeric_limits<int>::max(),std::numeric_limits<int>::max(),0,0};
             partFirstNon0 = -1;
             partLastNon0 = -1;
             for (auto part = partInit ; part < partEnd ; part++)
             {
                 const auto x = peaksPtr[personVector[part]-2];
                 const auto y = peaksPtr[personVector[part]-1];
                 const auto score = peaksPtr[personVector[part]];
-                if (score > 0)
+                if (personVector[part] > 0 && score > 0)
                 {
                     // ROI
                     if (roi.x > x)
@@ -105,7 +105,7 @@ namespace op
                         roi.y = y;
                     if (roi.width < x)
                         roi.width = x;
-                    if (roi.height > y)
+                    if (roi.height < y)
                         roi.height = y;
                     // First keypoint?
                     if (partFirstNon0 < 0)
@@ -114,11 +114,24 @@ namespace op
                     partLastNon0 = part;
                 }
             }
-            // From [p1, p2] to [p1, width, height]
-            roi.width -= roi.x;
-            roi.height -= roi.y;
-            // diameter
-            diameter = fastMax(roi.width, roi.height);
+            if (partLastNon0 > -1)
+            {
+                // Add margin
+                const auto marginX = roi.width * margin;
+                const auto marginY = roi.height * margin;
+                roi.x -= marginX;
+                roi.y -= marginY;
+                roi.width += 2*marginX;
+                roi.height += 2*marginY;
+                // partFirstNon0+1 for for loops
+                partLastNon0++;
+                // From [p1, p2] to [p1, width, height]
+                // +1 to account for 1-line keypoints
+                roi.width += 1 - roi.x;
+                roi.height += 1 - roi.y;
+                // diameter
+                diameter = fastMax(roi.width, roi.height);
+            }
         }
         catch (const std::exception& e)
         {
@@ -712,7 +725,8 @@ namespace op
             faceValidSubsetIndexes.reserve(peopleVector.size());
             // Face invalid sets
             std::vector<int> faceInvalidSubsetIndexes;
-            faceInvalidSubsetIndexes.reserve(peopleVector.size());
+            if (numberBodyParts >= 135)
+                faceInvalidSubsetIndexes.reserve(peopleVector.size());
             // For each person candidate
             for (auto person = 0u ; person < peopleVector.size() ; person++)
             {
@@ -768,6 +782,77 @@ namespace op
                     error("Bad personCounter (" + std::to_string(personCounter) + "). Bug in this"
                           " function if this happens.", __LINE__, __FUNCTION__, __FILE__);
             }
+            // Random standalone facial keypoints --> Merge into a more complete face
+            if (numberPeople > 0)
+            {
+                // Check invalid faces
+                for (const auto& personInvalid : faceInvalidSubsetIndexes)
+                {
+                    // Get ROI of current face
+                    Rectangle<int> roiInvalid;
+                    int diameterInvalid = -1;
+                    int partFirstNon0Invalid = -1;
+                    int partLastNon0Invalid = -1;
+                    getRoiDiameterAndBounds(
+                        roiInvalid, diameterInvalid, partFirstNon0Invalid, partLastNon0Invalid,
+                        peopleVector[personInvalid].first, peaksPtr, 65, 135, 0.2f);
+                    // Check all valid faces to find best candidate
+                    float keypointsRoiBest = 0.f;
+                    auto keypointsRoiBestIndex = -1;
+                    for (auto personId = 0u ; personId < faceValidSubsetIndexes.size() ; personId++)
+                    {
+                        const auto& personValid = faceValidSubsetIndexes[personId];
+                        // Get ROI of current face
+                        Rectangle<int> roiValid;
+                        int diameterValid = -1;
+                        int partFirstNon0Valid = -1;
+                        int partLastNon0Valid = -1;
+                        getRoiDiameterAndBounds(
+                            roiValid, diameterValid, partFirstNon0Valid, partLastNon0Valid, peopleVector[personValid].first,
+                            peaksPtr, 65, 135, 0.1f);
+                        // Get ROI between both faces
+                        const auto keypointsRoi = getKeypointsRoi(roiValid, roiInvalid);
+                        // Update best so far
+                        if (keypointsRoiBest < keypointsRoi)
+                        {
+                            keypointsRoiBest = keypointsRoi;
+                            keypointsRoiBestIndex = personId;
+                        }
+                    }
+                    // If invalid and best valid candidate overlap enough --> Merge them
+                    if (keypointsRoiBest > 0.3f || (keypointsRoiBest > 0.01f && faceValidSubsetIndexes.size() < 3))
+                    {
+                        const auto& personValid = faceValidSubsetIndexes[keypointsRoiBestIndex];
+                        // If it is from that face --> Combine invalid face keypoints into valid face
+                        for (auto part = partFirstNon0Invalid ; part < partLastNon0Invalid ; part++)
+                        {
+                            auto& personVectorValid = peopleVector[personValid].first;
+                            const auto scoreValid = peaksPtr[personVectorValid[part]];
+                            const auto& personVectorInvalid = peopleVector[personInvalid].first;
+                            const auto scoreInvalid = peaksPtr[personVectorInvalid[part]];
+                            // If the new one has a keypoint...
+                            if (personVectorInvalid[part] != 0)
+                            {
+                                // ... and the original face does not have it, then add it to it
+                                if (personVectorValid[part] == 0)
+                                {
+                                    if (personVectorInvalid[part] != 0)
+                                    {
+                                        personVectorValid[part] = personVectorInvalid[part];
+                                        peopleVector[personValid].second += scoreInvalid;
+                                    }
+                                }
+                                // ... and its score is higher than the original one, then replace it
+                                else if (scoreValid < scoreInvalid)
+                                {
+                                    personVectorValid[part] = personVectorInvalid[part];
+                                    peopleVector[personValid].second += scoreInvalid - scoreValid;
+                                }
+                            }
+                        }
+                    }
+                }
+            }
             // If no people found --> Repeat with maximizePositives = true
             // Result: Increased COCO mAP because we catch more foot-only images
             if (numberPeople == 0 && !maximizePositives)