Skip to content

Commit

Permalink
Added flags body+upsampling_scale, new tutorial example
Browse files Browse the repository at this point in the history
  • Loading branch information
@gineshidalgo99
gineshidalgo99 committed Feb 8, 2019
1 parent e635967 commit de11748
Show file tree
Hide file tree
Showing 51 changed files with 584 additions and 483 deletions.
4 changes: 3 additions & 1 deletion CMakeLists.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,9 @@ if (WIN32)
endif (WIN32)

# Unity
option(BUILD_UNITY_SUPPORT "Build OpenPose as a Unity plugin." OFF)
if (WIN32)
option(BUILD_UNITY_SUPPORT "Build OpenPose as a Unity plugin." OFF)
endif (WIN32)

# Build as shared library
option(BUILD_SHARED_LIBS "Build as shared lib." ON)
Expand Down
2 changes: 1 addition & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Note: Currently using [travis-matrix-badges](https://github.com/bjfish/travis-ma

[**OpenPose**](https://github.com/CMU-Perceptual-Computing-Lab/openpose) represents the **first real-time multi-person system to jointly detect human body, hand, facial, and foot keypoints (in total 135 keypoints) on single images**.

It is **authored by [Gines Hidalgo](https://www.gineshidalgo.com), [Zhe Cao](https://people.eecs.berkeley.edu/~zhecao), [Tomas Simon](http://www.cs.cmu.edu/~tsimon), [Shih-En Wei](https://scholar.google.com/citations?user=sFQD3k4AAAAJ&hl=en), [Hanbyul Joo](https://jhugestar.github.io), and [Yaser Sheikh](http://www.cs.cmu.edu/~yaser)**. Currently, it is being **maintained by [Gines Hidalgo](https://www.gineshidalgo.com) and [Yaadhav Raaj](https://www.linkedin.com/in/yaadhavraaj)**. In addition, OpenPose would not be possible without the [**CMU Panoptic Studio dataset**](http://domedb.perception.cs.cmu.edu). We would also like to thank all the people who helped OpenPose in any way. The main contributors are listed in [doc/contributors.md](doc/contributors.md).
It is **authored by [Gines Hidalgo](https://www.gineshidalgo.com), [Zhe Cao](https://people.eecs.berkeley.edu/~zhecao), [Tomas Simon](http://www.cs.cmu.edu/~tsimon), [Shih-En Wei](https://scholar.google.com/citations?user=sFQD3k4AAAAJ&hl=en), [Hanbyul Joo](https://jhugestar.github.io), and [Yaser Sheikh](http://www.cs.cmu.edu/~yaser)**. Currently, it is being **maintained by [Gines Hidalgo](https://www.gineshidalgo.com) and [Yaadhav Raaj](https://www.raaj.tech)**. In addition, OpenPose would not be possible without the [**CMU Panoptic Studio dataset**](http://domedb.perception.cs.cmu.edu). We would also like to thank all the people who helped OpenPose in any way. The main contributors are listed in [doc/contributors.md](doc/contributors.md).

<!-- The [original CVPR 2017 repo](https://github.com/ZheC/Multi-Person-Pose-Estimation) includes Matlab and Python versions, as well as the training code. The body pose estimation work is based on [the original ECCV 2016 demo](https://github.com/CMU-Perceptual-Computing-Lab/caffe_rtpose). -->

Expand Down
2 changes: 1 addition & 1 deletion doc/contributors.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ OpenPose is authored by [Gines Hidalgo](https://www.gineshidalgo.com/), [Zhe Cao
### Contributors
We would also like to thank the following people who have highly contributed to OpenPose:

1. [Yaadhav Raaj](https://www.linkedin.com/in/yaadhavraaj): OpenPose maintainer, CPU version, OpenCL version, Mac version, Python API, and person tracker.
1. [Yaadhav Raaj](https://www.raaj.tech): OpenPose maintainer, CPU version, OpenCL version, Mac version, Python API, and person tracker.
2. [Bikramjot Hanzra](https://www.linkedin.com/in/bikz05): Former OpenPose maintainer, CMake (Ubuntu and Windows) version, and initial Travis Build version for Ubuntu.
3. [Donglai Xiang](https://xiangdonglai.github.io): Camera calibration toolbox improvement, including the implementation of its bundle adjustment algorithm.
4. [Luis Fernando Fraga](https://github.com/fragalfernando): Implementation of Lukas-Kanade algorith and person ID extractor.
Expand Down
3 changes: 2 additions & 1 deletion doc/demo_overview.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -166,11 +166,12 @@ Each flag is divided into flag name, default value, and description.
- DEFINE_double(fps_max, -1., "Maximum processing frame rate. By default (-1), OpenPose will process frames as fast as possible. Example usage: If OpenPose is displaying images too quickly, this can reduce the speed so the user can analyze better each frame from the GUI.");

4. OpenPose Body Pose
- DEFINE_bool(body_disable, false, "Disable body keypoint detection. Option only possible for faster (but less accurate) face keypoint detection.");
- DEFINE_int32(body, 1, "Select 0 to disable body keypoint detection (e.g., for faster but less accurate face keypoint detection, custom hand detector, etc.), 1 (default) for body keypoint estimation, and 2 to disable its internal body pose estimation network but still still run the greedy association parsing algorithm");
- DEFINE_string(model_pose, "BODY_25", "Model to be used. E.g., `COCO` (18 keypoints), `MPI` (15 keypoints, ~10% faster), `MPI_4_layers` (15 keypoints, even faster but less accurate).");
- DEFINE_string(net_resolution, "-1x368", "Multiples of 16. If it is increased, the accuracy potentially increases. If it is decreased, the speed increases. For maximum speed-accuracy balance, it should keep the closest aspect ratio possible to the images or videos to be processed. Using `-1` in any of the dimensions, OP will choose the optimal aspect ratio depending on the user's input value. E.g., the default `-1x368` is equivalent to `656x368` in 16:9 resolutions, e.g., full HD (1980x1080) and HD (1280x720) resolutions.");
- DEFINE_int32(scale_number, 1, "Number of scales to average.");
- DEFINE_double(scale_gap, 0.25, "Scale gap between scales. No effect unless scale_number > 1. Initial scale is always 1. If you want to change the initial scale, you actually want to multiply the `net_resolution` by your desired initial scale.");
- DEFINE_double(upsampling_ratio, 0., "Upsampling ratio between the `net_resolution` and the output net results. A value less or equal than 0 (default) will use the network default value (recommended).");

5. OpenPose Body Pose Heatmaps and Part Candidates
- DEFINE_bool(heatmaps_add_parts, false, "If true, it will fill op::Datum::poseHeatMaps array with the body part heatmaps, and analogously face & hand heatmaps to op::Datum::faceHeatMaps & op::Datum::handHeatMaps. If more than one `add_heatmaps_X` flag is enabled, it will place then in sequential memory order: body parts + bkg + PAFs. It will follow the order on POSE_BODY_PART_MAPPING in `src/openpose/pose/poseParameters.cpp`. Program speed will considerably decrease. Not required for OpenPose, enable it only if you intend to explicitly use this information later.");
Expand Down
2 changes: 1 addition & 1 deletion doc/modules/3d_reconstruction_module.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ It should be similar to the following image.
You can copy and modify the OpenPose 3-D demo to use any camera brand by:

1. You can optionally turn off the `WITH_FLIR_CAMERA` while compiling CMake.
2. Copy `examples/tutorial_api_cpp/7_synchronous_custom_input.cpp` (or 9_synchronous_custom_all.cpp).
2. Copy `examples/tutorial_api_cpp/13_synchronous_custom_input.cpp` (or `17_synchronous_custom_all_and_datum.cpp`).
3. Modify `WUserInput` and add your custom code there. Your code should fill `Datum::name`, `Datum::cameraMatrix`, `Datum::cvInputData`, and `Datum::cvOutputData` (fill cvOutputData = cvInputData).
4. Remove `WUserPostProcessing` and `WUserOutput` (unless you want to have your custom post-processing and/or output).

Expand Down
84 changes: 44 additions & 40 deletions doc/release_notes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -277,63 +277,67 @@ OpenPose Library - Release Notes
8. Given that display can be disabled in all examples, they all have been added to the Travis build so they can be tested.
7. Added a virtual destructor to almost all clases, so they can be inherited. Exceptions (for performance reasons): Array, Point, Rectangle, CvMatToOpOutput, OpOutputToCvMat.
8. Auxiliary classes in errorAndLog turned into namespaces (Profiler must be kept as class to allow static parameters).
9. Added flag `--frame_step` to allow the user to select the step or gap between processed frames. E.g., `--frame_step 5` would read and process frames 0, 5, 10, etc.
10. Added sanity checks to avoid `--frame_last` to be smaller than `--frame_first` or higher than the number of total frames.
11. Array improvements for Pybind11 compatibility:
9. Added flags:
1. Added flag `--frame_step` to allow the user to select the step or gap between processed frames. E.g., `--frame_step 5` would read and process frames 0, 5, 10, etc.
2. Previously hardcoded `COCO_CHALLENGE` variable turned into user configurable flag `--maximize_positives`.
3. Added flag `--verbose` to plot the progress.
4. Added flag `--fps_max` to limit the maximum processing frame rate of OpenPose (useful to display results at a maximum desired speed).
5. Added sanit30. Added the flags `--prototxt_path` and `--caffemodel_path` to allow custom ProtoTxt and CaffeModel paths.
6. Added the flags `--face_detector` and `--hand_detector`, that enable the user to select the face/hand rectangle detector that is used for the later face/hand keypoint detection. It includes OpenCV (for face), and also allows the user to provide its own input. Flag `--hand_tracking` is removed and integrated into this flag too.
y checks to avoid `--frame_last` to be smaller than `--frame_first` or higher than the number of total frames.
7. Added the flag `--upsampling_ratio`, which controls the upsampling than OpenPose will perform to the frame before the greedy association parsing algorithm.
8. Added the flag `--body` (replacing `--body_disable`), which adds the possibility of disabling the OpenPose pose network but still running the greedy association parsing algorithm (on top of the user heatmaps, see the associated `tutorial_api_cpp` example).
10. Array improvements for Pybind11 compatibility:
1. Array::getStride() to get step size of each dimension of the array.
2. Array::getPybindPtr() to get an editable const pointer.
3. Array::pData as binding of spData.
4. Array::Array that takes as input a pointer, so it does not re-allocate memory.
12. Producer defined inside Wrapper rather than being defined on each example.
13. Reduced many Visual Studio warnings (e.g., uncontrolled conversions between types).
14. Added new keypoint-related auxiliary functions in `utilities/keypoints.hpp`.
15. Function `resizeFixedAspectRatio` can take already allocated memory (e.g., faster if target is an Array<T> object, no intermediate cv::Mat required).
16. Added compatibility for OpenCV 4.0, while preserving 2.4.X and 3.X compatibility.
17. Improved and added several functions to `utilities/keypoints.hpp` and Array to simplify keypoint post-processing.
18. Removed warnings from Spinnaker SDK at compiling time.
19. All bash scripts incorporate `#!/bin/bash` to tell the terminal that they are bash scripts.
20. Added flag `--verbose` to plot the progress.
21. Added find_package(Protobuf) to allow specific versions of Protobuf.
22. Video saving improvements:
11. Producer defined inside Wrapper rather than being defined on each example.
12. Reduced many Visual Studio warnings (e.g., uncontrolled conversions between types).
13. Added new keypoint-related auxiliary functions in `utilities/keypoints.hpp`.
14. Function `resizeFixedAspectRatio` can take already allocated memory (e.g., faster if target is an Array<T> object, no intermediate cv::Mat required).
15. Added compatibility for OpenCV 4.0, while preserving 2.4.X and 3.X compatibility.
16. Improved and added several functions to `utilities/keypoints.hpp` and Array to simplify keypoint post-processing.
17. Removed warnings from Spinnaker SDK at compiling time.
18. All bash scripts incorporate `#!/bin/bash` to tell the terminal that they are bash scripts.
19. Added find_package(Protobuf) to allow specific versions of Protobuf.
20. Video saving improvements:
1. Video (`--write_video`) can be generated from images (`--image_dir`), as long as they maintain the same resolution.
2. Video with the 3D output can be saved with the new `--write_video_3d` flag.
3. Added the capability of saving videos in MP4 format (by using the ffmpeg library).
4. Added the flag `write_video_with_audio` to enable saving these output MP4 videos with audio.
23. Added `--fps_max` flag to limit the maximum processing frame rate of OpenPose (useful to display results at a maximum desired speed).
24. Frame undistortion can be applied not only to FLIR cameras, but also to all other input sources (image, webcam, video, etc.).
25. Calibration improvements:
21. Frame undistortion can be applied not only to FLIR cameras, but also to all other input sources (image, webcam, video, etc.).
22. Calibration improvements:
1. Improved chessboard orientation detection, more robust and less errors.
2. Triangulation functions (triangulate and triangulateWithOptimization) public, so calibration can use them for bundle adjustment.
3. Added bundle adjustment refinement for camera extrinsic calibration.
4. Added `CameraMatrixInitial` field into the XML calibration files to keep the information of the original camera extrinsic parameters when bundle adjustment is run.
26. Added Mac OpenCL compatibility.
27. Added documentation for Nvidia TX2 with JetPack 3.3.
28. Added Travis build check for several configurations: Ubuntu (14/16)/Mac/Windows, CPU/CUDA/OpenCL, with/without Python, and Release/Debug.
29. Assigned 755 access to all sh scripts (some of them were only 644).
30. Added the flags `--prototxt_path` and `--caffemodel_path` to allow custom ProtoTxt and CaffeModel paths.
31. Replaced the old Python wrapper for an updated Pybind11 wrapper version, that includes all the functionality of the C++ API.
32. Function getFilesOnDirectory() can extra all basic image file types at once without requiring to manually enumerate them.
33. Added the flags `--face_detector` and `--hand_detector`, that enable the user to select the face/hand rectangle detector that is used for the later face/hand keypoint detection. It includes OpenCV (for face), and also allows the user to provide its own input. Flag `--hand_tracking` is removed and integrated into this flag too.
34. Maximum queue size per OpenPose thread is configurable through the Wrapper class.
35. Added pre-processing capabilities to Wrapper (WorkerType::PreProcessing), which will be run right after the image has been read.
36. Removed boost::shared_ptr and caffe::Blob dependencies from the headers. No 3rdparty dependencies left on headers (except dim3 for CUDA).
37. Added `poseNetOutput` to Datum so that user can introduce his custom network output.
23. Added Mac OpenCL compatibility.
24. Added documentation for Nvidia TX2 with JetPack 3.3.
25. Added Travis build check for several configurations: Ubuntu (14/16)/Mac/Windows, CPU/CUDA/OpenCL, with/without Python, and Release/Debug.
26. Assigned 755 access to all sh scripts (some of them were only 644).
27. Replaced the old Python wrapper for an updated Pybind11 wrapper version, that includes all the functionality of the C++ API.
28. Function getFilesOnDirectory() can extra all basic image file types at once without requiring to manually enumerate them.
29. Maximum queue size per OpenPose thread is configurable through the Wrapper class.
30. Added pre-processing capabilities to Wrapper (WorkerType::PreProcessing), which will be run right after the image has been read.
31. Removed boost::shared_ptr and caffe::Blob dependencies from the headers. No 3rdparty dependencies left on headers (except dim3 for CUDA).
32. Added Array `poseNetOutput` to Datum so that user can introduce his custom network output.
2. Functions or parameters renamed:
1. By default, python example `tutorial_developer/python_2_pose_from_heatmaps.py` was using 2 scales starting at -1x736, changed to 1 scale at -1x368.
2. WrapperStructPose default parameters changed to match those of the OpenPose demo binary.
3. WrapperT.configure() changed from 1 function that requries all arguments to individual functions that take 1 argument each.
4. Added `Forward` to all net classes that automatically selects between CUDA, OpenCL, or CPU-only version depending on the defines.
5. Previously hardcoded `COCO_CHALLENGE` variable turned into user configurable flag `--maximize_positives`.
6. Removed old COCO 2014 validation scripts.
7. WrapperStructOutput split into WrapperStructOutput and WrapperStructGui.
8. Replaced `--camera_fps` flag by `--write_video_fps`, given that it was a confusing name: It did not affect the webcam FPS, but only the FPS of the output video. In addition, default value changed from 30 to -1.
9. Renamed `--frame_keep_distortion` as `--frame_undistort`, which performs the opposite operation (the default value has been also changed to the opposite).
10. Renamed `--camera_parameter_folder` as `--camera_parameter_path` because it could also take a whole XML file path rather than its parent folder.
11. Default value of flag `--scale_gap` changed from 0.3 to 0.25.
12. Moved most sh scripts into the `scripts/` folder. Only models/getModels.sh and the `*.bat` files are kept under `models/` and `3rdparty/windows`.
13. For Python compatibility and scalability increase, template `TDatums` used for `include/openpose/wrapper/wrapper.hpp` has changed from `std::vector<Datum>` to `std::vector<std::shared_ptr<Datum>>`, including the respective changes in all the worker classes. In addition, some template classes have been simplified to only take 1 template parameter for user simplicity.
14. Renamed intRound, charRound, etc. by positiveIntRound, positiveCharRound, etc. so that people can realize it is not safe for negative numbers.
15. Flag `--hand_tracking` is a subcase of `--hand_detector`, so it has been removed and incorporated as `--hand_detector 3`.
5. Removed old COCO 2014 validation scripts.
6. WrapperStructOutput split into WrapperStructOutput and WrapperStructGui.
7. Replaced flags:
1. Replaced `--camera_fps` flag by `--write_video_fps`, given that it was a confusing name: It did not affect the webcam FPS, but only the FPS of the output video. In addition, default value changed from 30 to -1.
2. Flag `--hand_tracking` is a subcase of `--hand_detector`, so it has been removed and incorporated as `--hand_detector 3`.
8. Renamed `--frame_keep_distortion` as `--frame_undistort`, which performs the opposite operation (the default value has been also changed to the opposite).
9. Renamed `--camera_parameter_folder` as `--camera_parameter_path` because it could also take a whole XML file path rather than its parent folder.
10. Default value of flag `--scale_gap` changed from 0.3 to 0.25.
11. Moved most sh scripts into the `scripts/` folder. Only models/getModels.sh and the `*.bat` files are kept under `models/` and `3rdparty/windows`.
12. For Python compatibility and scalability increase, template `TDatums` used for `include/openpose/wrapper/wrapper.hpp` has changed from `std::vector<Datum>` to `std::vector<std::shared_ptr<Datum>>`, including the respective changes in all the worker classes. In addition, some template classes have been simplified to only take 1 template parameter for user simplicity.
13. Renamed intRound, charRound, etc. by positiveIntRound, positiveCharRound, etc. so that people can realize it is not safe for negative numbers.
3. Main bugs fixed:
1. CMake-GUI was forcing to Release mode, allowed Debug modes too.
2. NMS returns in index 0 the number of found peaks. However, while the number of peaks was truncated to a maximum of 127, this index 0 was saving the real number instead of the truncated one.
Expand Down
Loading

0 comments on commit de11748

Please sign in to comment.