Skip to content

Commit

Permalink
custom_video dir
Browse files Browse the repository at this point in the history
  • Loading branch information
@jeffheo
jeffheo committed Jun 7, 2023
1 parent aeb0907 commit 4f5c23d
Show file tree
Hide file tree
Showing 114 changed files with 17,306 additions and 4 deletions.
7 changes: 3 additions & 4 deletions README_jeff.md → custom_video/README_custom.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,10 +54,9 @@ VIBE is another 3D HMR model that predicts the SMPL parameters of human motion g
| | -- <ACTION>.<INDEX>.mp4
| | ......
| -- /exps
| | -- /your_experiment_directory_<ACTION>
| | | -- /<ACTION>.<INDEX>.frames
| | | -- /<ACTION>.<INDEX>.op
| | | -- /<ACTION>.<INDEX>.vibe
| | -- /<ACTION>.<INDEX>.frames
| | -- /<ACTION>.<INDEX>.op
| | -- /<ACTION>.<INDEX>.vibe
| | ......
| -- opt_cam_IMG_6287.pt
| -- opt_cam_IMG_6289.pt
Expand Down
19 changes: 19 additions & 0 deletions custom_video/VIBE_custom/.github/ISSUE_TEMPLATE/bug-report.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
---
name: Bug Report
about: Use this to report bugs
title: "[BUG]"
labels: bug
assignees: ''

---

Thanks for your interest in our research!

If you have problems running our code, please include;

1. your operating system and the version,
2. your python version,
3. your pytorch version,
4. the stack trace of the error that you see,

Specifically, if you have an issue with pyrender or OpenGL setup & installation, please refer to pyrender [docs](https://pyrender.readthedocs.io/en/latest/) or [github issues](https://github.com/mmatl/pyrender/issues).
20 changes: 20 additions & 0 deletions custom_video/VIBE_custom/.github/ISSUE_TEMPLATE/feature_request.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
name: Feature request
about: Use this to suggest an idea for this project
title: "[FEATURE]"
labels: enhancement
assignees: ''

---

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.
7 changes: 7 additions & 0 deletions custom_video/VIBE_custom/.gitignore
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
.idea/
data
__pycache__/
vibe-env/
output/
*.mp4
results
107 changes: 107 additions & 0 deletions custom_video/VIBE_custom/LICENSE
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
License

Software Copyright License for non-commercial scientific research purposes
Please read carefully the following terms and conditions and any accompanying documentation before you download
and/or use the VIBE model, data and software, (the "Model & Software"), including 3D meshes, software, and scripts.
By downloading and/or using the Model & Software (including downloading, cloning, installing, and any other use
of this github repository), you acknowledge that you have read these terms and conditions, understand them, and
agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use
the Model & Software. Any infringement of the terms of this agreement will automatically terminate your rights
under this License

Ownership / Licensees
The Software and the associated materials has been developed at the

Max Planck Institute for Intelligent Systems (hereinafter "MPI").

Any copyright or patent right is owned by and proprietary material of the

Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (hereinafter “MPG”; MPI and MPG hereinafter
collectively “Max-Planck”)

hereinafter the “Licensor”.

This software includes the SMPL Body Model. By downloading this software, you are agreeing to be bound by the terms of the SMPL Model License

https://smpl.is.tue.mpg.de/modellicense

which is necessary to create SMPL body models.

SMPL bodies that are generated with VIBE can be distributed freely under the SMPL Body License

https://smpl.is.tue.mpg.de/bodylicense

License Grant
Licensor grants you (Licensee) personally a single-user, non-exclusive, non-transferable, free of charge right:

To install the Model & Software on computers owned, leased or otherwise controlled by you and/or your organization;
To use the Model & Software for the sole purpose of performing non-commercial scientific research, non-commercial
education, or non-commercial artistic projects;
Any other use, in particular any use for commercial purposes, is prohibited. This includes, without limitation,
incorporation in a commercial product, use in a commercial service, or production of other artifacts for
commercial purposes. The Model & Software may not be reproduced, modified and/or made available in any form to
any third party without Max-Planck’s prior written permission.

The Model & Software may not be used for pornographic purposes or to generate pornographic material whether
commercial or not. This license also prohibits the use of the Model & Software to train methods/algorithms/neural
networks/etc. for commercial use of any kind. By downloading the Model & Software,
you agree not to reverse engineer it.

No Distribution
The Model & Software and the license herein granted shall not be copied, shared, distributed, re-sold, offered
for re-sale, transferred or sub-licensed in whole or in part except that you may make one copy for archive
purposes only.

Disclaimer of Representations and Warranties
You expressly acknowledge and agree that the Model & Software results from basic research, is provided “AS IS”,
may contain errors, and that any use of the Model & Software is at your sole risk. LICENSOR MAKES NO REPRESENTATIONS
OR WARRANTIES OF ANY KIND CONCERNING THE MODEL & SOFTWARE, NEITHER EXPRESS NOR IMPLIED, AND THE ABSENCE OF ANY
LEGAL OR ACTUAL DEFECTS, WHETHER DISCOVERABLE OR NOT. Specifically, and not to limit the foregoing, licensor
makes no representations or warranties (i) regarding the merchantability or fitness for a particular purpose of
the Model & Software, (ii) that the use of the Model & Software will not infringe any patents, copyrights or other
intellectual property rights of a third party, and (iii) that the use of the Model & Software will not cause any
damage of any kind to you or a third party.

Limitation of Liability
Because this Model & Software License Agreement qualifies as a donation, according to Section 521 of the German
Civil Code (Bürgerliches Gesetzbuch – BGB) Licensor as a donor is liable for intent and gross negligence only.
If the Licensor fraudulently conceals a legal or material defect, they are obliged to compensate the Licensee
for the resulting damage.

Licensor shall be liable for loss of data only up to the amount of typical recovery costs which would have
arisen had proper and regular data backup measures been taken. For the avoidance of doubt Licensor shall be
liable in accordance with the German Product Liability Act in the event of product liability. The foregoing
applies also to Licensor’s legal representatives or assistants in performance. Any further liability shall be excluded.
Patent claims generated through the usage of the Model & Software cannot be directed towards the copyright holders.
The Model & Software is provided in the state of development the licensor defines. If modified or extended by
Licensee, the Licensor makes no claims about the fitness of the Model & Software and is not responsible
for any problems such modifications cause.

No Maintenance Services
You understand and agree that Licensor is under no obligation to provide either maintenance services,
update services, notices of latent defects, or corrections of defects with regard to the Model & Software.
Licensor nevertheless reserves the right to update, modify, or discontinue the Model & Software at any time.

Defects of the Model & Software must be notified in writing to the Licensor with a comprehensible description
of the error symptoms. The notification of the defect should enable the reproduction of the error.
The Licensee is encouraged to communicate any use, results, modification or publication.

Publications using the Model & Software
You acknowledge that the Model & Software is a valuable scientific resource and agree to appropriately reference
the following paper in any publication making use of the Model & Software.

Citation:

@inproceedings{VIBE:CVPR:2020,
title = {{VIBE}: Video Inference for Human Body Pose and Shape Estimation},
author = {Kocabas, Muhammed and Athanasiou, Nikos and Black, Michael J.},
booktitle = {Computer Vision and Pattern Recognition (CVPR)},
month = jun,
year = {2020},
month_numeric = {6}
}

Commercial licensing opportunities
For commercial uses of the Software, please send email to [email protected]

This Agreement shall be governed by the laws of the Federal Republic of Germany except for the UN Sales Convention.
185 changes: 185 additions & 0 deletions custom_video/VIBE_custom/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,185 @@
# VIBE: Video Inference for Human Body Pose and Shape Estimation [CVPR-2020]
[![report](https://img.shields.io/badge/arxiv-report-red)](https://arxiv.org/abs/1912.05656) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1dFfwxZ52MN86FA6uFNypMEdFShd2euQA) [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vibe-video-inference-for-human-body-pose-and/3d-human-pose-estimation-on-3dpw)](https://paperswithcode.com/sota/3d-human-pose-estimation-on-3dpw?p=vibe-video-inference-for-human-body-pose-and)

<p float="center">
<img src="doc/assets/header_1.gif" width="49%" />
<img src="doc/assets/header_2.gif" width="49%" />
</p>

Check our YouTube videos below for more details.

| Paper Video | Qualitative Results |
|------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|
| [![PaperVideo](https://img.youtube.com/vi/rIr-nX63dUA/0.jpg)](https://www.youtube.com/watch?v=rIr-nX63dUA) | [![QualitativeResults](https://img.youtube.com/vi/fW0sIZfQcIs/0.jpg)](https://www.youtube.com/watch?v=fW0sIZfQcIs) |

<!-- <sub>Sources: left video - [https://www.youtube.com/watch?v=qlPRDVqYO74](https://www.youtube.com/watch?v=qlPRDVqYO74), right video - [https://www.youtube.com/watch?v=Opry3F6aB1I](https://www.youtube.com/watch?v=Opry3F6aB1I)
</sub> -->

> [**VIBE: Video Inference for Human Body Pose and Shape Estimation**](https://arxiv.org/abs/1912.05656),
> [Muhammed Kocabas](https://ps.is.tuebingen.mpg.de/person/mkocabas), [Nikos Athanasiou](https://ps.is.tuebingen.mpg.de/person/nathanasiou),
[Michael J. Black](https://ps.is.tuebingen.mpg.de/person/black),
> *IEEE Computer Vision and Pattern Recognition, 2020*
## Features

_**V**ideo **I**nference for **B**ody Pose and Shape **E**stimation_ (VIBE) is a video pose and shape estimation method.
It predicts the parameters of SMPL body model for each frame of an input video. Pleaser refer to our [arXiv report](https://arxiv.org/abs/1912.05656) for further details.

This implementation:

- has the demo and training code for VIBE implemented purely in PyTorch,
- can work on arbitrary videos with multiple people,
- supports both CPU and GPU inference (though GPU is way faster),
- is fast, up-to 30 FPS on a RTX2080Ti (see [this table](doc/demo.md#runtime-performance)),
- achieves SOTA results on 3DPW and MPI-INF-3DHP datasets,
- includes Temporal SMPLify implementation.
- includes the training code and detailed instruction on how to train it from scratch.
- can create an FBX/glTF output to be used with major graphics softwares.

<p float="center">
<img src="doc/assets/method_1.gif" width="49%" />
<img src="doc/assets/parkour.gif" width="49%" />
</p>

## Updates

- 05/01/2021: Windows installation tutorial is added thanks to amazing [@carlosedubarreto](https://github.com/carlosedubarreto)
- 06/10/2020: Support OneEuroFilter smoothing.
- 14/09/2020: FBX/glTF conversion script is released.

## Getting Started
VIBE has been implemented and tested on Ubuntu 18.04 with python >= 3.7. It supports both GPU and CPU inference.
If you don't have a suitable device, try running our Colab demo.

Clone the repo:
```bash
git clone https://github.com/mkocabas/VIBE.git
```

Install the requirements using `virtualenv` or `conda`:
```bash
# pip
source scripts/install_pip.sh

# conda
source scripts/install_conda.sh
```

## Running the Demo

We have prepared a nice demo code to run VIBE on arbitrary videos.
First, you need download the required data(i.e our trained model and SMPL model parameters). To do this you can just run:

```bash
source scripts/prepare_data.sh
```

Then, running the demo is as simple as:

```bash
# Run on a local video
python demo.py --vid_file sample_video.mp4 --output_folder output/ --display

# Run on a YouTube video
python demo.py --vid_file https://www.youtube.com/watch?v=wPZP8Bwxplo --output_folder output/ --display
```

Refer to [`doc/demo.md`](doc/demo.md) for more details about the demo code.

Sample demo output with the `--sideview` flag:

<p float="left">
<img src="doc/assets/sample_video.gif" width="30%" />
</p>

### FBX and glTF output (New Feature!)
We provide a script to convert VIBE output to standalone FBX/glTF files to be used in 3D graphics tools like
Blender, Unity etc. You need to follow steps below to be able to run the conversion script.

- You need to download FBX files for SMPL body model
- Go to [SMPL website](https://smpl.is.tue.mpg.de/) and create an account.
- Download the Unity-compatible FBX file through the [link](https://psfiles.is.tuebingen.mpg.de/downloads/smpl/SMPL_unity_v-1-0-0-zip)
- Unzip the contents and locate them `data/SMPL_unity_v.1.0.0`.
- Install Blender python API
- Note that we tested our script with Blender v2.8.0 and v2.8.3.
- Run the command below to convert VIBE output to FBX:
```
python lib/utils/fbx_output.py \
--input output/sample_video/vibe_output.pkl \
--output output/sample_video/fbx_output.fbx \ # specify the file extension as *.glb for glTF
--fps_source 30 \
--fps_target 30 \
--gender <male or female> \
--person_id <tracklet id from VIBE output>
```
### Windows Installation Tutorial

You can follow the instructions provided by [@carlosedubarreto](https://github.com/carlosedubarreto) to install and run VIBE on a Windows machine:

- VIBE windows installation tutorial: https://youtu.be/3qhs5IRJ1LI
- FBX conversion: https://youtu.be/w1biKeiQThY
- Helper github repo: https://github.com/carlosedubarreto/vibe_win_install

## Google Colab
If you do not have a suitable environment to run this project then you could give Google Colab a try.
It allows you to run the project in the cloud, free of charge. You may try our Colab demo using the notebook we have prepared:
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1dFfwxZ52MN86FA6uFNypMEdFShd2euQA)


## Training
Run the commands below to start training:

```shell script
source scripts/prepare_training_data.sh
python train.py --cfg configs/config.yaml
```

Note that the training datasets should be downloaded and prepared before running data processing script.
Please see [`doc/train.md`](doc/train.md) for details on how to prepare them.

## Evaluation

Here we compare VIBE with recent state-of-the-art methods on 3D pose estimation datasets. Evaluation metric is
Procrustes Aligned Mean Per Joint Position Error (PA-MPJPE) in mm.

| Models | 3DPW &#8595; | MPI-INF-3DHP &#8595; | H36M &#8595; |
|----------------|:----:|:------------:|:----:|
| SPIN | 59.2 | 67.5 | **41.1** |
| Temporal HMR | 76.7 | 89.8 | 56.8 |
| VIBE | 56.5 | **63.4** | 41.5 |

See [`doc/eval.md`](doc/eval.md) to reproduce the results in this table or
evaluate a pretrained model.

**Correction**: Due to a mistake in dataset preprocessing, VIBE trained with 3DPW results in Table 1 of the original paper are not correct.
Besides, even though training with 3DPW guarantees better quantitative performance, it does not give good
qualitative results. ArXiv version will be updated with the corrected results.

## Citation

```bibtex
@inproceedings{kocabas2019vibe,
title={VIBE: Video Inference for Human Body Pose and Shape Estimation},
author={Kocabas, Muhammed and Athanasiou, Nikos and Black, Michael J.},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}
```

## License
This code is available for **non-commercial scientific research purposes** as defined in the [LICENSE file](LICENSE). By downloading and using this code you agree to the terms in the [LICENSE](LICENSE). Third-party datasets and software are subject to their respective licenses.


## References
We indicate if a function or script is borrowed externally inside each file. Here are some great resources we
benefit:

- Pretrained HMR and some functions are borrowed from [SPIN](https://github.com/nkolot/SPIN).
- SMPL models and layer is from [SMPL-X model](https://github.com/vchoutas/smplx).
- Some functions are borrowed from [Temporal HMR](https://github.com/akanazawa/human_dynamics).
- Some functions are borrowed from [HMR-pytorch](https://github.com/MandyMo/pytorch_HMR).
- Some functions are borrowed from [Kornia](https://github.com/kornia/kornia).
- Pose tracker is from [STAF](https://github.com/soulslicer/openpose/tree/staf).

Loading

0 comments on commit 4f5c23d

Please sign in to comment.