Update readme

YuliangXiu · YuliangXiu · Jun 13, 2022 · Apr 13, 2022 · Apr 28, 2022 · May 11, 2022
commit 6a338a4ad07015fc12919d9a7177f456ca0885cf
diff --git a/README.md b/README.md
@@ -39,6 +39,7 @@
 <br />
 
 ## News :triangular_flag_on_post:
+
 - [2022/05/16] <a href="https://github.com/Arthur151/ROMP">BEV</a> is supported as optional HPS by <a href="https://scholar.google.com/citations?hl=en&user=fkGxgrsAAAAJ">Yu Sun</a>, see [commit #060e265](https://github.com/YuliangXiu/ICON/commit/060e265bd253c6a34e65c9d0a5288c6d7ffaf68e).
 - [2022/05/15] Training code is released, please check [Training Instruction](docs/training.md).
 - [2022/04/26] <a href="https://github.com/Jeff-sjtu/HybrIK">HybrIK (SMPL)</a> is supported as optional HPS by <a href="https://jeffli.site/">Jiefeng Li</a>, see [commit #3663704](https://github.com/YuliangXiu/ICON/commit/36637046dcbb5667cdfbee3b9c91b934d4c5dd05).
@@ -89,53 +90,48 @@
 <br />
 <br />
 
-
-
-
 ## Who needs ICON?
+
 - Given an RGB image, you could get:
  - image (png): segmentation, normal images (body + cloth), overlap result (rgb + normal)
  - mesh (obj): SMPL-(X) body, reconstructed clothed human
  - video (mp4): self-rotated clothed human
 
-|![Intermediate Results](assets/intermediate_results.png)|
-|:--:|
-|*ICON's intermediate results*|
-|![Iterative Refinement](assets/refinement.gif)|
-|*ICON's SMPL Pose Refinement*|
-|![Final Results](assets/overlap1.gif)![Final Results](assets/overlap2.gif)|
-|*ICON's normal prediction + reconstructed mesh (w/o & w/ smooth)*|
+| ![Intermediate Results](assets/intermediate_results.png) |
+| :------------------------------------------------------------------------: |
+| _ICON's intermediate results_ |
+| ![Iterative Refinement](assets/refinement.gif) |
+| _ICON's SMPL Pose Refinement_ |
+| ![Final Results](assets/overlap1.gif)![Final Results](assets/overlap2.gif) |
+| _ICON's normal prediction + reconstructed mesh (w/o & w/ smooth)_ |
 
 - If you want to create a **realistic and animatable 3D clothed avatar** direclty from video / sequential images
  - fully-textured with per-vertex color
  - can be animated by SMPL pose parameters
  - natural pose-dependent clothing deformation
 
-|![ICON+SCANimate+AIST++](assets/scanimate.gif)|
-|:--:|
-|*3D Clothed Avatar, created from 400+ images using **ICON+SCANimate**, animated by AIST++*|
-
-
+| ![ICON+SCANimate+AIST++](assets/scanimate.gif) |
+| :----------------------------------------------------------------------------------------: |
+| _3D Clothed Avatar, created from 400+ images using **ICON+SCANimate**, animated by AIST++_ |
 
 ## TODO
 
-- [x] testing code and pretrained models (*self-implemented version)
+- [x] testing code and pretrained models (\*self-implemented version)
  - [x] ICON (w/ & w/o global encoder, w/ PyMAF/HybrIK/BEV/PIXIE/PARE as HPS)
- - [x] PIFu* (RGB image + predicted normal map as input)
- - [x] PaMIR* (RGB image + predicted normal map as input, w/ PyMAF/HybrIK/BEV/PIXIE/PARE as HPS)
+ - [x] PIFu\* (RGB image + predicted normal map as input)
+ - [x] PaMIR\* (RGB image + predicted normal map as input, w/ PyMAF/HybrIK/BEV/PIXIE/PARE as HPS)
 - [x] colab notebook <a href='https://colab.research.google.com/drive/1-AWeWhPvCTBX0KfMtgtMk10uPU05ihoA?usp=sharing' style='padding-left: 0.5rem;'>
  <img src='https://colab.research.google.com/assets/colab-badge.svg' alt='Google Colab'>
- </a>
+  </a>
 - [x] dataset processing
  - [x] RGB/Normal Rendering
  - [x] Visibility computing
 - [ ] training codes
  - [x] ICON (w/ & w/o global encoder, w/ SMPL-X)
- - [x] PIFu* (RGB image + predicted normal map as input)
- - [ ] PaMIR* (RGB image + predicted normal map as input, w/ voxelized SMPL)
+ - [x] PIFu\* (RGB image + predicted normal map as input)
+ - [ ] PaMIR\* (RGB image + predicted normal map as input, w/ voxelized SMPL)
 - [ ] evaluation on CAPE (Hard, Easy)
 
-
 ## Installation
 
 Please follow the [Installation Instruction](docs/installation.md) to setup all the required packages, extra data, and models.
@@ -145,8 +141,44 @@ Please follow the [Installation Instruction](docs/installation.md) to setup all
 Please follow the [Dataset Instruction](docs/dataset.md) to generate the train/val/test dataset from raw scans (THuman2.0).
 
 ## Training
+
 Please follow the [Training Instruction](docs/training.md) to train your own model using THuman2.0.
 
+## Extracting clothes from bodies
+
+It is possible to extract only clothing item from the resulting 3d meshes. The user can provide a segmentation file along with a image to get the clothes. The segmentation file must be on the following format:
+
+```json
+ {
+ "item1": {
+ "segmentation": [
+ [
+ x1,y1,x2,y2,.......
+ ],
+ [
+ .....
+ ]
+ ],
+ "category_id": 0,
+ "category_name": "{clothing description}"
+ },
+ "item2": {
+ "segmentation": [
+ [
+ x1,y1,x2,y2,.......
+ ],
+ [
+ .....
+ ]
+ ],
+ "category_id": 0,
+ "category_name": "{clothing description}"
+ }
+}
+```
+
+Where x1, y1, ... are the coordinates of the segmentation boundaries. "category_id" identifies the type of clothing (we use https://github.com/switchablenorms/DeepFashion2#data-organization). The segmentation file and the image file must have the same name. Check out the examples in the "segmentation" folder and the corresponding images in the "example" folder. To run ICON with segmentation you need to add -seg_dir {path_to_segmentation_folder} to the run command.
+
 ## Demo
 
 ```bash
@@ -167,18 +199,15 @@ python -m apps.infer -cfg ./configs/icon-nofilter.yaml -gpu 0 -in_dir ./examples
 
 ## More Qualitative Results
 
-|![Comparison](assets/compare.gif)|
-|:--:| 
-| *Comparison with other state-of-the-art methods* |
-|![extreme](assets/normal-pred.png)|
-| *Predicted normals on in-the-wild images with extreme poses* |
-
-
+| ![Comparison](assets/compare.gif) |
+| :----------------------------------------------------------: |
+| _Comparison with other state-of-the-art methods_ |
+| ![extreme](assets/normal-pred.png) |
+| _Predicted normals on in-the-wild images with extreme poses_ |
 
 <br/>
 <br/>
 
-
 ## Citation
 
 ```bibtex
@@ -208,17 +237,17 @@ Here are some great resources we benefit from:
 - [CAPE](https://github.com/qianlim/CAPE) and [THuman](https://github.com/ZhengZerong/DeepHuman/tree/master/THUmanDataset) for Dataset
 - [PyTorch3D](https://github.com/facebookresearch/pytorch3d) for Differential Rendering
 
-
 Some images used in the qualitative examples come from [pinterest.com](https://www.pinterest.com/).
 
-This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No.860768 ([CLIPE Project](https://www.clipe-itn.eu)). 
-
+This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No.860768 ([CLIPE Project](https://www.clipe-itn.eu)).
 
 ## License
+
 This code and model are available for non-commercial scientific research purposes as defined in the [LICENSE](LICENSE) file. By downloading and using the code and model you agree to the terms in the [LICENSE](LICENSE).
 
 ## Disclosure
-MJB has received research gift funds from Adobe, Intel, Nvidia, Meta/Facebook, and Amazon. MJB has financial interests in Amazon, Datagen Technologies, and Meshcapade GmbH. While MJB was a part-time employee of Amazon during this project, his research was performed solely at, and funded solely by, the Max Planck Society.
+
+MJB has received research gift funds from Adobe, Intel, Nvidia, Meta/Facebook, and Amazon. MJB has financial interests in Amazon, Datagen Technologies, and Meshcapade GmbH. While MJB was a part-time employee of Amazon during this project, his research was performed solely at, and funded solely by, the Max Planck Society.
 
 ## Contact
 

diff --git a/examples/003883.jpg b/examples/003883.jpg
diff --git a/examples/028009.jpg b/examples/028009.jpg
diff --git a/lib/dataset/TestDataset.py b/lib/dataset/TestDataset.py
@@ -40,31 +40,6 @@
 
 ImageFile.LOAD_TRUNCATED_IMAGES = True
 
-# project related libs
-sys.path.append(os.path.join(os.path.dirname(__file__), "../../"))
-
-from lib.dataset.mesh_util import get_visibility, SMPLX
-from lib.dataset.body_model import TetraSMPLModel
-from lib.common.render import Render
-from lib.common.config import cfg
-
-# for pymaf
-from lib.pymaf.models import pymaf_net
-from lib.pymaf.core import path_config
-from lib.pymaf.utils.imutils import process_image, load_segmentation
-from lib.pymaf.utils.geometry import rotation_matrix_to_angle_axis
-
-# for pare
-from lib.pare.pare.core.tester import PARETester
-
-# for pixie
-import smplx
-from lib.pixielib.pixie import PIXIE
-from lib.pixielib.utils.config import cfg as pixie_cfg
-
-# for hybrik
-from lib.hybrik.models.simple3dpose import HybrIKBaseSMPLCam
-
 
 class TestDataset():
  def __init__(self, cfg, device):

diff --git a/lib/pymaf/utils/imutils.py b/lib/pymaf/utils/imutils.py
@@ -95,9 +95,6 @@ def process_image(img_file, det, hps_type, input_res=512, device = None, seg_pat
  img_ori = load_img(img_file)
 
  in_height, in_width, _ = img_ori.shape
-
-
-
  M = aug_matrix(in_width, in_height, input_res*2, input_res*2)
 
  # from rectangle to square

diff --git a/segmentation/003883.json b/segmentation/003883.json
@@ -0,0 +1,136 @@
+{
+ "item2": {
+ "segmentation": [
+ [
+ 232.29572649572654, 34.447388414055126, 237.0364672364673,
+ 40.57084520417861, 244.9377018043686, 47.089363722697165,
+ 252.04881291547974, 49.65726495726508, 262.5179487179489,
+ 51.43504273504287, 269.233998100665, 50.447388414055204,
+ 277.5446343779678, 49.12725546058881, 285.64339981006657,
+ 46.16429249762584, 294.9273504273506, 41.22602089268754,
+ 299.9377967711301, 36.514245014245084, 304.67853751187084,
+ 30.588319088319132, 306.0612535612536, 25.65004748338083,
+ 307.64150047483383, 23.477207977207982, 311.19705603038943,
+ 24.859924026590704, 317.12298195631536, 28.020417853751216,
+ 323.04890788224134, 29.008072174738874, 331.34520417853764,
+ 30.193257359924065, 339.4439696106365, 34.7364672364673,
+ 346.75261158594515, 39.279677113010536, 350.11063627730323,
+ 44.61301044634389, 355.00541310541314, 61.422317188983875,
+ 358.9560303893638, 77.6198480531815, 362.1165242165243,
+ 90.26182336182353, 364.88195631528976, 103.29886039886063,
+ 367.6473884140552, 118.11367521367552, 369.42516619183294,
+ 129.37293447293484, 369.2324786324788, 132.60550807217476,
+ 365.6769230769232, 134.77834757834762, 359.15840455840464,
+ 138.3339031339032, 353.43000949667623, 140.70427350427357,
+ 351.4547008547009, 141.4943969610637, 351.25716999050337,
+ 138.5314339981007, 351.05963912630585, 136.75365622032294,
+ 345.7263057929725, 137.34624881291552, 337.8250712250712,
+ 139.51908831908838, 331.5040835707502, 141.09933523266864,
+ 324.7880341880341, 143.66723646723653, 322.2201329534662,
+ 146.43266856600198, 322.2201329534662, 151.5684710351378,
+ 323.0102564102563, 160.6548907882243, 324.95185185185176,
+ 173.44615384615395, 325.34691358024685, 190.23627730294416,
+ 325.93950617283946, 205.64368471035164, 325.93950617283946,
+ 215.71775878442577, 325.93950617283946, 220.06343779677147,
+ 322.7790123456789, 223.22393162393197, 315.0753086419752,
+ 228.55726495726532, 309.34691358024673, 230.53257359924066,
+ 290.1866096866098, 230.87929724596398, 263.91500474833805,
+ 229.6941120607788, 236.45821462488112, 229.29905033238373,
+ 218.48290598290572, 226.73114909781583, 202.65650522317188,
+ 224.82811016144353, 197.71823361823357, 221.07502374169044,
+ 195.15033238366567, 214.55650522317188, 195.74292497625825,
+ 200.53181386514711, 197.125641025641, 180.5811965811964,
+ 197.33285849952523, 164.68736942070285, 198.51804368471042,
+ 154.21823361823365, 198.51804368471042, 138.61329534662863,
+ 193.5797720797721, 136.4404558404558, 185.08594491927823,
+ 133.08243114909774, 177.77730294396957, 128.73675213675205,
+ 174.41927825261152, 128.53922127255453, 173.82668566001894,
+ 133.2799620132953, 174.02421652421646, 136.24292497625825,
+ 172.83903133903127, 137.03304843304838, 167.11063627730283,
+ 134.86020892687554, 159.9995251661917, 130.51452991452985,
+ 159.01187084520404, 129.1318138651471, 159.60446343779662,
+ 123.60094966761622, 162.6012345679013, 111.57578347578357,
+ 165.95925925925934, 98.53874643874646, 170.30493827160504,
+ 82.7362773029439, 173.92307692307693, 70.05584045584048,
+ 177.08357075023744, 54.84596391263053, 180.58129154795822,
+ 41.73190883190885, 183.14919278252614, 34.423266856600165,
+ 188.51623931623936, 30.279962013295354, 195.6273504273505,
+ 25.539221272554588, 201.75080721747398, 22.971320037986676,
+ 211.23228869895553, 22.37872744539408, 221.10883190883212,
+ 20.996011396011355, 224.8619183285852, 20.996011396011355,
+ 226.04710351377042, 23.56391263057927, 229.01006647673339,
+ 30.279962013295354
+ ]
+ ],
+ "category_id": 1,
+ "category_name": "short sleeve top"
+ },
+ "item1": {
+ "segmentation": [
+ [
+ 201.51804815682925, 224.7401022799914, 218.41555508203712,
+ 227.23317707223518, 236.42109524824218, 228.89522693373104,
+ 256.91971020669104, 229.44924355422967, 280.188408267633,
+ 230.2802684849776, 296.53189857234224, 230.2802684849776,
+ 313.7064138077994, 229.72625186447897, 315.32667803111013,
+ 236.8076070743661, 317.8197528233539, 240.96273172810572,
+ 318.65077775410185, 246.2258896228426, 321.4208608565949,
+ 253.15109737907534, 322.8059024078415, 265.0624547197956,
+ 324.74496057958663, 273.6497123375242, 325.9612827615598,
+ 284.4076070743661, 325.40726614106114, 299.9200724483274,
+ 324.29923290006394, 316.8175793735353, 322.0831664180694,
+ 325.9588536117625, 320.16803750266354, 336.5366716386107,
+ 316.0129128489239, 344.01589601534204, 315.18188791817596,
+ 357.86631152780745, 312.4118048156829, 368.1156190070319,
+ 308.5336884721926, 378.64193479650567, 306.31762199019806,
+ 385.29013424248905, 305.76360536969946, 398.3095248242066,
+ 305.48659705945016, 409.6668655444283, 304.94393777967184,
+ 419.3418708715109, 302.7278712976774, 427.0981035584915,
+ 301.3428297464308, 433.74630300447495, 301.3428297464308,
+ 445.3806520349459, 300.5118048156829, 461.72414233965515,
+ 299.89735776688684, 467.352311953974, 297.9582995951417,
+ 477.60161943319844, 295.1882164926486, 491.7290432559132,
+ 293.52616663115276, 497.2692094608994, 291.8641167696569,
+ 503.36339228638417, 291.3101001491583, 510.8426166631155,
+ 289.37104197741314, 513.8897080758579, 287.4433411463882,
+ 519.2043682079693, 283.0112081823993, 519.7583848284679,
+ 275.5319838056679, 519.4813765182186, 270.26882591093107,
+ 518.096334966972, 265.8366929469421, 513.6642020029831,
+ 263.62062646494763, 509.78608565949276, 264.7286597059449,
+ 498.9827615597697, 265.2826763264435, 478.76115491157015,
+ 266.1137012571914, 467.1268058810992, 266.1137012571914,
+ 454.6614319198803, 264.17464308544623, 441.64204133816276,
+ 263.06660984444903, 424.19051779245626, 261.5834221180482,
+ 407.2581504368212, 259.92137225655233, 396.45482633709815,
+ 257.1512891540592, 380.1113360323889, 257.42829746430857,
+ 359.05870445344146, 256.8742808438099, 338.56008949499255,
+ 256.8742808438099, 321.3855742595354, 254.10419774131685,
+ 320.5545493287875, 251.05710632857443, 326.6487321542723,
+ 249.39505646707858, 339.1141061154912, 249.11804815682927,
+ 356.28862135094835, 248.28702322608135, 372.3551033454083,
+ 245.23993181333896, 387.59056040912026, 243.5766673769444,
+ 409.1404219049649, 241.91461751544855, 424.92989558917554,
+ 240.52957596420202, 440.4423609631369, 238.86752610270617,
+ 455.40080971659955, 238.86752610270617, 470.91327509056083,
+ 238.31350948220754, 486.42574046452216, 238.81966759002768,
+ 501.19639889196685, 239.6506925207756, 511.168698060942,
+ 236.0495844875346, 515.6008310249309, 229.40138504155118,
+ 519.4789473684212, 221.6451523545705, 520.3099722991692,
+ 216.65900277008296, 517.2628808864267, 213.33490304709125,
+ 509.50664819944615, 208.3487534626037, 491.50110803324105,
+ 205.8556786703599, 475.1576177285318, 203.63961218836545,
+ 460.75318559556774, 203.63961218836545, 443.3016620498613,
+ 203.63961218836545, 421.9720221606645, 200.59252077562303,
+ 415.60083102493036, 197.5052844662264, 406.9847858512679,
+ 195.28921798423193, 392.0263370978052, 193.35015981248677,
+ 370.97370551885774, 190.857085020243, 343.82689111442545,
+ 187.8099936075006, 322.77425953547794, 187.0028979330919,
+ 309.89237161730256, 186.17187300234397, 291.33281483059886,
+ 188.11093117408916, 266.67907521841033, 191.15802258683155,
+ 250.3355849137011, 196.69818879181773, 234.82311953973982
+ ]
+ ],
+ "category_id": 8,
+ "category_name": "trousers"
+ }
+}