HandRefiner: Refining Malformed Hands in Generated Images by Diffusion-based Conditional Inpainting

News

2023.12.1 The paper is post on arxiv! The code will be released soon.

Introduction

This is the official repository of the paper HandRefiner: Refining Malformed Hands in Generated Images by Diffusion-based Conditional Inpainting

Figure 1: Stable Diffusion (first two rows) and SDXL (last row) generate malformed hands (left in each pair), e.g., incorrect number of fingers or irregular shapes, which can be effectively rectified by our HandRefiner (right in each pair).

In this study, we introduce a lightweight post-processing solution called HandRefiner to correct malformed hands in generated images. HandRefiner employs a conditional inpainting approach to rectify malformed hands while leaving other parts of the image untouched. We leverage the hand mesh reconstruction model that consistently adheres to the correct number of fingers and hand shape, while also being capable of fitting the desired hand pose in the generated image. Given a generated failed image due to malformed hands, we utilize ControlNet modules to re-inject such correct hand information. Additionally, we uncover a phase transition phenomenon within ControlNet as we vary the control strength. It enables us to take advantage of more readily available synthetic data without suffering from the domain gap between realistic and synthetic hands.

Visual Results

Citation

If you find HandRefiner helpful, please consider giving this repo a star ⭐ and citing:

@misc{lu2023handrefiner,
      title={HandRefiner: Refining Malformed Hands in Generated Images by Diffusion-based Conditional Inpainting}, 
      author={Wenquan Lu and Yufei Xu and Jing Zhang and Chaoyue Wang and Dacheng Tao},
      year={2023},
      eprint={2311.17957},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}