Abstract

Neural networks need big annotated datasets for training. However, manual annotation can be too expensive or even unfeasible for certain tasks, like multi-person 2D pose estimation with severe occlusions. A remedy for this is synthetic data with perfect ground truth. Here we explore two variations of synthetic data for this challenging problem; a dataset with purely synthetic humans, as well as a real dataset augmented with synthetic humans. We then study which approach better generalizes to real data, as well as the influence of virtual humans in the training loss. We observe that not all synthetic samples are equally informative for training, while the informative samples are different for each training stage. To exploit this observation, we employ an adversarial student-teacher framework; the teacher improves the student by providing the hardest samples for its current state as a challenge. Experiments show that this student-teacher framework outperforms all our baselines.

Downloads

This page provides the datasets for the paper Learning to Train with Synthetic Humans. Our datasets provide 2D multi-person pose annotation, camera blur parameters, the camera matrix, the depth map, gender tags, normal maps, object Id maps, the SMPL+H pose coefficients, 3D joint locations, an occlusion label for each joint (heuristic), a scale parameter, body part segmentation maps, SMPL+H shapes, global translation for each synthetic human and the z-rotation of each synthetic human. A more detailed description of these ground truth modalities can be found here.

The paper contributes 3 datasets. A purely synthetic multi-person 2D pose dataset that is composed of synthetic humans in front of random backgrounds. A version of the MPII Human Pose Dataset which is augmented with synthetic humans and a stylized version of the latter. To download these datasets please register on this website. After logging in you will find the links in the download section.

26 March 2020

• The augmented version of the MPII multi-person pose dataset (mixed dataset) has been added.
• The stylized version of the mixed dataset is now available for downlaod.

Code

The code to generate your own synthetic dataset and demo code for loading ground truth can be found here.

Trainng code of our models can be found here.

Referencing the Datasets

If you use one of the datasets please cite:

@inproceedings{Hoffmann:GCPR:2019,
  title = {Learning to Train with Synthetic Humans},
  author = {Hoffmann, David T. and Tzionas, Dimitrios and Black, Michael J. and Tang, Siyu},
  booktitle = {German Conference on Pattern Recognition (GCPR)},
  month = sep,
  year = {2019},
  url = {https://ltsh.is.tue.mpg.de},
  month_numeric = {9}
}

If you use the mixed or stylized dataset please also cite:

@inproceedings{andriluka14cvpr,
               author = {Mykhaylo Andriluka and Leonid Pishchulin and Peter Gehler and Schiele, Bernt}
               title = {2D Human Pose Estimation: New Benchmark and State of the Art Analysis},
               booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
               year = {2014},
               month = {June}
}