ICCV 2022 Open Access Repository

Zekun Hao; Arun Mallya; Serge Belongie; Ming-Yu Liu. Proceedings of the IEEE/CVF Internation Conference on Computer Vision (ICCV) 2021, pp. 14072-14082

Abstract

We present GANcraft, a neural unsupervised rendering framework for generating realistic images of large 3D block worlds such as the ones created by Minecraft. MEGA BLOG Our approach takes a semantic block world as input, where each block is assigned a semantic label , such as dirt, grass, or water. MEGA BLOG We model the world as an ongoing volumetric function and train our model to render view-consistent photorealistic images to a camera controlled by the user. We devised a training method that relies on adversarial as well as pseudo-ground truth training, in the absence of actual images from the block world. This is contrary to previous work on neural rendering to assist view synthesis. This requires ground truth images to establish the geometry of the scene as well as the appearance dependent on view. GANcraft allows users to control the semantics of the scene as well as output style. Comparing GANcraft to strong baselines shows the effectiveness of GANcraft in this new area of photorealistic block world synthesis.