Publications

Mask3D: Mask Transformer for 3D Instance Segmentation

ICRA 2023

Modern 3D semantic instance segmentation approaches predominantly rely on specialized voting mechanisms followed by carefully designed …
PDF Code Project Source Document
Mask3D: Mask Transformer for 3D Instance Segmentation

Magic3D: High-Resolution Text-to-3D Content Creation

arXiv

DreamFusion has recently demonstrated the utility of a pre-trained text-to-image diffusion model to optimize Neural Radiance Fields …
PDF Project
Magic3D: High-Resolution Text-to-3D Content Creation

Omniverse AI Toybox

Interactive 3D Tools as Extensions in NVIDIA Omniverse

NVIDIA is using AI to improve the Content Creation process, giving the power to create virtual worlds to creators everywhere. AI Toybox …
Code
Omniverse AI Toybox

Neural Brushstroke Engine: Learning a Latent Style Space of Interactive Drawing Tools

SIGGRAPH Asia (TOG) 2022

Neural Brushstroke Engine (NeuBE) includes a GAN model that learns to mimic many drawing media styles by learning from unlabeled …
PDF Code Project
Neural Brushstroke Engine: Learning a Latent Style Space of Interactive Drawing Tools

Optimizing data collection for machine learning

NeurIPS 2022

Modern deep learning systems require huge data sets to achieve impressive performance, but there is little guidance on how much or what …
PDF Project
Optimizing data collection for machine learning

LION: Latent Point Diffusion Models for 3D Shape Generation

NeurIPS 2022

Denoising diffusion models (DDMs) have shown promising results in 3D point cloud synthesis. To advance 3D DDMs and make them useful for …
PDF Code Project
LION: Latent Point Diffusion Models for 3D Shape Generation

GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images

NeurIPS 2022

As several industries are moving towards modeling massive 3D virtual worlds, the need for content creation tools that can scale in …
PDF Code Project
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images

GENIE: Higher-Order Denoising Diffusion Solvers

NeurIPS 2022

Denoising diffusion models (DDMs) have emerged as a powerful class of generative models. A forward diffusion process slowly perturbs …
PDF Code Project
GENIE: Higher-Order Denoising Diffusion Solvers

XDGAN: Multi-Modal 3D Shape Generation in 2D Space

BMVC 2022

Generative models for 2D images has recently seen tremendous progress in quality, resolution and speed as a result of the efficiency of …
PDF Project
XDGAN: Multi-Modal 3D Shape Generation in 2D Space

Kaolin Wisp: A PyTorch Library and Engine for Neural Fields Research

Pytorch Library

NVIDIA Kaolin Wisp is a PyTorch library powered by NVIDIA Kaolin Core to work with neural fields (including NeRFs, NGLOD, instant-ngp …
Code Project Source Document
Kaolin Wisp: A PyTorch Library and Engine for Neural Fields Research

MvDeCor: Multi-view Dense Correspondence Learning for Fine-grained 3D Segmentation

ECCV 2022

We propose to utilize self-supervised techniques in the 2D domain for fine-grained 3D shape segmentation tasks. This is inspired by the …
PDF Project
MvDeCor: Multi-view Dense Correspondence Learning for Fine-grained 3D Segmentation

Language-Grounded Indoor 3D Semantic Segmentation in the Wild

ECCV 2022

Recent advances in 3D semantic segmentation with deep neural networks have shown remarkable success, with rapid performance increase on …
PDF Code Project
Language-Grounded Indoor 3D Semantic Segmentation in the Wild

Learning Smooth Neural Functions via Lipschitz Regularization

SIGGRAPH 2022

Neural implicit fields have recently emerged as a useful representation for 3D shapes. These fields are commonly represented as neural …
PDF Project
Learning Smooth Neural Functions via Lipschitz Regularization

Variable Bitrate Neural Fields

SIGGRAPH 2022

Neural approximations of scalar and vector fields, such as signed distance functions and radiance fields, have emerged as accurate, …
PDF Code Project Video
Variable Bitrate Neural Fields

ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters

SIGGRAPH 2022

The incredible feats of athleticism demonstrated by humans are made possible in part by a vast repertoire of general-purpose motor …
PDF Project
ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters

Polymorphic-GAN: Generating Aligned Samples across Multiple Domains with Learned Morph Maps

CVPR 2022 (Oral)

Modern image generative models show remarkable sample quality when trained on a single domain or class of objects. In this work, we …
PDF Project
Polymorphic-GAN: Generating Aligned Samples across Multiple Domains with Learned Morph Maps

Extracting Triangular 3D Models, Materials, and Lighting From Images

CVPR 2022 (Oral)

We present an efficient method for joint optimization of topology, materials and lighting from multi-view image observations. Unlike …
PDF Code Project
Extracting Triangular 3D Models, Materials, and Lighting From Images

Neural Fields as Learnable Kernels for 3D Reconstruction

CVPR 2022

We present Neural Kernel Fields: a novel method for reconstructing implicit 3D shapes based on a learned kernel ridge regression. Our …
PDF Project Poster Video
Neural Fields as Learnable Kernels for 3D Reconstruction

How Much More Data Do I Need? Estimating Requirements for Downstream Tasks

CVPR 2022

Given a small training data set and a learning algorithm, how much more data is necessary to reach a target validation or test …
PDF Project
How Much More Data Do I Need? Estimating Requirements for Downstream Tasks

Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior

CVPR 2022

Evaluating and improving planning for autonomous vehicles requires scalable generation of long-tail traffic scenarios. To be useful, …
PDF Project
Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior

Frame Averaging for Equivariant Shape Space Learning

CVPR 2022

The task of shape space learning involves mapping a train set of shapes to and from a latent representation space with good …
PDF Project
Frame Averaging for Equivariant Shape Space Learning

BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations

CVPR 2022

Annotating images with pixel-wise labels is a time-consuming and costly process. Recently, DatasetGAN showcased a promising alternative …
PDF Code Project
BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations

AUV-Net: Learning Aligned UV Maps for Texture Transfer and Synthesis

CVPR 2022

In this paper, we address the problem of texture representation for 3D shapes for the challenging and underexplored tasks of texture …
PDF Project
AUV-Net: Learning Aligned UV Maps for Texture Transfer and Synthesis

Score-Based Generative Modeling with Critically-Damped Langevin Diffusion

ICLR 2022 (Spotlight)

Score-based generative models (SGMs) have demonstrated remarkable synthesis quality. SGMs rely on a diffusion process that gradually …
PDF Code Project
Score-Based Generative Modeling with Critically-Damped Langevin Diffusion

Tackling the Generative Learning Trilemma with Denoising Diffusion GANs

ICLR 2022 (Spotlight)

A wide variety of deep generative models has been developed in the past decade. Yet, these models often struggle with simultaneously …
PDF Code Project
Tackling the Generative Learning Trilemma with Denoising Diffusion GANs

Domain Adversarial Training: A Game Perspective

ICLR 2022

The dominant line of work in domain adaptation has focused on learning invariant representations using domain-adversarial training. In …
PDF
Domain Adversarial Training: A Game Perspective

Low-Budget Active Learning via Wasserstein Distance: An Integer Programming Approach

ICLR 2022

Active learning is the process of training a model with limited labeled data by selecting a core subset of an unlabeled data pool to …
PDF
Low-Budget Active Learning via Wasserstein Distance: An Integer Programming Approach

Spectral Unions of Partial Deformable 3D Shapes

Eurographics 2022

Spectral geometric methods have brought revolutionary changes to the field of geometry processing – however, when the data to be …
PDF
Spectral Unions of Partial Deformable 3D Shapes

Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels

WACV 2022

The success of learning with noisy labels (LNL) methods relies heavily on the success of a warm-up stage where standard supervised …
PDF Code
Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels

ATISS: Autoregressive Transformers for Indoor Scene Synthesis

NeurIPS 2021

The ability to synthesize realistic and diverse indoor furniture layouts automatically or based on partial input, unlocks many …
PDF Code Project Poster Video
ATISS: Autoregressive Transformers for Indoor Scene Synthesis

Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis

NeurIPS 2021

We introduce DMTet, a deep 3D conditional generative model that can synthesize high-resolution 3D shapes using simple user guides such …
PDF Project Video
Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis

Score-based Generative Modeling in Latent Space

NeurIPS 2021

Score-based generative models (SGMs) have recently demonstrated impressive results in terms of both sample quality and distribution …
PDF Code Project
Score-based Generative Modeling in Latent Space

Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation

NeurIPS 2021

Autonomous driving relies on a huge volume of real-world data to be labeled to high precision. Alternative solutions seek to exploit …
PDF Project
Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation

Ultrahyperbolic Neural Networks

NeurIPS 2021 (Spotlight)

Riemannian space forms, such as the Euclidean space, sphere and hyperbolic space, are popular and powerful representation spaces in …
PDF
Ultrahyperbolic Neural Networks

DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer

NeurIPS 2021

We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiable …
PDF Project
DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer

Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence

NeurIPS 2021

Although machine learning models trained on massive data have led to breakthroughs in several areas, their deployment in …
PDF Project
Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence

EditGAN: High-Precision Semantic Image Editing

NeurIPS 2021

Generative adversarial networks (GANs) have recently found applications in image editing. However, most GAN based image editing methods …
PDF Project
EditGAN: High-Precision Semantic Image Editing

Mix3D: Out-of-Context Data Augmentation for 3D Scenes

3DV 2021 (Oral)

We present Mix3D, a data augmentation technique for segmenting large-scale 3D scenes. Since scene context helps reasoning about object …
PDF Code Project
Mix3D: Out-of-Context Data Augmentation for 3D Scenes

Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting

ICCV 2021 (Oral)

In this work, we address the problem of jointly estimating albedo, normals, depth and 3D spatially-varying lighting from a single …
PDF Project
Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting

3DStyleNet: Creating 3D Shapes with Geometric and Texture Style Variations

ICCV 2021 (Oral)

We propose a method to create plausible geometric and texture style variations of 3D objects in the quest to democratize 3D content …
PDF Project
3DStyleNet: Creating 3D Shapes with Geometric and Texture Style Variations

Vector Neurons: A General Framework for SO(3)-Equivariant Networks

ICCV 2021 (Oral)

Invariance and equivariance to the rotation group have been widely discussed in the 3D deep learning community for pointclouds. Yet …
PDF Code Project
Vector Neurons: A General Framework for SO(3)-Equivariant Networks

Physics-based Human Motion Estimation and Synthesis from Videos

ICCV 2021

Human motion synthesis is an important problem with applications in graphics, gaming and simulation environments for robotics. Existing …
PDF Project
Physics-based Human Motion Estimation and Synthesis from Videos

Self-Supervised Real-to-Sim Scene Generation

ICCV 2021

Synthetic data is emerging as a promising solution to the scalability issue of supervised deep learning, especially when real data are …
PDF Project
Self-Supervised Real-to-Sim Scene Generation

f-Domain-Adversarial Learning: Theory and Algorithms

ICML 2021

Unsupervised domain adaptation is used in many machine learning applications where, during training, a model has access to unlabeled …
PDF Code Project
f-Domain-Adversarial Learning: Theory and Algorithms

Image-Level or Object-Level? A Tale of Two Resampling Strategies for Long-Tailed Detection

ICML 2021

Training on datasets with long-tailed distributions has been challenging for major recognition tasks such as classification and …
PDF

Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Surfaces

CVPR 2021 (Oral)

Neural signed distance functions (SDFs) are emerging as an effective representation for 3D shapes. State-of-the-art methods typically …
PDF Code Project Video
Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Surfaces

DriveGAN: Towards a Controllable High-Quality Neural Simulation

CVPR 2021 (Oral)

Realistic simulators are critical for training and verifying robotics systems. While most of the contemporary simulators are …
PDF Project
DriveGAN: Towards a Controllable High-Quality Neural Simulation

DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort

CVPR 2021 (Oral)

We introduce DatasetGAN: an automatic procedure to generate massive datasets of high-quality semantically segmented images requiring …
Project
DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort

Weakly Supervised Learning of Rigid 3D Scene Flow

CVPR 2021 (Oral)

We propose a data-driven scene flow estimation algorithm exploiting the observation that many 3D scenes can be explained by a …
PDF Project
Weakly Supervised Learning of Rigid 3D Scene Flow

Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization

CVPR 2021

Training deep networks with limited labeled data while achieving a strong generalization ability is key in the quest to reduce human …
PDF Project
Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization

3DIoUMatch: Leveraging IoU Prediction for Semi-Supervised 3D Object Detection

CVPR 2021

3D object detection is an important yet demanding task that heavily relies on difficult to obtain 3D annotations. To reduce the …
PDF Project
3DIoUMatch: Leveraging IoU Prediction for Semi-Supervised 3D Object Detection

Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering

ICLR 2021 (Oral)

Differentiable rendering has paved the way to training neural networks to perform ‘inverse graphics’ tasks such as …
PDF Project
Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering

VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models

ICLR 2021 (Spotlight)

Energy-based models (EBMs) have recently been successful in representing complex distributions of small images. However, sampling from …
PDF Code
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models

gradSim: Differentiable simulation for system identification and visuomotor control

ICLR 2021

We consider the problem of estimating an object’s physical properties such as mass, friction, and elasticity directly from video …
PDF Code Project Video
gradSim: Differentiable simulation for system identification and visuomotor control

ReLMoGen: Integrating Reinforcement Learning and Motion Generation for Interactive Navigation

ICRA 2021

Many Reinforcement Learning (RL) approaches use joint control signals (positions, velocities, torques) as action space for continuous …
PDF Project
ReLMoGen: Integrating Reinforcement Learning and Motion Generation for Interactive Navigation

UniCon: Universal Neural Controller For Physics-based Character Motion

UniCon

The field of physics-based animation is gaining importance due to the increasing demand for realism in video games and films, and has …
PDF Project Video
UniCon: Universal Neural Controller For Physics-based Character Motion

Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research

Pytorch Library

Kaolin is a PyTorch library aiming to accelerate 3D deep learning research. Kaolin provides efficient implementations of differentiable …
PDF Code Project Source Document
Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research

Variational Amodal Object Completion

NeurIPS 2020

PDF

Learning Deformable Tetrahedral Meshes for 3D Reconstruction

NeurIPS 2020

3D shape representations that accommodate learning-based 3D reconstruction are an open problem in machine learning and computer …
PDF Code Project Video
Learning Deformable Tetrahedral Meshes for 3D Reconstruction

Ultrahyperbolic Representation Learning

NeurIPS 2020

In machine learning, data is usually represented in a (flat) Euclidean space where distances between points are along straight lines. …
PDF Code
Ultrahyperbolic Representation Learning

Federated Simulation or Medical Imaging

MICCAI 2020 (Nominated for the Young Scientist Award)

Labelling data is expensive and time consuming especially for domains such as medical imaging that contain volumetric imaging data and …
PDF Project
Federated Simulation or Medical Imaging

Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation

ECCV 2020

Procedural models are being widely used to synthesize scenes for graphics, gaming, and to create (labeled) synthetic datasets for ML. …
PDF Project

Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D

ECCV 2020

The goal of perception for autonomous vehicles is to extract semantic representations from multiple sensors and fuse these …
PDF Code Project Video

Learning to Simulate Dynamic Environments with GameGAN

CVPR 2020

Simulation is a crucial component of any robotic system. In order to simulate correctly, we need to write complex rules of the …
PDF Project Video
Learning to Simulate Dynamic Environments with GameGAN

Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer

NeurIPS 2019

Many machine learning models operate on images, but ignore the fact that images are 2D projections formed by 3D geometry interacting …
PDF Code Project Video
Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer

Neural Turtle Graphics for Modeling City Road Layouts

ICCV 2019 (Oral)

We propose Neural Turtle Graphics (NTG), a novel gen- erative model for spatial graphs, and demonstrate its ap- plications in modeling …
PDF Project Video
Neural Turtle Graphics for Modeling City Road Layouts

Meta Sim: Learning to Generate Synthetic Datasets

ICCV 2019 (Oral)

Training models to high-end performance requires availability of large labeled datasets, which are expensive to get. The goal of our …
PDF Code Project
Meta Sim: Learning to Generate Synthetic Datasets

Gated-SCNN Gated Shape CNNs for Semantic Segmentation

ICCV 2019

Current state-of-the-art methods for image segmentation form a dense image representation where the color, shape and texture …
PDF Code Project
Gated-SCNN Gated Shape CNNs for Semantic Segmentation

Devil is in the Edges: Learning Semantic Boundaries from Noisy Annotations

CVPR 2019 (Oral)

We tackle the problem of semantic boundary prediction, which aims to identify pixels that belong to object(class) boundaries. We notice …
PDF Code Project
Devil is in the Edges: Learning Semantic Boundaries from Noisy Annotations