Marcelo joined the lab as a PhD student. Welcome!

Congrats to Igor for receiving his PhD. He will be joining TRI as a research scientist.

A new joint hackathon on diffusion models.

A joint hackathon with threedle lab on the myriad ways to use CLIP embeddings.

Raymond joins TTIC as an RAP. He will be a faculty member at Purdue CS next year. Congrats Raymond!

Scale-ambiguity in 3D scene dimensions leads to magnitude-ambiguity of volumetric densities in neural radiance fields, i.e., the densities double when scene size is halved, and vice versa. We call this property alpha invariance. For NeRFs to better maintain alpha invariance, we recommend 1) parameterizing both distance and volume densities in log space, and 2) a discretization-agnostic initialization strategy to guarantee high ray transmittance. We revisit a few popular radiance field models and find that these systems use various heuristics to deal with issues arising from scene scaling. We test their behaviors and show our recipe to be more robust.

Haochen Wang*

Joshua Ahn*

Raymond A. Yeh

Greg Shakhnarovich

Scale-ambiguity in a 3D scene leads to magnitude ambiguity of volume density.
1) parameterie both distance and density in log space. 2) closed-form high transmittance offset.


Alpha Invariance: On Inverse Scaling Between Distance and Volume Density in Neural Radiance Fields

A diffusion model learns to predict a vector field of gradients. We propose to apply chain rule on the learned gradients, and back-propagate the score of a diffusion model through the Jacobian of a differentiable renderer, which we instantiate to be a voxel radiance field. This setup aggregates 2D scores at multiple camera viewpoints into a 3D score, and repurposes a pretrained 2D model for 3D data generation. We identify a technical challenge of distribution mismatch that arises in this application, and propose a novel estimation mechanism to resolve it. We run our algorithm on several off-the-shelf diffusion image generative models, including the recently released Stable Diffusion trained on the large-scale LAION dataset.

Xiaodan Du*

Jiahao Li*

Backpropagate 2D score into a differentiable renderer for 3D generation.
Estimate 2D score of a non-noisy input by convolving it with noise.


Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation

The core of our approach, Pixel Consensus Voting, is a framework for instance segmentation based on the generalized Hough transform. Pixels cast discretized, probabilistic votes for the likely regions that contain instance centroids. At the detected peaks that emerge in the voting heatmap, backprojection is applied to collect pixels and produce instance masks. Unlike a sliding window detector that densely enumerates object proposals, our method detects instances as a result of the consensus among pixel-wise votes. We implement vote aggregation and backprojection using native operators of a convolutional neural network. The discretization of centroid voting reduces the training of instance segmentation to pixel labeling, analogous and complementary to FCN-style semantic segmentation, leading to an efficient and unified architecture that jointly models things and stuff. We demonstrate the effectiveness of our pipeline on COCO and Cityscapes Panoptic Segmentation and obtain competitive results. Code will be open-sourced.

Haochen Wang

Ruotian Luo

Michael Maire

We frame pixelwise prediction for instance centroids as classification over discrete spatial cells.
Voting as Transposed Convolution; Backprojection as Filtering.


Pixel Consensus Voting for Panoptic Segmentation

We introduce DIODE, a dataset that contains thousands of diverse high resolution color images with accurate, dense, long-range depth measurements. DIODE (Dense Indoor/Outdoor DEpth) is the first public dataset to include RGBD images of indoor and outdoor scenes obtained with one sensor suite. This is in contrast to existing datasets that focus on just one domain/scene type and employ different sensors, making generalization across domains difficult. The dataset is available for download at http://diode-dataset.org

Igor Vasiljevic

Nicolas Kolkin

Shanyi Zhang

Falcon Z Dai

Andrea F. Daniele

Mohammadreza Mostajabi

Steven Basart

Matthew R Walter

DIODE (Dense Indoor and Outdoor DEpth) is a dataset that contains diverse high-resolution color images with accurate, dense, far-range depth measurements.
