Marcelo joined the lab as a PhD student. Welcome!

Congrats to Igor for receiving his PhD. He will be joining TRI as a research scientist.

A new joint hackathon on diffusion models.

A joint hackathon with threedle lab on the myriad ways to use CLIP embeddings.

Raymond joins TTIC as an RAP. He will be a faculty member at Purdue CS next year. Congrats Raymond!

Scale-ambiguity in 3D scene dimensions leads to magnitude-ambiguity of volumetric densities in neural radiance fields, i.e., the densities double when scene size is halved, and vice versa. We call this property alpha invariance. For NeRFs to better maintain alpha invariance, we recommend 1) parameterizing both distance and volume densities in log space, and 2) a discretization-agnostic initialization strategy to guarantee high ray transmittance. We revisit a few popular radiance field models and find that these systems use various heuristics to deal with issues arising from scene scaling. We test their behaviors and show our recipe to be more robust.

Haochen Wang*

Joshua Ahn*

Raymond A. Yeh

Greg Shakhnarovich

Scale-ambiguity in a 3D scene leads to magnitude ambiguity of volume density.
1) parameterie both distance and density in log space. 2) closed-form high transmittance offset.


Alpha Invariance: On Inverse Scaling Between Distance and Volume Density in Neural Radiance Fields

Personalized text-to-image generation models enable users to create images that depict their individual possessions in diverse scenes finding applications in various domains. To achieve the personalization capability existing methods rely on finetuning a text-to-image foundation model on a user's custom dataset which can be non-trivial for general users resource-intensive and time-consuming. Despite attempts to develope finetuning-free methods their generation quality is much lower compared to their finetuning counterparts. In this paper we propose Joint-Image Diffusion (jedi) an effective technique for learning a finetuning-free personalization model. Our key idea is to learn the joint distribution of multiple related text-image pairs that share a common subject. To facilitate learning we propose a scalable synthetic dataset generation technique. Once trained our model enables fast and easy personalization at test time by simply using reference images as input during the sampling process. Our approach does not require any expensive optimization process or additional modules and can faithfully preserve the identity represented by any number of reference images. Experimental results show that our model achieves state-of-the-art generation quality both quantitatively and qualitatively significantly outperforming both the prior finetuning-based and finetuning-free personalization baselines.

Yu Zeng

Vishal M. Patel

Haochen Wang

Xun Huang

Ting-Chun Wang

Ming-Yu Liu

Yogesh Balaji

Multi-image diffusion for personalized images.


JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation

A diffusion model learns to predict a vector field of gradients. We propose to apply chain rule on the learned gradients, and back-propagate the score of a diffusion model through the Jacobian of a differentiable renderer, which we instantiate to be a voxel radiance field. This setup aggregates 2D scores at multiple camera viewpoints into a 3D score, and repurposes a pretrained 2D model for 3D data generation. We identify a technical challenge of distribution mismatch that arises in this application, and propose a novel estimation mechanism to resolve it. We run our algorithm on several off-the-shelf diffusion image generative models, including the recently released Stable Diffusion trained on the large-scale LAION dataset.

Xiaodan Du*

Jiahao Li*

Backpropagate 2D score into a differentiable renderer for 3D generation.
Estimate 2D score of a non-noisy input by convolving it with noise.


Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation

The core of our approach, Pixel Consensus Voting, is a framework for instance segmentation based on the generalized Hough transform. Pixels cast discretized, probabilistic votes for the likely regions that contain instance centroids. At the detected peaks that emerge in the voting heatmap, backprojection is applied to collect pixels and produce instance masks. Unlike a sliding window detector that densely enumerates object proposals, our method detects instances as a result of the consensus among pixel-wise votes. We implement vote aggregation and backprojection using native operators of a convolutional neural network. The discretization of centroid voting reduces the training of instance segmentation to pixel labeling, analogous and complementary to FCN-style semantic segmentation, leading to an efficient and unified architecture that jointly models things and stuff. We demonstrate the effectiveness of our pipeline on COCO and Cityscapes Panoptic Segmentation and obtain competitive results. Code will be open-sourced.

Ruotian Luo

Michael Maire

We frame pixelwise prediction for instance centroids as classification over discrete spatial cells.
Voting as Transposed Convolution; Backprojection as Filtering.


Pixel Consensus Voting for Panoptic Segmentation

We introduce DIODE, a dataset that contains thousands of diverse high resolution color images with accurate, dense, long-range depth measurements. DIODE (Dense Indoor/Outdoor DEpth) is the first public dataset to include RGBD images of indoor and outdoor scenes obtained with one sensor suite. This is in contrast to existing datasets that focus on just one domain/scene type and employ different sensors, making generalization across domains difficult. The dataset is available for download at http://diode-dataset.org

Igor Vasiljevic

Nicolas Kolkin

Shanyi Zhang

Falcon Z Dai

Andrea F. Daniele

Mohammadreza Mostajabi

Steven Basart

Matthew R Walter

DIODE (Dense Indoor and Outdoor DEpth) is a dataset that contains diverse high-resolution color images with accurate, dense, far-range depth measurements.
