News

October 4, 2021: Available ICCV video presentation with closed captions and slides.
August 23, 2021: Web launched.

Abstract

The light transport matrix (LTM) is an instrumental tool in line-of-sight (LOS) imaging, describing how light interacts with the scene and enabling applications such as relighting or separation of illumination components. We introduce a framework to estimate the LTM of non-line-of-sight (NLOS) scenarios, coupling recent virtual forward light propagation models for NLOS imaging with the LOS light transport equation. We design computational projector-camera setups, and use these virtual imaging systems to estimate the transport matrix of hidden scenes. We introduce the specific illumination functions to compute the different elements of the matrix, overcoming the challenging wide-aperture conditions of NLOS setups. Our NLOS light transport matrix allows us to (re)illuminate specific locations of a hidden scene, and separate direct, first-order indirect, and higher-order indirect illumination of complex cluttered hidden scenes, similar to existing LOS techniques.

Downloads

Paper: [PDF, 6 MB] [Publisher's version*, 6 MB]
ICCV 2021 oral: [Youtube, 10 min] [MP4, 10 min, 50MB] [SRT]
ICCV 2021 slides: Oral [PDF, 2 MB], summary [PDF, < 1MB].

*Erratum: In the Publisher's version, denominator of Eq. 14 should be |x_s - x_b| instead of |x_v - x_s|.

Bibtex

@inproceedings{marco2021NLOSvLTM, author = {Marco, Julio and Jarabo, Adrian and Nam, Ji Hyun and Liu, Xiaochun and Cosculluela, Miguel Angel and Velten, Andreas and Gutierrez, Diego}, title = {Virtual Light Transport Matrices for Non-Line-of-Sight Imaging}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, month = {October}, year = {2021}, pages = {2440-2449} }

Related Bibtex

2019: Non-Line-of-Sight Imaging using Phasor Field Virtual Wave Optics

@article{liu2019non, title={Non-line-of-sight imaging using phasor-field virtual wave optics}, author={Liu, Xiaochun and Guill{\'e}n, Ib{\'o}n and La Manna, Marco and Nam, Ji Hyun and Reza, Syed Azer and Le, Toan Huu and Jarabo, Adrian and Gutierrez, Diego and Velten, Andreas}, journal={Nature}, volume={572}, number={7771}, pages={620--623}, year={2019}, publisher={Nature Publishing Group} }

2014: A Framework for Transient Rendering

@article{JaraboSIGA14, author = {Jarabo, Adrian and Marco, Julio and Mu\~{n}oz, Adolfo and Buisan, Raul and Jarosz, Wojciech and Gutierrez, Diego}, title = {A Framework for Transient Rendering}, journal = {ACM Transactions on Graphics (SIGGRAPH Asia 2014)}, volume = {33}, number = {6}, articleno = {177}, year = {2014}, }

2017: Recent Advances in Transient Imaging: A Computer Graphics and Vision Perspective

@article{Jarabo2017transient, title={Recent Advances in Transient Imaging: A Computer Graphics and Vision Perspective}, author={Jarabo, Adrian and Masia, Belen and Marco, Julio and Gutierrez, Diego}, journal={Visual Informatics}, Volume= {1}, Number={1}, year={2017} }

Acknowledgements

We would like to thank the reviewers for their feedback on the manuscript, as well as Ibón Guillén for discussions at early stages of the project. This work was funded by DARPA (REVEAL HR0011-16-C-0025), the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (CHAMELEON project, grant agreement No 682080), the Ministry of Science and Innovation of Spain (project PID2019-105004GBI00), and the University of Wisconsin-Madison Office of the Vice Chancellor for Research and Graduate Education with funding from the Wisconsin Alumni Research Foundation.