Hello! My name is Julio Marco, and I’m going to present our work
“Virtual light transport matrices for non-line-of-sight imaging.“
Non-line-of-sight imaging methods aim to recover information from scenes
that are not directly visible to an observer. The typical setup consists of a target scene
hidden behind an occluder, and a surface visible to the observer.
To recover information about the hidden scene, NLOS methods analyze the indirect illumination
at the visible surface produced by the hidden objects.
In particular, active-light NLOS methods simultaneously illuminate a visible relay surface,
and measure the resulting indirect illumination from the hidden scene at picosecond resolution.
Time-resolved indirect illumination reveals information about the hidden scene
that is usually lost in a regular steady-state image.
Thanks to these measurements, these methods
can reconstruct scenes with challenging transport effects,
such as surfaces with complex reflectance properties,
micron-scale geometry, or cluttered scenarios with occlusions and interreflections.
Currently NLOS imaging is mainly focused at reconstructing the geometry behind a corner.
However, given the variety of light transport effects that may take place, our goal is to
provide a method for general understanding of light transport in hidden scenes.
To enable this general understanding, we propose a framework to compute
and analyze light transport in NLOS configurations.
In order to achieve this, our framework combines two key components:
Recent NLOS methods based on phasor fields, which create virtual lights and cameras
faced towards the hidden scene,
and traditional computation of light transport matrices of line-of-sight scenes,
which encode global light transport between light sources and cameras.
Phasor fields methods have recently opened a new NLOS imaging paradigm
that turns the relay wall into a virtual lens that observes the hidden scene.
This is done by using specific imaging functions that transform real measurements
into a field of complex phasors that can be focused at specific planes.
This allows to create virtual imaging devices that observe the hidden scene
from a line-of-sight perspective and take pictures of it.
With the ability to create virtual cameras and light sources,
we propose to compute light transport matrices in hidden scenes
to provide general understanding of these.
The light transport equation defines the linear relationship between a set of light sources
and a set of measurements by means of the light transport matrix.
This matrix encodes the global light transport effects of the scene
for each light source and each measurement, and is typically unknown.
Efficient estimation of this light transport matrix has enabled a wide variety
of fundamental applications for scene understanding
such as separation of direct and global illumination,
or more complex matrix probing operations
that enable de-scattering, separation of specific ranges of illumination,
or masking certain light paths between objects.
These methods typically estimate the light transport matrix using projector-camera setups
with aligned pixel arrays that simultaneously illuminate and capture the visible scene.
In the resulting LTM, the columns represent projector pixels,
and rows represent camera pixels.
Similar to line-of-sight settings, our goal is to estimate light transport matrices
in NLOS configurations to provide general understanding of hidden scenes.
To achieve this, we couple recent phasor field imaging with the traditional LTM formulation,
and create virtual projector-camera setups in NLOS configurations
to estimate the virtual LTM of hidden scenes.
Under this setup, we estimate the LTM of a voxelized hidden scene
by systematically illuminating different voxels with a virtual projector,
and capturing the scene response with a virtual camera.
Let’s now have a closer look at how we compute different elements of the LTM.
Under co-located projector and camera pixels, each diagonal element of the LTM
corresponds to focusing projector and camera at the same location in the scene.
This focusing requires shifting the phasors at each laser and SPAD coordinate
by the corresponding distances to the voxel xv in the scene.
Since the diagonal elements result from focusing illumination and camera onto the same voxel,
the resulting magnitude corresponds mainly to the direct light at the voxel location.
Off-diagonal elements, in turn, focus illumination and image at different locations of the scene.
Here, xa represents the illuminated location, and xb the imaged location.
Projector focusing requires shifting phasors on the laser domain
with respect the distance to the illuminated voxel xa.
In contrast, camera focusing requires a phase shift in the domain of SPAD locations,
but with respect to the imaged voxel xb.
Since the off-diagonal elements focus illumination and camera onto different locations,
the resulting magnitude corresponds to indirect illumination cast by
the illuminated location xa to the imaged location xb.
While these operations effectively focus illumination at different locations of the scene,
the NLOS setting poses a fundamental challenge different from real projector-camera setups.
The size of the virtual aperture of both projector and camera
corresponds to the area captured in the relay wall, which is typically very large.
In consequence, the virtual lenses have a very shallow depth of field,
and the LTM elements are prone to contain significant out-of-focus illumination
if the lens is not focused at a geometry location.
This defocusing happens for both projector and camera lenses.
To mitigate these out-of-focus effects when computing the LTM
we propose a set of gating functions
that operate in the temporal domain when propagating light at different voxels.
To remove out-of-focus light from diagonal elements, we gate the signal at single-bounce
paths from the laser locations, to the voxel xv, and back to the SPAD location.
Off-diagonal elements contain light paths of at least two bounces.
If either projector or camera focus on an empty voxel, propagation will introduce out-of-focus illumination.
While gating two-bounce paths can mitigate this effect,
there may be out-of-focus paths with the same gating length,
as shown on the left diagram for point xa prime.
To solve this problem, we propose to use the diagonal of the LTM
as an oracle of geometry locations.
This way, when computing off-diagonal elements we can choose to focus
projector and camera lenses only at non-empty locations of the hidden space.
Combined with two-bounce temporal gating during propagation,
our off-diagonal elements represent in-focus first-order indirect illumination.
We now illustrate the results of our NLOS LTM computation methodology.
We show the effects of diagonal computation in a complex office scenario,
including single-bounce gating,
which produces a very clean estimation of the direct illumination.
If no gating is applied during light propagation, the resulting image is flooded
with geometry out of focus.
We now illustrate how our off-diagonal computation shows changes of indirect illumination
over a mannequin when changing the material of a reflector.
The blue mark represents the illuminated location on the reflector.
When the reflector specularity is low, the mannequin shows indirect light
over the entire torso, head and legs.
When increasing the reflector specularity, it concentrates the indirect light
towards the torso of the mannequin.
We now show how specific columns of the LTM show indirect light at highly cluttered scenario,
coming from the center of the lamps A, B, and C separately in the ceiling of the room.
Every column in the LTM corresponds to a different location on the lamps.
We can see how the profile of the indirect illumination changes as we move
the illuminated point from one lamp to the other.
By masking different distances from the LTM diagonal we can isolate indirect light
within specific path lengths when illuminating the three points simultaneously.
As we move away from the diagonal, we can see the changes
in the resulting indirect illumination.
Finally we demonstrate LTM computation in real NLOS scenes.
The resulting diagonal computation yields the direct component of the hidden scene.
While point-to-point focusing requires combining 2D arrays of both SPADs and lasers,
available 1D SPAD arrays are also compatible under our framework.
This yields simpler LTMs where the projector illuminates 1D vertical lines
instead of single points,
resulting in a vertical slice of the patch bouncing indirect light over the mannequin.
To summarize, we defined a framework that couples the LTM formulation
with NLOS forward propagation.
We provide the specific imaging and gating functions for different elements of the LTM
under the phasor field framework, addressing large aperture issues from NLOS setups.
We demonstrate how this serves to probe the virtual LTM
to separate light components and specific illumination path lengths.
We hope our framework to benefit from existing line-of-sight techniques
for LTM analysis, and apply them to the NLOS regime.
We also hope that it will help understanding hidden scenes with more complex effects
such as foggy environments,
or even to handle configurations with objects behind more than one corner.
I will be happy to discuss any questions you may have.
Thanks for your attention.