1 00:00:02,300 --> 00:00:05,158 Hello! My name is Julio Marco, and I’m going to present our work 2 00:00:05,158 --> 00:00:08,227 “Virtual light transport matrices for non-line-of-sight imaging.“ 3 00:00:09,467 --> 00:00:12,614 Non-line-of-sight imaging methods aim to recover information from scenes 4 00:00:12,614 --> 00:00:17,377 that are not directly visible to an observer. The typical setup consists of a target scene 5 00:00:17,377 --> 00:00:20,928 hidden behind an occluder, and a surface visible to the observer. 6 00:00:21,671 --> 00:00:26,384 To recover information about the hidden scene, NLOS methods analyze the indirect illumination 7 00:00:26,384 --> 00:00:29,757 at the visible surface produced by the hidden objects. 8 00:00:30,532 --> 00:00:35,496 In particular, active-light NLOS methods simultaneously illuminate a visible relay surface, 9 00:00:35,496 --> 00:00:40,809 and measure the resulting indirect illumination from the hidden scene at picosecond resolution. 10 00:00:40,809 --> 00:00:44,371 Time-resolved indirect illumination reveals information about the hidden scene 11 00:00:44,371 --> 00:00:47,480 that is usually lost in a regular steady-state image. 12 00:00:48,563 --> 00:00:50,053 Thanks to these measurements, these methods 13 00:00:50,053 --> 00:00:52,833 can reconstruct scenes with challenging transport effects, 14 00:00:52,833 --> 00:00:55,603 such as surfaces with complex reflectance properties, 15 00:00:55,603 --> 00:01:00,594 micron-scale geometry, or cluttered scenarios with occlusions and interreflections. 16 00:01:03,399 --> 00:01:08,120 Currently NLOS imaging is mainly focused at reconstructing the geometry behind a corner. 17 00:01:08,120 --> 00:01:12,430 However, given the variety of light transport effects that may take place, our goal is to 18 00:01:12,430 --> 00:01:16,179 provide a method for general understanding of light transport in hidden scenes. 19 00:01:17,262 --> 00:01:20,553 To enable this general understanding, we propose a framework to compute 20 00:01:20,553 --> 00:01:25,017 and analyze light transport in NLOS configurations. 21 00:01:27,000 --> 00:01:30,764 In order to achieve this, our framework combines two key components: 22 00:01:30,764 --> 00:01:34,637 Recent NLOS methods based on phasor fields, which create virtual lights and cameras 23 00:01:34,637 --> 00:01:36,676 faced towards the hidden scene, 24 00:01:36,676 --> 00:01:40,052 and traditional computation of light transport matrices of line-of-sight scenes, 25 00:01:40,052 --> 00:01:43,750 which encode global light transport between light sources and cameras. 26 00:01:46,383 --> 00:01:51,325 Phasor fields methods have recently opened a new NLOS imaging paradigm 27 00:01:51,325 --> 00:01:53,693 that turns the relay wall into a virtual lens that observes the hidden scene. 28 00:01:53,693 --> 00:01:58,660 This is done by using specific imaging functions that transform real measurements 29 00:01:58,660 --> 00:02:03,283 into a field of complex phasors that can be focused at specific planes. 30 00:02:04,104 --> 00:02:08,437 This allows to create virtual imaging devices that observe the hidden scene 31 00:02:08,437 --> 00:02:11,800 from a line-of-sight perspective and take pictures of it. 32 00:02:16,294 --> 00:02:19,657 With the ability to create virtual cameras and light sources, 33 00:02:19,657 --> 00:02:21,866 we propose to compute light transport matrices in hidden scenes 34 00:02:21,866 --> 00:02:23,689 to provide general understanding of these. 35 00:02:28,564 --> 00:02:33,123 The light transport equation defines the linear relationship between a set of light sources 36 00:02:33,123 --> 00:02:37,175 and a set of measurements by means of the light transport matrix. 37 00:02:37,175 --> 00:02:40,963 This matrix encodes the global light transport effects of the scene 38 00:02:40,963 --> 00:02:44,120 for each light source and each measurement, and is typically unknown. 39 00:02:45,504 --> 00:02:49,637 Efficient estimation of this light transport matrix has enabled a wide variety 40 00:02:49,637 --> 00:02:52,137 of fundamental applications for scene understanding 41 00:02:52,137 --> 00:02:56,867 such as separation of direct and global illumination, 42 00:02:56,867 --> 00:02:59,933 or more complex matrix probing operations 43 00:02:59,933 --> 00:03:03,954 that enable de-scattering, separation of specific ranges of illumination, 44 00:03:03,955 --> 00:03:06,930 or masking certain light paths between objects. 45 00:03:09,600 --> 00:03:13,807 These methods typically estimate the light transport matrix using projector-camera setups 46 00:03:13,807 --> 00:03:18,170 with aligned pixel arrays that simultaneously illuminate and capture the visible scene. 47 00:03:18,904 --> 00:03:21,745 In the resulting LTM, the columns represent projector pixels, 48 00:03:21,745 --> 00:03:23,379 and rows represent camera pixels. 49 00:03:24,555 --> 00:03:28,132 Similar to line-of-sight settings, our goal is to estimate light transport matrices 50 00:03:28,132 --> 00:03:32,518 in NLOS configurations to provide general understanding of hidden scenes. 51 00:03:33,606 --> 00:03:38,696 To achieve this, we couple recent phasor field imaging with the traditional LTM formulation, 52 00:03:38,696 --> 00:03:41,866 and create virtual projector-camera setups in NLOS configurations 53 00:03:41,866 --> 00:03:45,000 to estimate the virtual LTM of hidden scenes. 54 00:03:46,653 --> 00:03:50,364 Under this setup, we estimate the LTM of a voxelized hidden scene 55 00:03:50,364 --> 00:03:53,852 by systematically illuminating different voxels with a virtual projector, 56 00:03:53,852 --> 00:03:56,460 and capturing the scene response with a virtual camera. 57 00:03:58,257 --> 00:04:01,741 Let’s now have a closer look at how we compute different elements of the LTM. 58 00:04:02,898 --> 00:04:06,668 Under co-located projector and camera pixels, each diagonal element of the LTM 59 00:04:06,668 --> 00:04:11,124 corresponds to focusing projector and camera at the same location in the scene. 60 00:04:15,692 --> 00:04:20,061 This focusing requires shifting the phasors at each laser and SPAD coordinate 61 00:04:20,061 --> 00:04:22,584 by the corresponding distances to the voxel xv in the scene. 62 00:04:27,086 --> 00:04:31,211 Since the diagonal elements result from focusing illumination and camera onto the same voxel, 63 00:04:31,211 --> 00:04:35,618 the resulting magnitude corresponds mainly to the direct light at the voxel location. 64 00:04:40,521 --> 00:04:45,375 Off-diagonal elements, in turn, focus illumination and image at different locations of the scene. 65 00:04:45,375 --> 00:04:50,369 Here, xa represents the illuminated location, and xb the imaged location. 66 00:04:51,198 --> 00:04:54,050 Projector focusing requires shifting phasors on the laser domain 67 00:04:54,050 --> 00:04:56,685 with respect the distance to the illuminated voxel xa. 68 00:04:57,199 --> 00:05:02,190 In contrast, camera focusing requires a phase shift in the domain of SPAD locations, 69 00:05:02,190 --> 00:05:04,413 but with respect to the imaged voxel xb. 70 00:05:07,524 --> 00:05:11,245 Since the off-diagonal elements focus illumination and camera onto different locations, 71 00:05:11,245 --> 00:05:15,245 the resulting magnitude corresponds to indirect illumination cast by 72 00:05:15,245 --> 00:05:18,000 the illuminated location xa to the imaged location xb. 73 00:05:20,659 --> 00:05:24,520 While these operations effectively focus illumination at different locations of the scene, 74 00:05:24,520 --> 00:05:29,661 the NLOS setting poses a fundamental challenge different from real projector-camera setups. 75 00:05:29,661 --> 00:05:32,822 The size of the virtual aperture of both projector and camera 76 00:05:32,822 --> 00:05:36,776 corresponds to the area captured in the relay wall, which is typically very large. 77 00:05:38,381 --> 00:05:42,107 In consequence, the virtual lenses have a very shallow depth of field, 78 00:05:42,107 --> 00:05:45,720 and the LTM elements are prone to contain significant out-of-focus illumination 79 00:05:45,720 --> 00:05:48,246 if the lens is not focused at a geometry location. 80 00:05:49,753 --> 00:05:53,550 This defocusing happens for both projector and camera lenses. 81 00:05:59,080 --> 00:06:02,850 To mitigate these out-of-focus effects when computing the LTM 82 00:06:02,850 --> 00:06:04,162 we propose a set of gating functions 83 00:06:04,162 --> 00:06:08,316 that operate in the temporal domain when propagating light at different voxels. 84 00:06:09,461 --> 00:06:13,690 To remove out-of-focus light from diagonal elements, we gate the signal at single-bounce 85 00:06:13,690 --> 00:06:18,668 paths from the laser locations, to the voxel xv, and back to the SPAD location. 86 00:06:21,081 --> 00:06:25,000 Off-diagonal elements contain light paths of at least two bounces. 87 00:06:25,023 --> 00:06:30,371 If either projector or camera focus on an empty voxel, propagation will introduce out-of-focus illumination. 88 00:06:32,171 --> 00:06:35,868 While gating two-bounce paths can mitigate this effect, 89 00:06:35,868 --> 00:06:38,779 there may be out-of-focus paths with the same gating length, 90 00:06:38,779 --> 00:06:42,000 as shown on the left diagram for point xa prime. 91 00:06:47,680 --> 00:06:51,411 To solve this problem, we propose to use the diagonal of the LTM 92 00:06:51,411 --> 00:06:53,250 as an oracle of geometry locations. 93 00:06:54,058 --> 00:06:56,374 This way, when computing off-diagonal elements we can choose to focus 94 00:06:56,374 --> 00:07:01,915 projector and camera lenses only at non-empty locations of the hidden space. 95 00:07:03,617 --> 00:07:06,320 Combined with two-bounce temporal gating during propagation, 96 00:07:06,320 --> 00:07:10,508 our off-diagonal elements represent in-focus first-order indirect illumination. 97 00:07:13,349 --> 00:07:16,834 We now illustrate the results of our NLOS LTM computation methodology. 98 00:07:17,392 --> 00:07:21,146 We show the effects of diagonal computation in a complex office scenario, 99 00:07:21,146 --> 00:07:22,697 including single-bounce gating, 100 00:07:22,697 --> 00:07:26,057 which produces a very clean estimation of the direct illumination. 101 00:07:27,723 --> 00:07:31,252 If no gating is applied during light propagation, the resulting image is flooded 102 00:07:31,252 --> 00:07:32,837 with geometry out of focus. 103 00:07:37,270 --> 00:07:40,859 We now illustrate how our off-diagonal computation shows changes of indirect illumination 104 00:07:40,859 --> 00:07:43,597 over a mannequin when changing the material of a reflector. 105 00:07:44,307 --> 00:07:46,984 The blue mark represents the illuminated location on the reflector. 106 00:07:48,772 --> 00:07:52,667 When the reflector specularity is low, the mannequin shows indirect light 107 00:07:52,667 --> 00:07:54,856 over the entire torso, head and legs. 108 00:07:56,485 --> 00:08:00,373 When increasing the reflector specularity, it concentrates the indirect light 109 00:08:00,373 --> 00:08:02,095 towards the torso of the mannequin. 110 00:08:04,802 --> 00:08:09,520 We now show how specific columns of the LTM show indirect light at highly cluttered scenario, 111 00:08:09,520 --> 00:08:14,639 coming from the center of the lamps A, B, and C separately in the ceiling of the room. 112 00:08:14,639 --> 00:08:18,341 Every column in the LTM corresponds to a different location on the lamps. 113 00:08:18,341 --> 00:08:22,015 We can see how the profile of the indirect illumination changes as we move 114 00:08:22,015 --> 00:08:24,461 the illuminated point from one lamp to the other. 115 00:08:28,000 --> 00:08:31,630 By masking different distances from the LTM diagonal we can isolate indirect light 116 00:08:31,630 --> 00:08:35,837 within specific path lengths when illuminating the three points simultaneously. 117 00:08:35,837 --> 00:08:38,684 As we move away from the diagonal, we can see the changes 118 00:08:38,684 --> 00:08:40,378 in the resulting indirect illumination. 119 00:08:43,690 --> 00:08:47,664 Finally we demonstrate LTM computation in real NLOS scenes. 120 00:08:47,664 --> 00:08:51,813 The resulting diagonal computation yields the direct component of the hidden scene. 121 00:08:52,572 --> 00:08:56,790 While point-to-point focusing requires combining 2D arrays of both SPADs and lasers, 122 00:08:56,790 --> 00:09:00,304 available 1D SPAD arrays are also compatible under our framework. 123 00:09:00,304 --> 00:09:04,900 This yields simpler LTMs where the projector illuminates 1D vertical lines 124 00:09:04,900 --> 00:09:05,900 instead of single points, 125 00:09:05,900 --> 00:09:09,775 resulting in a vertical slice of the patch bouncing indirect light over the mannequin. 126 00:09:12,984 --> 00:09:16,872 To summarize, we defined a framework that couples the LTM formulation 127 00:09:16,872 --> 00:09:18,376 with NLOS forward propagation. 128 00:09:18,909 --> 00:09:22,706 We provide the specific imaging and gating functions for different elements of the LTM 129 00:09:22,706 --> 00:09:27,811 under the phasor field framework, addressing large aperture issues from NLOS setups. 130 00:09:27,811 --> 00:09:31,880 We demonstrate how this serves to probe the virtual LTM 131 00:09:31,880 --> 00:09:35,855 to separate light components and specific illumination path lengths. 132 00:09:38,314 --> 00:09:42,280 We hope our framework to benefit from existing line-of-sight techniques 133 00:09:42,280 --> 00:09:45,316 for LTM analysis, and apply them to the NLOS regime. 134 00:09:45,984 --> 00:09:49,553 We also hope that it will help understanding hidden scenes with more complex effects 135 00:09:49,553 --> 00:09:51,000 such as foggy environments, 136 00:09:51,000 --> 00:09:54,595 or even to handle configurations with objects behind more than one corner. 137 00:09:56,751 --> 00:09:59,383 I will be happy to discuss any questions you may have. 138 00:09:59,383 --> 00:10:00,951 Thanks for your attention.