2021 SCIEN Affiliates Meeting Poster Presentations
Index to Posters (check back for updates)
3D amplified MRI (aMRI) by Itamar Terem, Leo Dang, Allen Champagne, Javid Abderezaei, Aymeric Pionteck, Zainab Almadan, Anna-Maria Lydon, Mehmet Kurt, Miriam Scadeng, Samantha J. Holdsworth
WaveY-Net: Physics-augmented deep learning for high-speed electromagnetic simulation and optimization by Mingkun Chen, Robert Lupoiu, Chenkai Mao, Der-HanHuang, Jiaqi Jiang, Philippe Lalanne, and Jonathan A. Fan
Snapshot Imaging Hyperspectral Polarization Camera by Thaibao Phan, Evan Wang, Jonathan Fan
ACORN: Adaptive Coordinate Networks for Neural Scene Representation by Julien N. P. Martel*, David B. Lindell*, Connor Z. Lin, Eric R. Chan, Marco Monteiro, Gordon Wetzstein
Neural Lumigraph Representations by Alexander W. Bergman, Petr Kellnhofer, Lars Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, Gordon Wetzstein
Neural 3D Compression via Model Compression by Berivan Isik and Tsachy Weissman
Accurate image systems simulation with 3D scenes by Zheng Lyu, Thomas Goossens, Joyce Farrell, Brian Wandell
BACON: Band-limited Coordinate Networks for Multiscale Scene Representation by David Lindell, Dave Van Veen, Jeong Joon Park, Gordon Wetzstein
Unknown-View Non-Line-of-Sight Imaging and Tracking of Moving Objects by Mark Nishimura, Chris Metzler, David B. Lindell, Gordon Wetzstein
Neural Holography Pro: Enabling Next Generation Holographic Displays with Artificial Intelligence by Manu Gopakumar, Suyeon Choi, Yifan (Evan) Peng, Jonghyun Kim, Gordon Wetzstein
GPU enabled ray-tracing creates new opportunities for soft-prototyping imaging systems by Zhenyi Liu, Zheng Lyu, Thomas Goossens, David Cardinal, Joyce Farrell, Brian Wandell
Programmable Sensors for Real-World, High Dynamic Range Object Detection by Orr Zohar, Anthea Li, Julien Nicolas Pascal Martel, Cindy Nguyen, Gordon Wetzstein
Zemax Prison Break: Equivalent ray optics model to enable imaging system simulation of 3D scenes by Thomas Goossens, Zheng Lyu, Joyce Farrell, Brian Wandell
Learning to Solve PDE-constrained Inverse Problems with Graph Networks by Qingqing Zhao, David Lindell, Gordon Wetzstein
Fourier Domain Neural Representations for Simulation-Based Inference in Cryo-EM by Axel Levy, Julien Martel, Ariana Peck, Youssef Nashed, Frederic Poitevin, Daniel Ratner, Mike Dunne, Gordon Wetzstein
Efficient Geometry-aware 3D Generative Adversarial Networks by Eric Chan*, Connor Lin*, Matthew Chan*, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Tero Karras, Sameh Khamis, Jonathan Tremblay, Leonidas Guibas, Gordon Wetzstein
Ray Tracing at 1000's of FPS for Reinforcement Learning by Brennan Shacklett, Erik Wijmans, Dhruv Batra, Vladlen Koltun, Kayvon Fatahalian
Compact optical convolution using meta-crystal slabs by Haiwen Wang, Weiliang Jin, Cheng Guo, Nathan Zhao, Sean P Rodrigues, Shanhui Fan
Neural Sensors: Programmable Sensors for In-Pixel Intelligence by Cindy M. Nguyen, Haley So, Julien N.P. Martel
Abstracts
Title: 3D amplified MRI (aMRI)
Authors: Itamar Terem, Leo Dang, Allen Champagne, Javid Abderezaei, Aymeric Pionteck, Zainab Almadan, Anna-Maria Lydon, Mehmet Kurt, Miriam Scadeng, Samantha J. Holdsworth
Abstract: Amplified MRI (aMRI) has been introduced as a new method of detecting and visualizing pulsatile brain motions. However, the original aMRI approach was a 2D post-processing approach which does take into account motion in all three planes. Here, we improve aMRI by introducing a novel 3D aMRI approach. 3D aMRI was developed and tested for its ability to amplify sub-voxel motion in all three directions. In addition, 3D aMRI was qualitatively compared to 2D aMRI on multi-slice and 3D (volumetric) balanced steady-state free precession (BSSFP) cine data and phase contrast (PC-MRI) acquired on healthy volunteers at 3T. Finally, Optical flow maps and 4D animations were produced from volumetric 3D aMRI data. Our results suggest that 3D aMRI exhibits better image quality and fewer motion artifacts compared to 2D aMRI. The tissue motion was seen to match that of PC-MRI, with the predominant brain tissue displacement occurring in the cranial-caudal direction. Optical flow maps capture the brain tissue motion and display the physical change in shape of the ventricles by the relative movement of the surrounding tissues, and the 4D animations show the complete brain tissue and cerebrospinal fluid (CSF) motion, helping to highlight the “piston-like” motion of the ventricles.
Bio: Itamar Terem is a first year PhD student at the department of Electrical Engineering at Stanford University. His main field of research is bio-imaging and in particular Magnetic Resonance Imaging (MRI) and Optical Coherence Tomography (OCT). He is the developer of a new MRI brain technique called amplified MRI (aMRI), which enables the visualization of brain tissue motion induced by blood flow and CSF motion. In addition, he explores the potential of increasing the speed, accuracy, and invasiveness of diagnosis, particularly as they relate to skin cancer, through the development of virtual biopsy using OCT imaging and artificial intelligence (AI).
Title: WaveY-Net: Physics-augmented deep learning for high-speed electromagnetic simulation and optimization
Authors: Mingkun Chen, Robert Lupoiu, Chenkai Mao, Der-HanHuang, Jiaqi Jiang, Philippe Lalanne, and Jonathan A. Fan
Abstract: We present WaveY-Net, a a hybrid data- and physics-augmented convolutional neural network that can predict electromagnetic field distributions with ultra fast speeds and high accuracy for entire classes of dielectric photonic structures. We have demonstrated that this surrogate Maxwell solver accelerates electromagnetic simulations by nearly 10000x time. We demonstrated the effectiveness of WaveY-Net in accelerating the inverse design of nanophotonic devices by applying it into the local and global freeform optimization of metagratings.
Bio: Mingkun Chen is currently a fourth-year PhD candidate majoring in Electrical Engineering with a PhD minor in Computer Science at Stanford University. As a research assistant in Prof. Jonathan Fan’s lab, he has his research experience in nanophotonics, computational electromagnetics, and machine learning.
Title: Snapshot Imaging Hyperspectral Polarization Camera
Authors: Thaibao (Peter) Phan, Evan Wang, Jonathan Fan
Abstract: Light contains information in the form of spectrum and polarization which is highly useful in applications such as material inspection and medical diagnostics. However, most imaging systems cannot acquire this information, and those that do are expensive and slow. We develop a camera system that is capable of recording an image along with its spectral and polarization information with a single exposure. The system works in real time as the acquired image needs relatively little computational processing. This hyperspectral polarization camera is constructed from commercially-available components and is a low-cost tool for obtaining scientific image data at high speeds.
Bio: Peter is a 6th-year PhD student in Electrical Engineering.
Title: ACORN: Adaptive Coordinate Networks for Neural Scene Representation
Authors: Julien N. P. Martel*, David B. Lindell*, Connor Z. Lin, Eric R. Chan, Marco Monteiro, Gordon Wetzstein
Abstract: Neural representations have emerged as a new paradigm for applications in rendering, imaging, geometric modeling, and simulation. Compared to traditional representations such as meshes, point clouds, or volumes they can be flexibly incorporated into differentiable learning-based pipelines. While recent improvements to neural representations now make it possible to represent signals with fine details at moderate resolutions (e.g., for images and 3D shapes), adequately representing large-scale or complex scenes has proven a challenge. Current neural representations fail to accurately represent images at resolutions greater than a megapixel or 3D scenes with more than a few hundred thousand polygons. Here, we introduce a new hybrid implicit-explicit network architecture and training strategy that adaptively allocates resources during training and inference based on the local complexity of a signal of interest. Our approach uses a multiscale block-coordinate decomposition, similar to a quadtree or octree, that is optimized during training. The network architecture operates in two stages: using the bulk of the network parameters, a coordinate encoder generates a feature grid in a single forward pass. Then, hundreds or thousands of samples within each block can be efficiently evaluated using a lightweight feature decoder. With this hybrid implicit-explicit network architecture, we demonstrate the first experiments that fit gigapixel images to nearly 40 dB peak signal-to-noise ratio. Notably this represents an increase in scale of over 1000x compared to the resolution of previously demonstrated image-fitting experiments. Moreover, our approach is able to represent 3D shapes significantly faster and better than previous techniques; it reduces training times from days to hours or minutes and memory requirements by over an order of magnitude.
Bio: David B. Lindell (https://davidlindell.com) is a postdoctoral scholar at Stanford University and an incoming Assistant Professor in the Department of Computer Science at University of Toronto. He received a PhD in Electrical Engineering from Stanford University and is the recipient of the ACM SIGGRAPH 2021 outstanding dissertation honorable mention award. His research spans the areas of computational imaging, computer vision, and machine learning with a focus on new methods for active 3D imaging and physics-based machine learning.
Title: Neural Lumigraph Representations
Authors: Alexander W. Bergman, Petr Kellnhofer, Lars Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, Gordon Wetzstein
Abstract: Novel view synthesis is a long-standing problem in machine learning and computer vision. While significant progress has been made in developing neural scene representations which achieve photorealistic synthesized image quality, these representations are very slow to train and render. Inspired by neural variants of image-based rendering, we propose neural rendering pipelines which accelerate SOTA neural representation training and rendering by multiple orders of magnitude. This is accomplished by using high-capacity neural shape representations with periodic activation functions, on-surface 2D CNN-based image feature aggregation, meta-learned priors over representation parameterizations, and exporting meshes with view dependent texture information to render with traditional computer graphics pipelines. We demonstrate that that these approaches achieve similar or better novel view synthesis results in a fraction of the time that competing methods require.
Bio: Alexander W. Bergman is a fourth year PhD student in the Stanford Computational Imaging Lab. His research interests include neural rendering and 3D imaging.
Title: Neural 3D Compression via Model Compression
Authors: Berivan Isik, Tsachy Weissman
Abstract: Rendering 3D scenes requires access to arbitrary viewpoints from the scene. Storage of such a 3D scene can be done in two ways; (1) storing 2D images taken from the 3D scene that can reconstruct the scene back through interpolations, or (2) storing a representation of the 3D scene itself that already encodes views from all directions. So far, traditional 3D compression methods have focused on the first type of storage and compressed the original 2D images with image compression techniques. With this approach, the user first decodes the stored 2D images and then renders the 3D scene. However, this separated procedure is inefficient since a large amount of 2D images have to be stored. In this work, we take a different approach and compress a functional representation of 3D scenes. In particular, we introduce a method to compress 3D scenes by compressing the neural networks that represent the neural radiance fields. Our method provides more efficient storage of 3D scenes since it does not store 2D images -- which are redundant when we render the scene from the neural functional representation.
Bio: Berivan Isik is a Ph.D. student in the Department of Electrical Engineering at Stanford University, advised by Tsachy Weissman. Her research interests include machine learning, data compression, and information theory, with a current focus on learned data compression, 3D compression, neural network compression, and federated learning. She previously interned at Google Research in summer 2021 under the supervision of Phil Chou. She is a recipient of the Stanford Graduate Fellowship.
Title: Accurate image systems simulation with 3D scenes
Authors: Zheng Lyu, Thomas Goossens, Joyce Farrell, Brian Wandell
Abstract: Accurate image system simulation can reduce prototyping cost, development time, and also enable automatic dataset generation for training algorithms.. In this work, we assess the precision of an end-to-end analysis that includes a physical description of a 3D scene with multiple surfaces and complex lighting (Cornell box), proprietary optics, and a Sony image sensor. The validation assesses accuracy by comparing pixel values of real and simulated camera images.
Bio: Zheng Lyu is a PhD student in the Department of Electrical Engineering at Stanford University, advised by Prof. Brian Wandell and Dr. Joyce Farrell. His research interests focus on physically based imaging system full pipeline simulation for consumer photography and medical imaging.
Title: BACON: Band-limited Coordinate Networks for Multiscale Scene Representation
Authors: David Lindell, Dave Van Veen, Jeong Joon Park, Gordon Wetzstein
Abstract: Coordinate-based networks have emerged as a powerful tool for 3D representation and scene reconstruction. These networks are trained to map continuous input coordinates to the value of a signal at each point. Still, current architectures are black boxes: their spectral characteristics cannot be easily analyzed, and their behavior at unsupervised points is difficult to predict. Moreover, these networks are typically trained to represent a signal at a single scale, and so naive downsampling or upsampling results in artifacts. We introduce band-limited coordinate networks (BACON), a network architecture with an analytical Fourier spectrum. BACON has predictable behavior at unsupervised points, can be designed based on the spectral characteristics of the represented signal, and can represent signals at multiple scales without explicit supervision. We demonstrate that BACON outperforms alternative approaches for multiscale neural representation of images, radiance fields, and 3D scenes using signed distance functions in terms of interpretability and quality.
Bio: Dave is a first-year PhD student in Electrical Engineering excited about designing machine learning algorithms to improve signal reconstruction. Prior to his time at Stanford, Dave spent two years at a Bay Area start-up developing medical imaging algorithms for clinical deployment.
Title: Unknown-View Non-Line-of-Sight Imaging and Tracking of Moving Objects
Authors: Mark Nishimura, Chris Metzler, David B. Lindell, Gordon Wetzstein
Abstract: Non-line-of-sight (NLOS) imaging and tracking is an emerging technology that allows the shape or position of objects around corners or behind diffusers to be recovered from transient, time-of-flight measurements. However, existing NLOS approaches require the imaging system to scan a large area on a visible surface, where the indirect light paths of hidden objects are sampled. In many applications, such as robotic vision or autonomous driving, optical access to a large scanning area may not be available, which severely limits the practicality of existing NLOS techniques. Here, we propose a new approach that captures a sequence of transient measurements from a single optical path. Assuming that the hidden object of interest moves during the acquisition time, we effectively capture a series of time-resolved projections of the object's shape from unknown viewpoints. We derive inverse methods for jointly recovering the object's location and shape, and show successful experimental results with a prototype imaging system.
Bio: Mark Nishimura is a 4th year PhD student in the Stanford Computational Imaging Lab. His interests include machine learning (especially deep learning), time-of-flight imaging, and 3D imaging. His past projects range from RGB-SPAD sensor fusion, uncertainty estimation for semantic segmentation, and intrinsic image decomposition.
Title: Neural Holography Pro: Enabling Next Generation Holographic Displays with Artificial Intelligence
Authors: Manu Gopakumar, Suyeon Choi, Yifan (Evan) Peng, Jonghyun Kim, Gordon Wetzstein
Abstract: We will describe a new class of computational display modality, Neural Holography, which applies the unique combination of machine intelligence and physics to solve long-standing problems of computer-generated holography. Our approach frames several holographic display architectures that leverage the advantages of camera-in-the-loop optimization and neural network model representation to deliver full-color, high-quality holographic images. We envision this algorithmic framework can unlock the full potential of traditional displays and enable next-generation holographic VR/AR systems.
Bio: Suyeon Choi is a Ph.D. student in the Department of Electrical Engineering at Stanford University. His research interests include computational displays, holography, algorithmic frameworks for optical systems, and light transport.
Manu Gopakumar is a PhD student in the Department of Electrical Engineering at Stanford University. His research interests are centered on the co-design of optical systems and computational algorithms.
Yifan (Evan) Peng is a Postdoctoral Research Fellow at Stanford University. His research interest rides across the interdisciplinary fields of optics/photonics, computer graphics, computer vision, and AI.
Title: GPU enabled ray-tracing creates new opportunities for soft-prototyping imaging systems
Authors: Zhenyi Liu, Zheng Lyu, Thomas Goossens, David Cardinal, Joyce Farrell, Brian Wandell
Abstract: Soft-prototyping using physically based ray-tracing and sensor modeling has proven to be valuable for the design of image systems in a range of applications, including consumer photography, medical imaging, and autonomous driving. Slow rendering time has been a limitation of ray-tracing methods. In this poster, I will describe how recent advances in physically based ray tracing that incorporate GPUs dramatically reduce simulation time. The reduced rendering time enables soft-prototyping of features, including image-control algorithms for focusing and exposure, that have been impractical to simulate up to now. We will describe the features we have implemented, the computational speedup, and recent simulation results including time-of-flight and optical ray-transfer function calculations. Finally, we briefly describe empirical validations.
Bio: Zhenyi Liu is a postdoctoral scholar in the Psychology Department at Stanford University, advised by Prof. Brian Wandell and Dr. Joyce Farrell. His research interests focus on physically based imaging system full pipeline simulation for autonomous driving and consumer photography.
Title: Programmable Sensors for Real-World, High Dynamic Range Object Detection
Authors: Orr Zohar, Anthea Li, Julien Nicolas Pascal Martel, Cindy Nguyen, Gordon Wetzstein
Abstract: Real-world scenes have a dynamic range much larger than today’s imaging sensors, leading to frequent over/under exposure of different image portions. Object detection under such extreme lighting conditions is easily confounded, which challenges existing object detection pipelines [1]. The conventional cameras limited dynamic range stems from the analog to digital conversion at the sensor plane, which could be thought of as a bottleneck. We can therefore formulate the problem of object detection on real-world scenes as an encoder-bottleneck-decoder scheme. Programmable photosensors – which are sensors that can perform some computation in the sensor plane itself – lend themselves as a possible solution to this problem. By developing a differentiable model for these sensors, we aim to build a pipeline that will allow us to directly learn an optimal physical encoding- digital decoding scheme, trained specifically for object detection (see figure 1).
Bio: Orr Zohar is a 2021 Knight-Hennesy scholar and a first-year Electrical Engineering Ph.D. candidate at Stanford. Prior to coming to Stanford, Orr was a Junior Researcher at the Technion, where he leveraged his multidisciplinary background(ChemE/EE) to solve problems in medical OCT imaging, ultrafast photonics, and soft self-healing electronics. In his Ph.D., Orr aims to leverage computational imaging and machine learning approaches to solve task-specific problems in neuroscience/neurosurgical navigation.
Anthea Li is a first year CS Ph.D. student at Stanford. Her research interest lies in designing learning-based and vision algorithms for autonomous systems such as robotics and autonomous vehicles. She has worked with Professor Leo Guibas focus on the 3D vision for autonomous assemblies. She also worked with Professor Emily Whiting on AR/VR, and Professor Kate Saenko on vision-based knowledge transfer algorithms. She works to bridge the field of 2D and 3D vision to help automonous systems to better understand the world.
Title: Zemax Prison Break: Equivalent ray optics model to enable imaging system simulation of 3D scenes
Authors: Thomas Goossens, Zheng Lyu, Joyce Farrell, Brian Wandell
Abstract: Combining image sensor simulation with physically based renderers (e.g., PBRT) offers new possibilities for designing and optimizing novel imaging systems. Lens manufacturers, however, rarely disclose the lens design but often offer a black-box model that runs only within ZEMAX. To break out of this prison, we demonstrate that it is possible to construct an equivalent lens model that reproduces the ray-transfer properties of the black-box ZEMAX model. This equivalent ‘ray transfer function’ can then be implemented in any ray trace tool to enable image system simulation.
Bio: Thomas Goossens is a postdoctoral researcher at the Stanford Center for Image systems Engineering (SCIEN) working with Prof. Brian Wandell and Dr. Joyce Farrell. His research interests focus on camera simulation, hyperspectral imaging and thin-film filter optics.
Title: Learning to Solve PDE-constrained Inverse Problems with Graph Networks
Authors: Qingqing Zhao, David Lindell, Gordon Wetzstein
Abstract: Learned graph neural networks (GNNs) have recently been established as fast and accurate alternatives for principled solvers in simulating the dynamics of physical systems. In many application domains across science and engineering, however, we are not only interested in a forward simulation but also in solving inverse problems with constraints defined by a partial differential equation (PDE). Here we explore GNNs to solve such PDE-constrained inverse problems. %Given a sparse set of measurements, we are interested in recovering the initial condition or parameters of the PDE. We demonstrate that GNNs combined with autodecoder-style priors are well-suited for this task, achieving more accurate estimates of initial conditions or parameters of the wave equation and incompressible Navier Stokes equations than other learned physics approaches in significantly less time than principled solvers.
Bio: Qingqing Zhao is a second-year EE Ph.D. student supervised by Prof. Gordon Wetzstein. She is interested in learning dynamics for scientific computing and graphics-related applications.
Title: Fourier Domain Neural Representations for Simulation-Based Inference in Cryo-EM
Authors: Axel Levy, Julien Martel, Ariana Peck, Youssef Nashed, Frederic Poitevin, Daniel Ratner, Mike Dunne, Gordon Wetzstein
Abstract: Cryogenic electron microscopy (cryo-EM) provides images from different copies of the same biomolecule in arbitrary orientations. Here, we present an end-to-end unsupervised approach that jointly learns the 3D electron density of the biomolecule and the individual particle poses. Our techniques leverages the computational advantage offered by the Fourier slice theorem. Specifically, we maintain an implicit representation of the 3D electron density in Fourier domain. The approach relies on an encoder-decoder architecture where the latent space is explicitly interpreted as orientations used by a physics-based decoder simulating the Cryo-EM image formation model. We evaluate our method on simulated data and show that it is able to reconstruct a 3D volume from noisy- and CTF-corrupted 2D projections of unknown poses.
Bio: Axel Levy is a second year PhD student in EE jointly supervised by Mike Dunne at SLAC and by Gordon Wetzstein at Stanford. His research focuses on implicit representations and their use to solve inverse problems in imaging (inverse graphics, scientific imaging).
Title: Efficient Geometry-aware 3D Generative Adversarial Networks
Authors: Eric Chan*, Connor Lin*, Matthew Chan*, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Tero Karras, Sameh Khamis, Jonathan Tremblay, Leonidas Guibas, Gordon Wetzstein
Abstract: Unsupervised generation of high-quality multi-view-consistent images and 3D shapes using only collections of single-view 2D photographs has been a long-standing challenge. Existing 3D GANs are either compute-intensive or make approximations that are not 3D-consistent; the former limits quality and resolution of the generated images and the latter adversely affects multi-view consistency and shape quality. In this work, we improve the computational efficiency and image quality of 3D GANs without overly relying on these approximations. For this purpose, we introduce an expressive hybrid explicit–-implicit network architecture that, together with other design choices, synthesizes not only high-resolution multi-view-consistent images in real-time but also produces high-quality 3D geometry. By decoupling feature generation and neural rendering, our framework is able to leverage state-of-the-art 2D CNN generators, such as StyleGAN2, and inherit their efficiency and expressiveness. We demonstrate state-of-the-art 3D-aware synthesis with FFHQ and AFHQ Cats, among other experiments.
Bio: I am a first-year Ph.D. student advised by Prof. Gordon Wetzstein. My interests lie in visual scene understanding, and my goal is to teach algorithms to perceive and reason about the 3D world.
Title: Ray Tracing at 1000's of FPS for Reinforcement Learning
Authors: Brennan Shacklett, Erik Wijmans, Dhruv Batra, Vladlen Koltun, Kayvon Fatahalian
Abstract: Deep reinforcement learning (RL) is a promising new tool for training agents to perform real-world tasks such as navigation in virtual 3D environments. However, RL algorithms consume hundreds of millions of frames of agent observations during training. Rendering all these frames represents a challenging and unusual graphics workload: agent observations are typically low resolution, but the training environments often are detailed and visually complex, and end-to-end training time is directly dependent on rendering performance. To optimize for this new workload, we've built a new renderer designed from the ground up to efficiently render images for RL and machine learning tasks. The renderer leverages the inherent parallelism available in these workloads using a batched architecture that amortizes synchronization and memory costs across large batches of simultaneously rendered observations. Using this new renderer, we built a RL training system for Pointgoal navigation in the MatterPort3D and Gibson datasets that can train agents at over 10000 frames per second per-GPU, two orders of magnitude faster than prior work. Our current work focuses on bringing more realistic and dynamic rendering, specifically accurate light transport simulation, to RL training, by implementing a new high-performance path tracing backend in our renderer. By utilizing the hardware ray tracing acceleration available in modern GPUs, the path tracer is able to rapidly simulate complex lighting effects like soft shadows, global illumination, and specular reflections, which we hope will help to narrow the visual gap between training and the real world.
Bio: Brennan Shacklett is a 3rd year PhD student advised by Kayvon Fatahalian in the Graphics group of the Computer Science department. His interests lie in building high-performance rendering systems, in particular targeting workloads that have yet to be extensively explored by the graphics community.
Title: Compact optical convolution using meta-crystal slabs
Authors: Haiwen Wang, Weiliang Jin, Cheng Guo, Nathan Zhao, Sean P Rodrigues, Shanhui Fan
Abstract: Many image processing algorithms and neural networks rely heavily on convolution operations. Photonic structures have great potential in efficiently implementing those computations directly in the optical domain, and process with low energy cost and fast speed. We propose meta-crystal slabs for general implementation of optical convolution. Using an optimization approach, we designed structures to perform several low order differentiation kernels with Gaussian envelope. Our structure is very compact and work directly on the optical image fields. Our work may lead to high performance integrated optical computing hardware, and also points to the possibility of creating novel optical components with volumetric metamaterials for imaging and sensing applications.
Bio: Haiwen Wang is a fifth year PhD student in the department of Applied Physics. His research involves exploring nonlocal metasurfaces for imaging applications and topological phenomena.
Title: Neural Sensors: Programmable Sensors for In-Pixel Intelligence
Authors: Cindy M. Nguyen, Haley So, Julien N.P. Martel
Abstract: Today’s cameras are fundamentally limited by fixed exposure times. Any reconstruction performed on these captured images must work in the constraints of a single globally applied exposure, despite the dynamic range of light and dynamic motion in a scene. Spatially varying pixel exposures have been introduced as a powerful method to optically encode information on timing and irradiance, allowing us to recover more information than ever before of a scene. However, existing methods rely on heuristic coding schemes and bulky hardware to implement these functions. At the forefront of today’s camera advancements are emerging focal-plane sensor-processors, sensors which offer simultaneous sensing and processing capabilities within each pixel. These specialized sensors with their customizable pixel-wise shutter functions can be jointly optimized with neural network-based reconstruction algorithms for challenging computational photography tasks, such as HDR, video compressive sensing, and motion deblurring. We present work on approaching each of these tasks using end-to-end optimization on fully differentiable sensor models and image processing networks, and demonstrate that spatially varying pixel exposures can provide a powerful avenue for approaching these long-standing challenges in imaging.
Bio: Cindy M. Nguyen is a PhD candidate in the Stanford Computational Imaging Lab. Her research interests lie in computational photography, specifically denoising and deblurring.
Haley M. So is a PhD student in the Stanford Computational Imaging Lab, currently researching in computational photography and high dynamic range imaging.
Julien Martel (http://www.jmartel.net/) is a postdoctoral scholar in the Stanford Computational Imaging Lab. His research interests are in unconventional visual sensing and computing. More specifically, his current topics of research include the co-design of hardware and algorithms for visual sensing, the design of methods for vision sensors with in-pixel computing capabilities, and the use of novel neural representations to store and compute on visual data.
