On September 8, 2026, AlpsenTek delivered an invited talk at the Event‑Based Multimodal Vision: Imaging, Perception, and Understanding (EBMV) Workshop during the 19th European Conference on Computer Vision (ECCV 2026) in Malmö, Sweden.
Yan CHEN, Senior R&D Director and Algorithm Expert at AlpsenTek, presented “Multimodal Fusion Event‑Based Visual Perception for Physical AI Systems”, detailing the architecture, characteristics, and algorithmic paradigms enabled by the company’s proprietary Hybrid Vision Sensor (HVS) technology.
Redefining the “Input” of Computer Vision
As one of the top three global computer vision conferences, ECCV 2026 placed world models, autonomous driving, and embodied intelligence at the center of its agenda. This shifts a fundamental question to the forefront: What visual capabilities must next‑generation Physical AI possess, and where should foundational algorithmic innovation lead?
For decades, computer vision has operated on an implicit assumption: a camera is a frame generator. Standard image sensors capture discrete exposures at fixed frame rates, leaving algorithms to infer motion, depth, and trajectories across static snapshots. While effective in controlled settings, this discrete sampling mechanism faces a fundamental bottleneck when applied to continuous physical environments, such as robotics, autonomous vehicles, and drones.
Consider a standard 30 FPS camera, a complete frame arrives roughly every 33 ms. What happens during that interval remains unobserved. Increasing the frame rate mitigates temporal gaps but incurs significant costs: higher memory bandwidth, increased storage, heavy ISP/NPU computational load, and elevated system power consumption—largely driven by processing massive amounts of redundant, unchanged pixel data.
Event‑based vision offers a different approach. Instead of reading out an entire image periodically, an event sensor (EVS) generates events only when a pixel detects a change in brightness exceeding a predefined threshold. Each event encodes spatial position (x, y), precise timestamp (t), and polarity (p) of the change. This architecture inherently delivers microsecond‑level response, high temporal resolution, wide dynamic range, and sparse data output.
Yet, while events excel at answering “where did change occur?”, they struggle with “what exactly is there?”—and that is precisely where AlpsenTek’s HVS fusion technology comes into play.
HVS: Chip‑Level Native Fusion of Image and Event Modalities
Unlike dual-sensor setups combining a Conventional CMOS Image Sensor (CIS) with a standalone Event Camera—which suffer from spatial parallax, optical baseline discrepancies, complex temporal synchronization, and high processing overhead—the Hybrid Vision Sensor integrates frame sampling and event detection within the exact same pixel array on a single die. Through co-optimization of pixel layout, read-out circuits, and data paths, HVS natively supports three distinct operational modes:
- Image Mode: Delivers complete spatial structure, fine texture, and grayscale/color
- Event Mode – Streams high-temporal-resolution sparse motion signals;
- Hybrid Mode – Simultaneously outputs spatial frames and asynchronous event streams from the shared pixel array
Because both signals originate from identical spatial locations at the hardware level, HVS inherently guarantees spatial-temporal alignment without post-capture registration. Frames describe static scene structure and semantic context, while events track continuous dynamics and motion vectors.
Shift in Perception: From Full-Frame Acquisition to Event-Driven Distillation
HVS is designed not merely as a high-speed capture device, but as an architectural rethinking of visual data acquisition for intelligent systems:
• Traditional Pipeline: Capture full scene data → Compute on all pixels → Filter for salient changes.
• Event-Driven Sensing: Detect localized changes → Respond exclusively to variations.
By performing temporal data distillation directly at the sensing node, unchanged background pixels consume zero readout bandwidth and downstream processing power. This shift directly addresses the latency, power, and compute constraints critical to real-time closed-loop control in autonomous mobility and robotics.
Algorithmic Evolution Enabled by Dual-Modality Sensing
By supplying paired spatial-temporal inputs natively, HVS allows researchers to reframe classical computer vision problems:
1. Motion Deblurring: Traditional deblurring relies on ill-posed inverse problems to estimate intra-exposure motion. In the HVS framework, events act as high-frequency temporal priors during the exposure window, transforming motion deblurring into a well-constrained optimization problem.
2. Video Interpolation: Conventional interpolation estimates intermediate motion between discrete keyframes (I(t_0) and I(t_1)). The continuous event stream explicitly fills this temporal gap, offering a deterministic trajectory for high-frame-rate reconstruction and slow-motion generation.
Conventional frame interpolation infers intermediate frames from two known timestamps,
3. Tracking and Optical Flow: Frame-to-frame correspondence struggles under rapid displacement due to motion blur and spatial offset. HVS leverages event streams to supply high-rate spatiotemporal contours for tracking and SLAM, while leveraging periodic frames for semantic classification and drift correction.
4. Low‑Power Always‑On Sensing：Edge AI applications benefit from modal switching. Systems can remain in an ultra-low-power Event Mode to monitor environmental changes, triggering full Image Mode execution only when relevant motion or targets are detected (as demonstrated in AlpsenTek’s APX004 sensor platform).
5. From Passive Imaging to Active Vision: High temporal event sensitivity enables structured light systems to utilize temporal encoding. Modulated illumination patterns can be demodulated directly in the time domain, presenting an alternative pathway for rapid 3D reconstruction and dynamic obstacle avoidance (targeted by AlpsenTek’s APX014 platform).
Next-Generation Hardware Preview: APX006
AlpsenTek also previewed its upcoming APX006 sensor, engineered for AI-driven mobile imaging. The single-chip architecture integrates:
• A 50-megapixel APS imaging array, and
• A 3-megapixel event-sensing array
Outputting standard RGB RAW data alongside synchronized event streams, the APX006 is designed for integration into standard mobile ISP and SoC pipelines, enabling applications such as single-frame ghost-free HDR and ultra-low-power always-on vision.
Building AI-Native Perception for Physical Systems
As AI advances beyond digital applications into physical environments, sensor design is shifting from human-centric imaging parameters—such as display aesthetic and pixel count—to machine-centric efficiency metrics: response latency, data sparsity, and computational load. By co-designing spatial and temporal capture directly at the pixel level, HVS provides an AI-native sensing framework that presents explicit temporal variations directly to downstream models. As embodied AI and autonomous systems scale, the evolution of computer vision will depend not only on larger backbones and downstream model capacity, but on reimagining how raw information is captured at the very first photon.
About ECCV 2026 and the EBMV Workshop
The European Conference on Computer Vision (ECCV) is one of the world’s premier academic conferences in computer vision, held biennially alongside CVPR and ICCV. This year’s edition brought together distinguished scholars including Turing Award laureate Yann LeCun, with world models, autonomous driving, and embodied intelligence as central themes.
The concurrent Event‑Based Multimodal Vision (EBMV) workshop is a specialised forum dedicated to the intersection of event‑based vision and other sensing modalities. It serves as a key platform for researchers and engineers at the forefront of neuromorphic and event‑driven perception, covering the full chain from sensing systems and low‑level imaging to high‑level perception and understanding — an important bridge between academic innovation and industrial deployment.
ECCV 2026 / EBMV Links:
- ECCV 2026 Official Website: https://eccv2026.ecva.net/
- EBMV Workshop Website: https://eventbasemultimodalvision.github.io/
