Multi Camera Systems with GPU Image Processing

Multi-camera system can do the job which is impossible for a single camera. This is the way to get bigger field of view, higher resolution, better data rate, different spectral sensitivity, etc. Nowadays multi-camera systems are utilized in many industries for various applications. Such a system is able to provide much more information in comparison with a single camera. Nevertheless, multicamera systems are quite complicated to handle, and we need to take into account quite a lot to make them working properly and at full speed.


multi-camera system IR

Fig.1. Full view of multi-camera rig from Infinite Realities company

Applications for multi-camera systems

  • VR and Digital Cinema: Creation of 3D models is usually done via multicamera systems. 3D computer-animated movies is a current trend in digital cinema. The art of digital visual effects has matured to the point where it’s possible to recreate almost anything. Precision of 3D model and character animation depends on the number of cameras, their resolution, bit depth, etc. 3D modelling software relies on image series from modern multiple camera systems.
  • Medical imaging: Various imaging systems such as endoscopes, microscopes, X-ray devices or operating room monitoring cameras are used in medical applications. A major challenge is to control all these video sources in the clinical environment, to provide the most diverse cable concepts, operate the cameras, transport, transform, display, document and archive images and video.
  • Mapping: there is a task of multi-camera tracking and mapping with non-overlapping fields of view (real-time pose estimation systems). Multicam camera cluster could be constructed by using multiple perspective cameras mounted on a multirotor aerial vehicle and augmented with tracking markers to collect high-precision ground-truth motion measurements from an optical indoor positioning system.
  • Video surveillance task includes multi-camera calibration, computing the topology of camera networks, multi-camera tracking, object re-identification, multi-camera activity analysis and cooperative video surveillance both with active and static cameras. With the development of surveillance multi-camera systems, the scales and complexities of camera networks are increasing and the monitored environments are becoming much more complicated.
  • Motion tracking: multi-target tracking (MTT) and multi-object tracking (MOT) are active and challenging research topics to increase the robustness of tracking. Typical approaches for multicamera tracking assume overlapping cameras observing the same 3D scene, exploiting several real-world constraints like a common geometry. Most multiple camera approaches for multiple object tracking rely on background subtraction and project all foreground pixels onto a common ground plane.
  • Sport: high-end multi-camera broadcasting (live multi-cam streaming) for sport competitions is a global trend. Apart from that, realtime sports video analytics, replay and in-depth event sports coverage could be much more effective with multicamera systems.
  • 3D scanning and photogrammetry: modern state-of-the-art photogrammetric 3D/4D scanner systems are based on multi-camera technology. Such multicamera rigs enable rapid capturing of 2D images for creating 3D models of still or moving objects, animals and humans.
  • Robotics, UAV, self-driving units need precise info about their own position and positions of other surrounding objects which could be measured with multiple camera systems. Multi-camera system could cover full 360-degree field-of-view around the car to avoid blind spots which can otherwise lead to accidents. Standard image processing pipeline for 3D mapping, visual localization, obstacle detection, need to be changed to take full advantage of the availability of multiple cameras instead of treating each camera individually.
  • Machine vision and industrial cameras are utilized for sofisticated control in production. Such control and automation tasks just can't be solved with one camera, so multi-camera systems are necessary.
  • High speed imaging also benefits from multiple camera systems. Recordings from different angles at the same time offer much more information for scientific research, testing, diagnostics.

Architecture of multi-camera systems

To build a multi-camera system, we need to take into account many aspects of future imaging solution. We could briefly describe the main ideas which are really important to create viable multicamera system.

Cameras and accessories for multi-camera system

  • Image sensors: mono/color, NIR, spectral sensitivity, dimensions, pixel size, bit depth, frame rate, resolution, ROI, etc.
  • Additional sensors: lidar, GPS
  • Illumination: laser, strobe, synchro, structured/modulated, wavelength-specific
  • Trigger: master, master-slave, jitter
  • Optics: mount, quality, focal lengths, filters, remote control, etc.
  • Other constraints: size, weight, heat, temperature, power consumption

Hardware-related issues

  • GPU model, achitecture, memory size
  • PCIe switch: upstream PCIe x8 Gen3, downstream PCIe or USB, trigger & power
  • Frame grabber if necessary
  • Availability of interfaces: PCIe, USB3, GigE / Gig10E, CameraLink, Coax, etc.
  • Cable connections: Flat ribbon, iPass, FireFly, Coax
  • Cable length, ability for cable replication
  • Power: external power supply, battery

Software for multi-camera systems

  • OS: Windows/Linux/ARM
  • Multi-camera control software
  • Realtime data acquisition and recording
  • Image sensor control
  • Image processing pipeline for each camera
  • Realtime or offline image processing on CPU or GPU
  • Output compression
  • Storage or streaming
  • Post processing

Multi camera system configurations

  • Inside In (180/360 panorama systems)
  • Inside Out (3D Dome)
  • Camera array
  • Cluster of cameras (set of cameras at one box/body)


multi camera system 3D dome

Fig.2. 3D scanning multi-camera system from Infinite Realities company

Input calibration parameters for each camera

  • Dark frame
  • Flat-Field frame for FFC
  • Digital Camera Profile (DCP)
  • Lens Correction Profile (LCP)
  • Camera intrinsic parameters: principal point, aspect ratio, focal length, distortion center and coefficients
  • Camera extrincis parameters: translation, rotation, homography


embedded multi camera system on jetson

Fig.3. Embedded multi camera system on NVIDIA Jetson from XIMEA company

Image processing pipeline for multi-camera system usually is essentially the same as for a single camera. The difference comes from synchronization, relative positions, individual camera parameters and from high data rate. The most difficult problem is high data rate. When the amount of frames per second is increased in many times, we just can't scale CPU performance accordingly at the same hardware, as soon as even for a single camera we have quite complicated multithreaded software for image processing on CPU. Further scaling is not possible in many cases, so we need to implement another approach.

Now we can see that performance boost with GPU for image processing in camera applications could be a good solution to solve the matter. We do have excellent benchmarks for GPU-based image processing, and this is the way to scale the performance for multi-camera system. On GPU we can process raw streams from multiple cameras in realtime. We just need to supply individual calibration data for each frame to let GPU know corresponding parameters for each camera.

We've implemented quite a lot of such solutions for different camera applications. Our experience helps our customers with their projects. We have GPU-based Image & Video Processing SDK which is capable to solve such multicamera tasks.


image processing sdk on nvidia gpu

Fig.4. Fastvideo SDK to process raw images on GPU for multi-camera systems

Pipeline example for multicam system on GPU

  • RAW Image acquisition data import to GPU memory
  • Dark frame subtraction
  • Flat-Field Correction
  • White Balance
  • RAW 1D LUT (composite or per-channel)
  • Debayer
  • Denoiser (wavelet, bilateral, NLM)
  • Color correction with matrix profile
  • Exposure correction
  • Curves and levels for RGB/HSV
  • Projection together with undistortion
  • Crop and Resize
  • Sharpening
  • 3D LUT (RGB/HSV)
  • Gamma with output color space conversion
  • Image Compression to JPEG, JPEG2000, TIFF, EXR
  • Video Compression to MJPEG, MJPEG2000, H.264, H.265, AV1
  • Data storage or streaming via RTSP

Please send us your request concerning your anticipated image processing pipeline for your multi-camera system. To evaluate the quality and the performance of our solution, please download Fast CinemaDNG Processor software for Windows. At the same link you can download DNG image series for testing as well.

FastVCR software

We would also recommend testing our FastVCR camera application which is ready to test with XIMEA machine vision cameras USB3 and PCIe. We've implemented the above image processing pipeline and it's capable of working at real time. This is essentially important for camera applications to work at high data rates and without frame drops. On a good GPU it's possible to achieve processing performance in the range of 2-4 GPix/s or even more. For a simplified pipeline one could get up to 10 GPix/s for RAW to RGB conversion.


cuda software for multi camera pipeline

Fig.5. Full image processing pipeline on NVIDIA CUDA for each camera in multi-camera system

The above image processing pipeline for NVIDIA GPUs is capable to solve standard task of RAW transform to RGB (compressed or uncompressed) for off-line and realtime applications. We've utilized it for different multi-camera systems and in all cases we've got very good results. GPU is a powerful means to do realtime image processing for multicamera systems and we suggest to try our SDK for these applications.

Contact Form

This form collects your name and email. Check out our Privacy Policy on how we protect and manage your personal data.