Realtime image processing for XIMEA CB500High resolution cameras are getting more and more popular nowadays with resolution parameter ever growing due to advancement in the latest image sensor technology. The newest of them offer remarkable resolution of 50 or more Megapixel pushing the data bandwidth to the edge where it can become a bottleneck. In the past, high resolution meant slow transfer speed (low fps) which was far from ideal for a smooth video on the monitor where you expect to get at least 20-30 fps to have an output with low latency and not just a sequence of separate frames. Even the simplest task of full RAW stream transfer from high resolution image sensor to computer with maximum bit depth and at full image resolution can be complicated.
Fig.1. High resolution 48 MP camera from XIMEA with active EF-mount Camera vendor XIMEA offers a portfolio of various types of industrial cameras including the xiB series. The xiB camera line includes a model called CB500 with 48 MPix Global shutter CMOS image sensor providing 22 Fps (frames per second) at 12-bit readout or 30 Fps at 8-bit. This results in a substantial data stream where the camera like CB500 equipped with PCI-Express interface securing 20 Gbits throughput comes in handy. Real data stream of this 8K camera can reach 1550 MBytes/s. One option to handle that RAW stream is to store data to high-end SSD, yet there could be a more effective solution. It is possible to process the data on GPU in real time, to show the results on the monitor and to save the compressed color frames to conventional SSD with final data rate being less than 500 MB/s. This approach allows to solve important tasks of real-time applications:
That's why we can send RAW data from XIMEA CB500 camera directly to system memory and then copy it to NVIDIA GPU memory, thus all image processing will be done on the GPU - below there are two pipelines and corresponding benchmarks for CB500 camera on NVIDIA GeForce RTX 2080TI.
Fig.2. Application example: PCB inspection of motherboard with high resolution camera Realtime image processing on GPU for 8k camera XIMEA CB500This is not a full image processing pipeline, but an example of a common one. It includes camera calibration data (dark frame, flat field, dcp profile, lcp profile) and has an option for JPEG compression to get output bandwidth around 400-450 MB/s which should comply with conventional SSD.
Time measurements on NVIDIA GeForce RTX 2080 TI for XIMEA CB500These are benchmarks for camera application at full resolution and 12-bit output. That pipeline can be tested with Fast CinemaDNG Processor software in offline mode to tune the parameters, to check image quality and performance, and it could be implemented in real time afterwards.
If we are working with images which reside on SSD, then time for GPU processing on NVIDIA GeForce RTX 2080 TI could be around 30-40 ms per frame which is close to maximum frame rate of the camera. For camera application the situation is different. In that case we can send packed data (two 12-bit values in 3 Bytes), so the file size of each frame is less. We unpack such frames on GPU, which is much faster than on CPU. We also can hide Host-to-Device transfers due to overlappping of data copy and computations. That approach could save up to ~7 ms at GeForce RTX 2080 TI, which is significant. It means that we can offer realtime processing for CB500 camera on NVIDIA GPU for the suggested pipeline. You can download source codes for that demo application for XIMEA cameras from Github. For more complicated image processing pipeline which could include bad pixels removal, denoising, intermediate color space transforms, defringe, resize, rotate, crop, sharp, histogram, parade, image and video compression, etc., the second GPU could assist in accomplishing these tasks in realtime. If there is only one GPU installed, the total time to process one frame (GPU + CPU) could reach 60-70 ms making it important to optimize both software and hardware for getting the maximum performance from the system. To create a fast multithreaded solution with XIMEA CB500 camera, high-end software and hardware (CPU, GPU, SSD) is essential. For such a workflow, JPEG store to fast SSD or NMVe is implemented in a separate CPU thread making it asynchronous and thus the time of jpg storing is not added to the total time. Video output is also implemented in a separate thread. This is the way to make as much work as possible in parallel to get maximum performance. The main idea is to divide the whole task to parts and to process them in parallel both on CPU and GPU.
Fig.3. Application example: Aerial shot from bird's eye view Preview mode for XIMEA CB500Here you can review the results of time measurements for preview mode. We need that mode in the case if it's not necessary to compress and to store processed frames. In that case image processing pipepline is very simple and the performance is higher.
It's possible to get better results by overlapping Host-to-Device transfers with computations to exclude transfer time from the benchmarks. The above benchmarks still include this time. To see that you can get with overlapping, please download source codes for that demo application from Github. Custom software design for camera applications with GPU-based processingThere are a lot of different tasks which require 12-bit camera with high image resolution and high FPS. For example, XIMEA CB500 camera is successfully utilized in applications like aerial mapping, 3D scanning, flat panel inspection (FPD), solar panel analysis, printed circuit board (PCB) examination, wide area surveillance, persistent stadium and border security, cinematography, sports and entertainment, 360 panorama, UAV and Autonomous, Unmanned vehicles, etc. The Fastvideo company offers high performance software solutions for such applications and most of them are based on GPU image processing pipeline from PRO version of Fast CinemaDNG Processor software, which is highly optimized and has digital cinema workflow inside to get excellent image quality. That solution can help you to process images with quality comparable to Adobe CameraRaw and Raw Therapee, but with much higher performance.
Fig.4. Application example: Aerial architecture - houses Additionally, the CB500 camera comes with a flat ribbon flex cable (MX500 model) making it a perfect fit for embedded vision systems or multiple camera setups. Software for that kind of complex, integrated solutions is based on Fastvideo SDK for Jetson and it's available for NVIDIA Jetson TX2, TX2i, NX/AGX Xavier and Orin hardware. With the help of Fastvideo SDK it is also possible to implement desired pipeline for any specific task for CB500 camera and similar models in a single or multi-camera system. For example, you can take advantage of 12-bit per channel JPEG compression on GPU at the end of image processing pipeline to compress and to store more information at each frame. Where to get GPU-based software for XIMEA cameras?To create your own software for XIMEA cameras, please have a look and download the following open source gpu-camera-sample project from Github. You can download source codes, add camera controls to build your high performance application with GPU image processing. You will be able to evaluate image quality, performance and latency (G2G test). From the same link you can also download binaries to work with your camera or with raw images in PGM format from SSD. That software is based on Fastvideo SDK and it could work with all NVIDIA GPUs, including Jetson. You can download FastVCR software for XIMEA cameras on Windows for evaluation. The software is offering GPU-based image processing for XIMEA cameras, so it can work at realtime for high fps settings. You can get more info about that software here and download that software for Linux and L4T (Jetson). If your task is quite different from the idea of that project, please contact us or our European partner MRTech SK, the company that provides cross-platform image processing software and customized machine vision solutions. We can offer custom software design for XIMEA cameras according to your needs. |