FastVCR software for MIPI CSI2 cameras: realtime processing on NVIDIA Jetson

In today's technologically advanced world, capturing high-quality images and videos has become an integral part of various industries. One of the key components responsible for this remarkable feat is the MIPI CSI2 camera interface. MIPI CSI2 (Mobile Industry Processor Interface Camera Serial Interface 2) is a widely adopted camera interface standard designed specifically for mobile devices, but it has found its way into other applications as well. It enables seamless communication between image sensors and application processors, ensuring efficient data transfer and high-quality image capture.

MIPI CSI2 cameras offer several notable features and benefits

High Data Transfer Speed: MIPI CSI2 cameras support high-speed data transfer, allowing for the capture of high-resolution images and videos without compromising on quality.
Low Power Consumption: These cameras are designed to operate efficiently with minimum power consumption, making them ideal for battery-powered devices such as smartphones, tablets, and portable cameras.
Compact Size: MIPI CSI2 cameras are compact in size, making them suitable for integration into small form factor devices without sacrificing performance.
Scalability: The MIPI CSI2 interface supports multiple lanes, enabling scalability and accommodating different camera configurations, including multi-camera setups.

MIPI CSI2 cameras find applications in various industries

Mobile Devices: Smartphones and tablets extensively utilize MIPI CSI2 cameras to deliver high-quality photos and videos to users.
Automotive: MIPI CSI2 cameras are used in automotive applications for advanced driver-assistance systems (ADAS), surround-view cameras, and rearview cameras, enhancing safety and convenience.
Industrial and Surveillance: These cameras are employed in industrial automation, robotics, and surveillance systems, providing accurate imaging for inspection, monitoring and security.
Medical Imaging: MIPI CSI2 cameras play a crucial role in medical imaging devices, such as endoscopes and surgical cameras, enabling precise visualization during medical procedures.

MIPI CSI2 cameras have revolutionized the way we capture and process images and videos. With their high data transfer speed, low power consumption, and compact size, they have become an integral part of various industries, ranging from mobile devices to automotive and medical applications. As technology continues to advance, we can expect further enhancements in MIPI CSI2 cameras, enabling even more impressive imaging capabilities in the future.

MIPI CSI2 cameras and Jetson

By leveraging the capabilities of NVIDIA Jetson platforms and integrating them with MIPI cameras, developers can unlock a whole new level of possibilities. These combinations empower devices to perform complex computer vision tasks, including object detection, tracking, facial recognition, and much more. The real-time processing capabilities of Jetson, combined with the high-quality imaging from MIPI cameras, create a potent solution for a wide range of applications.

The Jetson platform provides developers with an extensive set of tools and libraries to develop and optimize software for Jetson devices. It includes NVIDIA's deep learning framework, CUDA, which allows developers to leverage the GPU's parallel processing capabilities to accelerate AI inferencing.

Software for MIPI CSI2 cameras

In most cases MIPI cameras are connected to NVIDIA Jetson modules and the software for processing is based on libargus library. This is the way of utilizing hardware-based ISP which is available at each Jetson. That approach has important limitations which could be overcome with GPU-based processing on Jetson. We've implemented our own ISP pipeline on CUDA and user can tune it accoding to the task to be solved.

To fully utilize the potential of MIPI cameras, we've extended our FastVCR software to take advantage of the Jetson hardware for various camera applications. This is a key approach to utilize the power of Jetson's GPU to perform real-time image processing tasks on MIPI cameras.

To work with a particular MIPI camera, we need a driver which is offering access to RAW frames and various image sensor modes of operation. This is not always the case, but it's a mandatory requirement.

Why do we need high performance software for MIPI cameras?

If we have a look at MIPI camera with IMX477 image sensor (12 MPix / 12-bit / 30 fps), we can see that real time processing performance at maximim resolution, max bit depth and at 30 fps is not a simple task. Currently it can be solved either with libargus library or with GPU-based SDK on CUDA, because camera bandwidth is quite high.

Modern MIPI cameras could offer both high resolution and high fps, so it's quite a complicated task to perform high quality real time processing without dropped frames. That is exactly what we can do with FastVCR software which is running on NVIDIA Jetson. In our pipeline we've included high quality demosaicing algorithms together with other image processing modules to take into account color science and high performance requirements.

FastVCR software for MIPI CSI2 camera control

Camera initialization (camera model, default values for frame resolution, fps, etc.)
Input bit depth and pixel format
Image resolution and region of interest (ROI)
Frame rate (FPS) and/or exposure time
Gain
Automatic exposure/gain control

16-bit ISP for MIPI CSI2 cameras on GPU

Image acquisition
Frame unpacking
Image linearization
Dark frame subtraction (FPN)
Flat-Field Correction (shading correction)
Bad pixel removal
White Balance / AWB
Adaptive Exposure and Gain control
High quality demosaicing with MG algorithm
Color correction with matrix profile or DCP profile
Highlight recovery
Exposure correction (brightness control)
Curves and Levels
NLM denoiser
Rotation to 90/180/270 degrees and flip/flop
Crop
Resize (downscale and upscale)
Rotation to an arbitrary angle
Undistortion via LCP or via calibrated maps
Sharpening (local contrast)
Gamma transform
JPEG compression and storage on SSD
Optional conversion to NV12 and h264/h265 encoding
Automatic realtime partitioning of AVI/MP4 video files to the specified file size
Built-in RTSP server for low latency video streaming and broadcasting
Realtime output to monitor

Software outputs

Video output to monitor via OpenGL in real time
Camera statistics
Histogram and Parade
Realtime processing and JPEG compression with image storage on SSD
Video encoding to MJPEG (AVI), H.264/H.265/AV1 (MP4) and storage to video container on SSD
Low-latency video streaming via RTSP (player and server/client are included)
RTSP streaming is compatible with VLC and with our own video player
Real time RAW/PGM data storage on SSD
Interoperability with third-party GPU-based SDKs
Interoperability with GPU-based AI libraries and applications at the GPU level

FastVCR performance benchmarks

That software was built with the idea of GPU image processing to be capable of working at high performance camera applications and for multicamera systems. FastVCR software can process RAW images in real time from 12-MPix MIPI camera with IMX477 image sensor (4056×3040, 12-bit, 30 fps) on the NVIDIA Jetson Orin, the pipeline includes RAW import, RAW-to-RGB transform, color correction, JPEG encoding, storage to SSD. Actually, on that hardware we need around 20 msec to process one frame on the GPU, which could be an indicator of maximum performance.

Optional image processing modules on GPU

RAW frames capture, preview and storage to SSD
Histograms for raw and processed data
Curves and levels for RAW, RGB, HSV
DCP support
3D LUT support
Geometric and affine transforms
LCP support
HDR support
Local and global tonemapping
Multiple camera support
Camera and lens calibration: FPN, FFC, DCP, undistortion

FastVCR command-line application

Quite often we need to run a software without any GUI, and it could happen in different situations. This is the case for drone applications or for any other task with remote camera control. This is also the case for any long-term unsupervised video recording and streaming.

To meet these requirements, we've developed a CLI application which has all the above features of FastVCR software and it could work without GUI. We are still able to have full control on image sensor and image processing parameters at real time. For video preview we supply our own player with RTSP client or one can use VLC instead.

All the above features are designed on the basis of our CUDA image processing SDK. Now our customers have an opportunity to utilize GPU-accelerated software components in their applications as a part of video processing pipeline.

Apart from FastVCR software, we are offering custom software design to build a solution according to customers' requirements. We can implement application-specific modes of operation to solve the task you need. Please fill the contact form below with your project info and send it to us.

Compatibility

CUDA-11.4 for NVIDIA Jetson NX, AGX Xavier, Orin
CUDA-10.2 for NVIDIA Jetson Nano, TX2, NX
MIPI CSI2 cameras

Software downloads

FastVCR software for mipi camera with IMX477 image sensor and Jetson Orin (CUDA-11.4)

FastVCR Roadmap

NLM denoiser on the GPU - done
Accelerated GPU kernels to get better performance on Jetson Nano/TX2/NX - in progress
Compatibility with YOLO v5-v8 - in progress