Orin SDK for ISP on GPU

NVIDIA Jetson Orin is the latest mobile processor with high-performance Ampere GPU architecture, which includes 8-core 64-bit ARM and CUDA Tensor cores, 8K video encoding and decoding. Jetson Orin brings high CPU and GPU performance to mobile heterogeneous computing. Imaging applications that require high performance in real time, low energy consumption and large amount of memory, can now be created with NVIDIA Jetson Orin. We've ported our high performance Image & Video Processing SDK to Jetson Orin to offer fast solutions for realtime imaging and video applications.

That high performance SDK for image and video processing on GPU is a set of components which correspond to standard (classic) image processing pipeline for camera applications. Jetson Orin SDK covers all image processing stages starting from raw image acquisition from a camera to JPEG and JPEG2000 compression with storage to RAM or SSD. All image processing is fully done on GPU and this leads to exceptionally high performance. That SDK is also compatible with all other Jetson modules.

Jetson Orin SDK Features

• Image acquisition from a camera, CPU RAM or GPU memory
• Dark frame subtraction (FPN)
• Flat-Field Correction (Shading Correction)
• Bad pixels removal
• White Balance
• Raw Bayer Curves and Levels (master and RGB)
• Raw Denoiser (wavelets)
• Debayer HQLI, DFPD, MG algorithms
• Denoising: wavelet-based (RAW, YCbCr, grayscale), NLM (RAW, RGB, YCbCr, grayscale), bilateral (RGB, YCbCr)
• Color Correction with matrix profile
• Highlight recovery
• Color Transforms: RGB->YCbCr, YCbCr->RGB, RGB->HSV, HSV->RGB, etc.
• Tone mapping at RGB or HSV with 1D LUTs
• 3D LUTs for RGB and HSV
• RGB Histogram and Parade (waveform monitor)
• Crop
• Rotation to 90/180/270 degrees and flip/flop
• Resize (downscale or upscale)
• Remap (rotation to arbitrary angle, affine transform, perspective transform, projection, arbitrary mapping)
• Sharpening (USM)
• Gamma (composite or per-color component)
• Realtime output via OpenGL
• JPEG codec (8/12 bits per channel)
• Raw Bayer codec for lossy encoding and decoding (8/12 bits per channel)
• JPEG2000 codec (8-16 bits per channel), lossy and lossless
• Output video encoding via H.264/H.265
• Trace
• Multiplexor
• Stream-per-thread support for better performance

Compatibility

• OS Linux4Tegra (L4T)
• CUDA-11.4
• NVIDIA Jetson Orin NX and AGX

Roadmap for Orin SDK

• Fast VCR software for XIMEA cameras (realtime raw image processing on Jetson Orin with integrated camera control and streaming) - done
• Highlights recovery - done
• HDR processing - done
• Interoperability with neural network libraries and applications - done
• High performance NLM denoiser on the GPU - done
• High quality color calibration software - done
• Chromatic aberration removal in raw domain - in progress
• Accelerated Remap - in progress