Fast RAW Bayer Codec on CUDAThere are a lot of image-processing tasks in camera apps that need storing RAW frames without converting them to RGB. You can do this conversion at any time, even offline. This is true for a bunch of different uses, like street view and aerial imaging. So, they developed the RAW Bayer Codec to compress RAW data before storing it on an SSD. That codec has been developed to handle raw image and video data captured by cameras using a Bayer filter. This data follows a Bayer pattern (RGGB, BGGR, GRBG, or GBRG), where each pixel captures only one color channel. Using lossless or visually lossless compression techniques can keep the integrity of the original data intact, making it available for processing down the road. Applications for RAW Bayer Codec
In some cases, we can apply lossless encoding to the captured RAW images. But the compression ratio (CR) of these algorithms is usually around two, which isn't very high. Even so, this is a mathematically lossless algorithm, meaning that each image will be restored exactly as it was right after it was taken (before compression). Here are some lossless algorithms to consider: Lossless JPEG (1–16 bits), JPEG-LS (1–16 bits), and JPEG2000 (up to 16 bits). Sometimes we've got to use lossless compression, especially for scientific stuff. It's a good idea to use something called "lossy encoding" on RAW images to get a better compression ratio. But we've got to be careful and make sure that the quality of the images doesn't get too bad. That's a key part of these kinds of algorithms. Even small problems with the RAW-encoded frames could lead to major problems that you'd only notice after converting the RAW files to RGB. This could get a compression ratio way higher than two, so we've got to find a solution that hits the sweet spot between image quality and compression ratio.
Benefits of RAW Bayer Compression
Key Features of RAW Bayer Codec
We can integrate RAW Bayer Codec into your imaging system to perform the whole job fully on GPU. Please check the description of our GPU Image & Video Processing SDK to evaluate what we can do on GPU. Image Quality and CR ConsiderationsFor RAW bayer frames from conventional 12-bit image sensors from Sony, Gpixel, OnSemi, and CMOSIS, we usually achieve a compression ratio of CR~2 for lossless algorithms. This is because useful data also includes noise, which could be important in scientific applications that requires lossless data compression. Lossless encoding also has lower performance compared to lossy codecs. Applying lossy encoding means that by default we get both losses and better performance, which depends on the image content, utilized parameters, and hardware. It's important to know how to control the level of loss to store high-quality images for a particular task. In most cases, we can compress RAW frames at a CR of 5–10 times and achieve a PSNR of 50–55 dB or better. The PSNR is computed for the original RAW bayer image and the same image after processing/encoding/decoding/restoring. We understand that PSNR metrics has limited use, and we also perform visual image quality assesment to choose appropriate default processing parameters for a project. We visually compare not RAW images, but RGB which were processed with the same parameters. For example, we shouldn't see any significant differences at viewing source and processed images at 500% zoom. For applications which require the best image quality we could limit CR by the range of 4-5 and in that case we can achieve PSNR more than 60 dB. We need to emphasize that our RAW Bayer Codec with lossy encoding is not just a codec. This is a sophisticated solution with a wide range of capabilities, extending far beyond data compression and decompression. It strikes a balance between preserving raw data and performing additional image processing to improve CR and minimize image distortion. Thanks to our technology, we can achieve significantly better CR and PSNR than conventional JPEG and JPEG2000 codecs for raw bayer images. We can also customize our software to optimize results for specific applications once we understand the project requirements. GPU pipeline for RAW decoding, processing and visualization
The above pipeline is computationally demanding and it's not easy to achieve high performance for that task. For visualizing we need to reach up to 60 fps, and for most resolutions this could be done on NVIDIA GPU in realtime. We utilize our GPU Image Processing SDK for the above pipeline and it can handle data rates up to 2-3 GPix/s on GeForce RTX 4090. Such a solution is available for all NVIDIA GPUs: mobile, laptop, desktop, server. Useful links on the subject |