RAW Image Compression with Lossy and Lossless algorithmsThe process of compressing raw images using a lossy (non-reversible) algorithm is a very challenging task. This is due to the fact that the implications of such compression are intricate and ambiguous. Typically, we do not engage directly with raw frames; rather, we must apply a conversion to RGB in order to preview the image. In such instances, the manifestation of raw image compression artifacts occurs indirectly, thus remaining imperceptible to the naked eye. Furthermore, the spacing between pixels with the same color in a raw image is greater in comparison with an RGB image. Therefore, it is necessary to take this into consideration in order to cope with the new artifacts that are unusual for RGB images obtained after demosaicing and encoding. The following task could be formulated: the process of RAW Image Compression entails the data conversion of Bayer or monochrome image sensors, which typically operate with 12-16 bits per pixel, through the implementation of either lossless or lossy algorithms. The task could be reformulated and adapted to suit any given application, with the specific requirements of each case taken into consideration. We don't consider here an option to convert RAW to RGB and then to apply lossy encoding, we focus on direct RAW Image Compression. 
Lossless RAW Image CompressionLossless encoding is employed to minimize the file size of captured RAW images, ensuring the absence of any artifacts, but the compression ratio (CR) is usually around 2 or up to 2.5, which is not enough in many cases. Still, this is a mathematically lossless approach, meaning that each image will be restored exactly with zero loss. These are some lossless algorithms for raw image compression to consider: Lossless JPEG (1–16 bits), JPEG-LS (1–16 bits), JPEG 2000 (up to 16 bits), JPEG XS (up to 16 bits), JPEG XL (up to 16 bits), etc. In certain cases, the application of lossless compression is mandatory, particularly within the context of scientific research. Usually the performance of lossless algorithms is less in comparison with lossy, but this is a "must have" option for any solution for raw image compression. Additionally, there are some options for the utilization of lossless video codecs. Video codecs typically exhibit a higher compression ratio in comparison with the results obtained for still image compression, but they are not very fast. The lossless H.265 encoding algorithm is capable of processing color video streams; however, its application is limited to 8/10-bit color (NV12, YUV) output, and it's not working with RAW formats. This assertion holds true for the lossless AV1 codec as well, which is capable of encoding up to 10-bit YUV (4:2:0, 4:2:2, 4:4:4) but not RAW. As we see, these methods are not applicable to the most common use case, which is the raw image compression from Bayer or monochrome image sensors with bits per pixel of 12 or higher, up to 16 bits. Still, there are some cases when video codecs could be viable for such a task. Lossy RAW Image CompressionIt's a good idea to use something called "lossy encoding" on RAW images to get a better compression ratio. But we've got to be careful and make sure that the quality of the images doesn't get too bad. That's a key part of these kinds of algorithms. Even small problems with the RAW-encoded frames could lead to major problems that you'd only notice after converting the RAW files to RGB. This could get a compression ratio way higher than two, so we've got to find a solution that hits the sweet spot between image quality and compression ratio. At this juncture, it is imperative to make an important remark. The fundamental premise of the RAW image concept is inherently problematic. The term "RAW image" is often erroneously interpreted to signify unprocessed data from an image sensor, which is a misconception. Subsequent to the absorption of photons by a pixel, a considerable number of hardware-based image processing actions occur prior to the export of RAW data. It has been demonstrated that the issue can be mitigated to a certain extent through the process of camera calibration. This process encompasses a range of calibration techniques, including dark frame, PRNU, bad/hot pixels, linearization, shading correction, undistortion, color profile, noise profile, and lens profile. However, it is important to note that this approach merely serves to demonstrate that the RAW images captured by the camera are not truly RAW. Nonetheless, any high-quality ISP necessitates camera calibration data. This can be a ground of obtaining excellent images; however, it is imperative to acknowledge that our original images are not precisely RAW frames. The application of a lossy encoding algorithm directly to a raw Bayer image reveals the primary challenge associated with the task at hand. This approach has been demonstrated to result in the presence of significant artifacts, which, by definition, are not acceptable. It is evident that these artifacts will become discernible following the conversion of the RAW data to the RGB format. This indicates that raw frames are not prepared for direct lossy encoding and necessitate preprocessing to render them suitable for encoding. The idea of preprocessing is evident due to its similarity across various lossy encoding algorithms for monochrome images. To compress a monochrome image in a lossy way, we need to convert input data to a latent space via shift and DCT for JPEG or DWT for JPEG 2000, this is a fundamental the first step. Quantization, which constitutes the primary lossy component of the algorithm, is then followed by regrouping and data compression with an entropy codec, which by default operate in a lossless way. The same approach should be applied to RAW encoding to obtain a satisfactory result. Image Quality and CR ConsiderationsBelow we present some results of our new solution for RAW Image Compression. In the context of RAW bayer frames derived from conventional 12-bit image sensors manufactured by Sony, Gpixel, On-Semi, and CMOSIS, a compression ratio of CR~2 is typically attained for lossless algorithms. The aforementioned considerations pertain to 14-bit raw data from photographic cameras manufactured by Sony, Canon, Nikon, and other companies. This is due to the fact that useful data also includes noise, which could be very important in scientific applications that require lossless data compression. Apart from small CR, lossless encoding has been demonstrated to exhibit lower performance in comparison to its lossy counterparts. The application of lossy encoding typically results in a trade-off between loss (higher CR) and enhanced performance, the efficacy of which is contingent upon the content of the image, the parameters employed, and the hardware utilized. It is imperative to understand the mechanisms of controlling the level of loss in order to ensure the preservation of high-quality images for a specific task. In most cases we are able to compress RAW frames with CR around 5–10 times, thereby achieving a peak signal-to-noise ratio (PSNR) of 50–55 decibels (dB) or better. The PSNR is computed for the original RAW Bayer image and the same image after processing, encoding, decoding, and restoring. It is acknowledged that PSNR metrics are of limited utility. To this end, visual image quality assessment is performed to determine the most suitable default processing parameters for a given project. A visual comparison is conducted between RGB images, which undergo processing under identical parameters, as opposed to RAW images. For instance, there should be no perceptible differences between the source image and its processed counterpart when viewed at a 500% zoom level. However, this requirement is contingent upon the specific task at hand. For applications that require maximum image quality with lossy compression, we've got a CR in the range of range of 5:1 and more, thereby attaining PSNR values exceeding 60 dB. There are numerous examples of cases that exhibit significantly weaker image quality requirements and a pronounced need for enhanced CR. Achieving a CR of up to 20:1 for such applications is a feasible objective. It is imperative to emphasize that our solution with lossy encoding is not merely a codec in the conventional sense. This solution is characterized by its sophistication and the extensive range of its capabilities, which extend well beyond data compression and decompression. The proposed methodology strikes a balance between preserving raw data and performing additional image processing to improve contrast-to-noise ratio and minimize image distortion. Conventional wisdom suggests that the utilization of advanced technological frameworks is instrumental in attaining superior performance metrics in comparison to traditional JPEG and JPEG 2000 codecs. This assertion is substantiated by empirical evidence demonstrating the efficacy of our technology in achieving notably enhanced CR and PSNR values for raw Bayer and monochrome images. Once the requirements of a given project are thoroughly understood, the software can be customized to optimize results for specific applications. As an example, we've processed a raw frame with ColorChecker from the DALSA/FLIR camera (20 MPix, 4512 x 4512, 12-bit, RGGB). We've applied lossy raw bayer encoding with CR = 10 and got PSNR = 45.3 dB. The standard deviation (σ) has changed at average less than 2%, and the SNR has changed less than 0.15 dB. We've taken measurements for each patch of the ColorChecker. Benefits of RAW Image Compression
Conclusions
How to proceedGet in touch with our team to learn how to reduce your RAW images in size by up to 10× with our software and how to integrate that solution into your pipeline with our SDK.
Useful links on the subject |