Dynamic bad pixel correction in bayer images on GPUModern image sensors usually have well over a megapixel resolution, and we might expect all of those pixels to be fully functional. Unfortunately, this is not the case, and among the millions of pixels there are some that are defective. It's a very complicated task to make megapixel image sensors without such pixels. These pixels are usually called "bad pixels" and they can be classified as follows:
If we look at the detailed specifications of any image sensor from any manufacturer, we will see a confidential document describing bad pixels and their locations. Yes, every image sensor manufacturer knows that there are some bad pixels and that the number of bad pixels will increase as the sensor is used. In order to consider a particular image sensor as good, we could compare the actual positions and number of bad pixels with the specification. Usually, bad pixels are not allowed to be near the centre of the image sensor, but closer to the edges. Camera manufacturers usually sort image sensors before making cameras, because different camera applications do not have the same image quality requirements. If we compare a surveillance camera with a microscope camera, they simply can't have the same number of bad pixels. Image sensor manufacturers rate their devices according to the number and type of defects present. Bad pixels are really bad if we don't remove them before image segmentation, image recognition, stitching, 3D modelling, etc. These bad pixels aren't good for viewing either. So we see too many reasons to suppress them. Once bad pixels are fairly common, there are several ways to remove them:
Bad pixel suppression in the FPGA is a very convenient way and it's super fast, but we need to keep the list of bad pixels up to date. To check the appearance of new bad pixels, we need to run calibration tests to update this list. Unfortunately, this is not always convenient to do, so this approach is not 100% applicable. The most flexible approach is to remove bad pixels from RAW data using software. We can either interpolate bad pixels according to a known list of coordinates, or we can implement a special dynamic filter that is able to detect and interpolate bad pixels in RAW data. If we haven't removed bad pixels from RAW, we can still remove them from RGB. In such a case, removal is still possible, but it's done after demosaicing, which is why some "traces" of bad pixels might be found on the neighbouring pixels. This method is also possible, but it's not the best. Bad pixel correction filterWe've implemented a powerful filter for Bad Pixel Correction (BPC). Basically, we need to remove impulse noise, and such removal is almost always good for the image. The main idea is not to apply this filter to all pixels, but only to those that look like bad/hot/dead pixels. We analyse the environment of each RAW pixel and formulate some criteria to recognise the pixel as bad. This is done in the RAW domain, so we apply different criteria for green and red/blue pixels. The algorithm can work over a wide range of illumination levels and we've done some tests to investigate these situations. We see almost the same situation when we calibrate a camera to get a Flat-Field Correction (FFC) image. Hot pixels can affect the results of FFC, so we need to interpolate them before we do any calculations.
Apart from repairing bad pixels, we should always keep an eye on the total number of pixels processed by this filter. This is an indirect way of checking image sensor degradation and quality. This solution is practical and could be applied to RAW data from any camera. Soon we will extend this filter to work with RGB data as well, because there is such a need to fix bad RBG pixels. Bad pixel replacement on NVIDIA GPUWe've implemented the BPC filter on the GPU to provide good performance and quality even for cameras with high resolution image sensors in real-time applications. We process bad pixels in the RAW domain and we can do it really fast. A frame from a 12 MPix 12-bit image sensor could be fixed in less than ~1 ms on NVIDIA GeForce GTX 1080 even without having all the coordinates of the bad pixels - we detect and interpolate bad pixels in real time. This is actually a dynamic bad pixel correction algorithm. This feature is now part of the Fastvideo SDK and it's available for NVIDIA mobile, laptop, desktop and server GPUs.
To check performance and quality for BPC algorithms, you can download Fast CinemaDNG Processor software for evaluation. |