IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 20, NO.

4, APRIL 2011

959

Fuzzy Random Impulse Noise Removal From Color Image Sequences

Tom Mlange, Mike Nachtegael, and Etienne E. Kerre
AbstractIn this paper, a new fuzzy lter for the removal of random impulse noise in color video is presented. By working with different successive ltering steps, a very good tradeoff between detail preservation and noise removal is obtained. One strong ltering step that should remove all noise at once would inevitably also remove a considerable amount of detail. Therefore, the noise is ltered step by step. In each step, noisy pixels are detected by the help of fuzzy rules, which are very useful for the processing of human knowledge where linguistic variables are used. Pixels that are detected as noisy are ltered, the others remain unchanged. Filtering of detected pixels is done by blockmatching based on a noise adaptive mean absolute difference. The experiments show that the proposed method outperforms other state-of-the-art lters both visually and in terms of objective quality measures such as the mean absolute error (MAE), the peak-signal-to-noise ratio (PSNR) and the normalized color difference (NCD). Index TermsCircuits and systems, computers and information processing, computational and articial intelligence, ltering, lters, fuzzy logic, image denoising, logic, nonlinear lters.

I. INTRODUCTION MAGES and videos belong to the most important information carriers in todays world (e.g., trafc observations, surveillance systems, autonomous navigation, etc.). However, the images are likely to be corrupted by noise due to bad acquisition, transmission or recording. Such degradation negatively inuences the performance of many image processing techniques and a preprocessing module to lter the images is often required. Among those lters, more and more fuzzy techniques start to appear in literature [7], [11][18], [33], [41], [43], [44][47], [49], [52], [53]. Fuzzy set theory was introduced by Zadeh in 1965 [24] and is a generalization of classical set theory. A classical crisp set over a universe can be modelled by a mapping (characteristic function): an element belongs to the set or does not belong to it. Fuzzy sets are now modelled as mappings (membership functions). So the characteristic function is extended to a membership function where also membership degrees between zero and one are allowed. An elcan now also belong to some degree to the set, ement

Manuscript received March 25, 2010; revised July 12, 2010; accepted August 31, 2010. Date of publication September 20, 2010; date of current version March 18, 2011. This work was supported by the GOA project B/04138/01 of Ghent University. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Kenneth K. M. Lam. The authors are with the Fuzziness and Uncertainty Modeling Research Unit, Department of Applied Mathematics and Computer Science, Ghent University, 9000 Ghent, Belgium (e-mail: tom.melange@ugent.be; mike.nachtegael @ugent.be; etienne.kerre@ugent.be; http://www.fuzzy.ugent.be). Color versions of one or more of the gures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identier 10.1109/TIP.2010.2077305

which allows a more gradual transition between belonging to and not belonging to. Such gradual transition makes fuzzy sets very useful for the processing of human knowledge in terms of linguistic variables (e.g., large, small, etc.). There is for example no need to use a threshold to decide whether a difference in color component value between two pixels is large or not. Two differences that differ only one unit (which is not noticeable by the human eye) could then be respectively large and not large. It is better to allow a difference to be large to some intermediate degree. For a larger difference, this degree will be higher than that of a smaller difference. For an illustration of the effectiveness of fuzzy set theory in image processing, we refer to, e.g., [25]. Most lters in literature, that are developed for video, are intended for sequences corrupted by additive Gaussian noise (e.g., [3][7]). Only few video lters for the impulse noise case can be found (e.g., [19][23], [30], [31], [45], [46]). However, several impulse noise lters for still images exist. The best known among them are the median based rank-order lters (e.g., [8][10], [32], [34][40], [48]. But also some fuzzy techniques can be found [11][18], [33], [41]. Such 2-D lters could be used to lter each of the frames of a video successively. However, temporal inconsistencies will arise due to the neglection of the temporal correlation between successive frames. A better alternative would be to use 3-D ltering windows, in which also pixels from neighboring frames are taken into account [19][23], [30], [31], [45], [46]. The main problem in using neighboring frames is motion between them. Using pixels at corresponding spatial positions in neighboring frames for noise removal may introduce ghosting artifacts in the presence of camera and object motion. In the method proposed in this paper, we will therefore only in non-moving areas assign a temporal impulse between two corresponding spatial positions to noise (detection phase) and for the replacement of a noisy pixel (ltering phase) motion compensation will be applied to nd the most reliable pixel in the previous frame. Analogously, a distinction between lters intended for greyscale images and for color images needs to be made. Filters for greyscale images could be used for color images by applying them on each of the color bands of the image separately. In this paper, we consider the images to be modeled in the RGB color space and we thus have three color bands: red, green and blue. However, such approach will generally introduce many color artefacts, especially in textured areas, due to the neglection of the correlation between the different color bands. To incorporate this correlation, vector-based methods were introduced. Most of these methods are based on ordering the vectors in a predened ltering window. The output for a given color pixel is then the pixel in the window around the given pixel, that has

1057-7149/$26.00 2011 IEEE

MLANGE et al.: FUZZY RANDOM IMPULSE NOISE REMOVAL FROM COLOR IMAGE SEQUENCES

969

is based on fuzzy rules in which information from spatial and temporal neighbors as well as from the other color bands is used. Detected noisy components are ltered based on blockmatching where a noise adaptive mean absolute difference is used and where the search region contains pixels blocks from both the previous and current frame. The experiments showed that the proposed method outperforms other state-of-the-art methods both in terms of objective measures such as MAE, PSNR and NCD and visually. REFERENCES
[1] E. Abreu, M. Lightstone, S. K. Mitra, and K. Arakawa, A new efcient approach for the removal of impulse noise from highly corrupted images, IEEE Trans. Image Process., vol. 5, no. 6, pp. 10121025, 1996. [2] R. H. Chan, C. Hu, and M. Nikolova, An iterative procedure for removing random-valued impulse noise, IEEE Signal Process. Lett., vol. 11, no. 12, pp. 921924, 2004. [3] S. M. M. Rahman, M. O. Ahmad, and M. N. S. Swamy, Video denoising based on inter-frame statistical modelling of wavelet coefcients, IEEE Trans. Circuits Syst. Video Technol., vol. 17, no. 2, pp. 187198, 2007. [4] L. Jovanov, A. Pizurica, V. Zlokolica, S. Schulte, P. Schelkens, A. Munteanu, E. E. Kerre, and W. Philips, Combined wavelet-domain and motion-compensated video denoising based on video codec motion estimation methods, IEEE Trans. Circuits Syst. Video Technol., vol. 19, no. 3, pp. 417421, 2009. [5] H. B. Yin, X. Z. Fang, Z. Wei, and X. K. Yang, An improved motion-compensated 3-D LLMMSE lter with spatio-temporal adaptive ltering support, IEEE Trans. Circuits Syst. Video Technol., vol. 17, no. 12, pp. 17141727, 2007. [6] L. Guo, O. C. Au, M. Ma, and Z. Liang, Temporal video denoising based on multihypothesis motion compensation, IEEE Trans. Circuits Syst. Video Technol, vol. 17, no. 10, pp. 14231429, 2007. [7] T. Mlange, M. Nachtegael, E. E. Kerre, V. Zlokolica, S. Schulte, V. De Witte, A. Pizurica, and W. Philips, Video denoising by fuzzy motion and detail adaptive averaging, J. Electron. Imaging, vol. 17, no. 4, pp. 043005, 2008. [8] T. Chen, K. K. Ma, and L. H. Chen, Tri-state median lter for image denoising, IEEE Trans. Image Process., vol. 8, pp. 18341838, 1999. [9] R. C. Hardie and C. G. Boncelet, LUM lters: A class of rank-order-based lters for smoothing and sharpening, IEEE Trans. Signal Process., vol. 41, pp. 10611076, 1993. [10] S. J. Ko and Y. H. Lee, Center weighted median lters and their applications to image enhancement, IEEE Trans. Circuits Syst., vol. 38, pp. 984993, 1991. [11] S. M. Guo, C. S. Lee, and C. Y. Hsu, An intelligent image agent based on soft-computing techniques for color image processing, Expert Systems With Appl., vol. 28, pp. 483494, 2005. [12] S. Schulte, V. De Witte, M. Nachtegael, D. Van der Weken, and E. E. Kerre, Fuzzy random impulse noise reduction method, Fuzzy Sets Syst., vol. 158, pp. 270283, 2007. [13] S. Schulte, M. Nachtegael, V. De Witte, D. Van der Weken, and E. E. Kerre, A fuzzy impulse noise detection and reduction method, IEEE Trans. Image Process., vol. 15, no. 5, pp. 11531162, 2006. [14] F. Russo, Fire operators for image processing, Fuzzy Sets Syst., vol. 103, no. 2, pp. 265275, 1999. [15] F. Russo, Hybrid neuro-fuzzy lter for impulse noise removal, Pattern Recognit., vol. 32, pp. 18431855, 1999. [16] H. K. Kwan, Fuzzy lters for noise reduction in images, in Fuzzy Filters for Image Processing, M. Nachtegael, D. Van der Weken, D. Van De Ville, and E. E. Kerre, Eds. Heidelberg, Germany: Springer, 2003, pp. 5471. [17] J. H. Wang, W. J. Liu, and L. D. Lin, An histogram-based fuzzy lter for image restoration, IEEE Trans. Syst. Man Cybern. B, Bern., vol. 32, no. 2, pp. 230238, 2002. [18] H. Xu, G. Zhu, H. Peng, and D. Wang, Adaptive fuzzy switching lter for images corrupted by impulse noise, Pattern Recognit. Lett., vol. 25, pp. 16571663, 2004. [19] F. Cocchia, S. Carrato, and G. Ramponi, Design and real-time implementation of a 3-D rational lter for edge preserving smoothing, IEEE Trans. Consum. Electron., vol. 43, no. 4, pp. 12911300, 1997. [20] J.-S. Kim and H. W. Park, Adaptive 3-D median ltering for restoration of an image sequence corrupted by impulse noise, Signal Process.: Image Commun., vol. 16, pp. 657668, 2001.

Fig. 9. 20th frame of the Deadline sequence (top-left to bottom-right): original, noisy (p = 25%), INRC, AVMF, VAVDMF and Proposed.

TABLE III AVERAGE RUNNING TIME (SECONDS PER FRAME) FOR THE PROCESSING OF THE SALESMAN SEQUENCE

stage could be sped up by using fast motion estimation techniques such as those presented in [27][29]. Next, for higher noise levels, it might be useful to do the block matching for blocks of pixels and to lter each of the noisy pixels in the blocks at the same time instead of applying the block matching for each noisy pixel separately. Further, in each of the successive steps in the algorithm, the detection and ltering of a pixel does not depend on the detection and ltering of the other pixels in the frame, such that the algorithm could be further sped up by performing this detection and ltering for several pixels in parallel. IV. CONCLUSION In this paper, we have presented a new ltering framework for color videos corrupted with random valued impulse noise. In order to preserve the details as much as possible, the noise is removed step by step. The detection of noisy color components

970

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 20, NO. 4, APRIL 2011

[21] P. S. Windyga, Fast impulsive noise removal, IEEE Trans. Image Process., vol. 10, no. 1, pp. 173179, 2001. [22] R. Lukac and S. Marchevsky, LUM smoother with smooth control for noisy image sequences, EURASIP J. Appl. Signal Process., vol. 2001, no. 2, pp. 110120, 2001. [23] M. El Hassouni, H. Cheri, and D. Aboutajdine, HOS-based image sequence noise removal, IEEE Trans. Image Process., vol. 15, no. 3, pp. 572581, 2006. [24] L. A. Zadeh, Fuzzy sets, Information and Control, vol. 8, pp. 338353, 1965. [25] , E. E. Kerre and M. Nachtegael, Eds., Fuzzy techniques in image processing, in Studies in Fuzziness and Soft Computing. Heidelberg, Germany: Physica-Verlag, 2000, vol. 52. [26] S. Weber, A general concept of fuzzy connectives, negations and implications based on t-norms and t-conorms, Fuzzy Sets Syst., vol. 11, no. 2, pp. 115134, 1983. [27] L. M. Po and W. C. Ma, A novel four-step search algorithm for fast block motion estimation, IEEE Trans. Circuits Syst. Video Technol., vol. 6, no. 3, pp. 313317, 1996. [28] S. Zhu and K. K. Ma, A new diamond search algorithm for fast blockmatching motion estimation, IEEE Trans. Image Process., vol. 9, no. 2, pp. 287290, 2000. [29] C. Zhu, X. Lin, and L. P. Chau, Hexagon-based search pattern for fast block motion estimation, IEEE Trans. Circuits Syst. Video Technol., vol. 12, no. 5, pp. 349355, 2002. [30] V. Ponomaryov, A. Rosales-Silva, and V. Golikov, Adaptive and vector directional processing applied to video colour images, Electron. Lett., vol. 42, no. 11, pp. 623624, 2006. [31] R. Lukac, Vector order-statistics for impulse detection in noisy color image sequences, in Proc. 4th EURASIP-IEEE Region 8 Int. Symp. Video/Image Process. Multimedia Commun., Zadar, Croatia, 2002. [32] R. Lukac, Adaptive vector median ltering, Pattern Recognit. Lett., vol. 24, pp. 18891899, 2003. [33] S. Schulte, S. Morillas, V. Gregori, and E. E. Kerre, A new fuzzy color correlated impulse noise reduction method, IEEE Trans. Image Process., vol. 16, no. 10, pp. 25652575, 2007. [34] R. Lukac, K. N. Plataniotis, A. N. Venetsanopoulos, and B. Smolka, A statistically-switched adaptive vector median lter, J. Intell. Robot. Syst., vol. 42, no. 4, pp. 361391, 2005. [35] J. Camacho, S. Morillas, and P. Latorre, Efcient impulse noise suppression based on statistical condence limits, J. Imag. Sci. Technol., vol. 50, no. 5, pp. 427436, 2006. [36] S. Hore, B. Qiu, and H. R. Wu, Improved vector ltering for color images using fuzzy noise detection, Opt. Eng., vol. 42, no. 6, pp. 16561664, 2003. [37] R. Lukac and K. N. Plataniotis, A taxonomy of color image ltering and enhancement solutions, Adv. Imag. Electron. Phys., vol. 140, pp. 187264, 2006. [38] R. Lukac, Adaptive color image ltering based on center-weighted vector directional lters, Multidimen. Syst. Signal Process., vol. 15, no. 2, pp. 169196, 2004. [39] Z. H. Ma, H. R. Wu, and B. Qiu, A robust structure-adaptive hybrid vector lter for color image restoration, IEEE Trans. Image Process., vol. 14, no. 12, pp. 19902001, 2005. [40] V. Chatzis and I. Pitas, Fuzzy scalar and vector median lters based on fuzzy distances, IEEE Trans. Image Process., vol. 8, no. 5, pp. 731734, 1999. [41] S. Morillas, V. Gregori, G. Peris-Fajarns, and P. Latorre, A fast impulsive noise color image lter using fuzzy metrics, Real-Time Imag., vol. 11, no. 56, pp. 417428, 2005. [42] S. J. Sangwine and R. E. N. Horne, The Colour Image Processing Handbook. London, U.K.: Chapman & Hall, 1998. [43] J. G. Camarena, V. Gregori, S. Morillas, and A. Sapena, Fast detection and removal of impulsive noise using peer groups and fuzzy metrics, J. Vis. Commun. Image Represent., vol. 19, no. 1, pp. 2029, 2008. [44] S. Morillas, V. Gregori, and A. Hervs, Fuzzy peer groups for reducing mixed Gaussian-impulse noise from color images, IEEE Trans. Image Process., vol. 18, no. 7, pp. 14521466, 2009. [45] V. Ponomaryov, A. Rosales, and F. Gallegos, 3D ltering of colour video sequences using fuzzy logic and vector order statistics, in Proc. Advanced Concepts for Intelligent Vision Systems, LNCS 5807, 2001, pp. 210221. [46] V. F. Kravchenko, V. I. Ponomaryov, and V. I. Pustovoit, Three-dimensional ltration of multichannel video sequences on the basis of fuzzy-set theory, Doklady Phys., vol. 55, no. 2, pp. 5863, 2010. [47] V. Ponomaryov, F. Gallegos-Funes, and A. Rosales-Silva, Fuzzy directional (FD) lter to remove impulse noise from colour images, IEICE Trans. Fundament. Electron., Commun. Comput. Sci., vol. E93-A, no. 2, pp. 570572, 2010.

[48] Z. Xu, H. R. Wu, B. Qiu, and X. Yu, Geometric features based ltering for suppression of impulse noise in color images, IEEE Trans. Image Process., vol. 18, no. 8, pp. 17421759, 2009. [49] C.-C. Kang and W.-J. Wang, Fuzzy reasoning-based directional median lter design, Signal Process., vol. 89, no. 3, pp. 344351, 2009. [50] V. I. Ponomaryov, Real-time 2D-3D ltering using order statistics based algorithms, J. Real-Time Image Process., vol. 1, no. 3, pp. 173194, 2007. [51] J. G. Camarena, V. Gregori, S. Morillas, and A. Sapena, Some improvements for image ltering using peer group techniques, Image Vis. Comput., vol. 28, no. 1, pp. 188201, 2010. [52] S. Morillas, V. Gregori, and G. Peris-Fajarns, Isolating impulsive noise pixels in color images by peer group techniques, Comput. Vis. Image Underst., vol. 110, no. 1, pp. 102116, 2008. [53] S. Morillas, V. Gregori, G. Peris-Fajarns, and A. Sapena, Local self-adaptive fuzzy lter for impulsive noise removal in color images, Signal Process., vol. 88, no. 2, pp. 390398, 2008. [54] Q. Ma, L. Zhang, and B. Wang, New strategy for image and video quality assessment, J. Electron. Imaging, vol. 19, no. 011019, 2010. [55] H. R. Sheikh and A. C. Bovik, A visual information delity approach to video quality assessment, in Proc. 1st Int. Workshop on Video Process. and Quality Metrics for Consumer Electron., 2005, pp. 2325.

Tom Mlange was born in Kortrijk, Belgium, in 1984. He received the M.Sc. degree in mathematics from Ghent University, Ghent, Belgium, in 2006. In October 2006, he joined the Department of Applied Mathematics and Computer Science, Ghent University, where he is a member of the Fuzziness and Uncertainty Modeling Research Unit. In 2010 he received the Ph.D. degree with a thesis on fuzzy techniques for noise reduction in video and interval-valued fuzzy mathematical morphology, under Prof. Dr. E. E. Kerre.

Mike Nachtegael was born on February 16, 1976. He received the M.Sc. degree in mathematics from Ghent University, Ghent, Belgium, in 1998. In the same year he joined the Department of Applied Mathematics and Computer Science, where he is a member of the Fuzziness and Uncertainty Modeling Research Unit. In May 2002 he received the Ph.D. in mathematics, on the topic of fuzzy techniques in image processing. In 2002, he became an Assistant Professor in his Department and since 2008 he has held the position of Guest Professor.

Etienne E. Kerre was born in Zele, Belgium, on May 8, 1945. He received the M.Sc. degree in mathematics and the Ph.D. in mathematics from Ghent University, Ghent, Belgium, in 1967 and 1970, respectively. Since 1984, he has been a Lector, and since 1991, a full Professor at Ghent University. He is a referee for more than 30 international scientic journals, and a member of the editorial board of international journals and conferences on fuzzy set theory. He has been an honorary chairman at various international conferences. In 1976, he founded the Fuzziness and Uncertainty Modeling Research Unit (FUM) and since then his research has been focused on the modeling of fuzziness and uncertainty, and has resulted in a great number of contributions in fuzzy set theory and its various generalizations. Especially the theories of fuzzy relational calculus and of fuzzy mathematical structures owe a great deal to him. Over the years he has also been a promotor of 29 Ph.D.s on fuzzy set theory. His current research interests include fuzzy and intuitionistic fuzzy relations, fuzzy topology, and fuzzy image processing. He has authored or co-authored 25 books, and more than 450 papers in international refereed journals and proceedings.