Computer Vision Technology for Food Quality Evaluation

Ireland

Preface

Based on image processing and analysis, computer vision is a novel technology for recognizing objects and extracting quantitative information from digital images in order to provide objective, rapid, non-contact, and non-destructive quality evaluation.

Driven by significant increases in computer power and rapid developments in image-processing techniques and software, the application of computer vision has been extended to the quality evaluation of diverse and processed foods. In recent years in particular, computer vision has attracted much research and development attention; as a result, rapid scientific and technological advances have increasingly taken place regarding the quality inspection, classification, and evaluation of a wide range of food and agricultural products. As the first book in this area, Computer Vision Technology for Food Quality Evaluation focuses on these recent advances.

The book is divided into five parts. Part I provides an outline of the fundamentals of the technology, addressing the principles and techniques for image acquisition, segmentation, description, and recognition. Part II presents extensive coverage of the application in the most researched areas of fresh and cooked meats, poultry, and seafood. Part III details the application of computer vision in the quality evaluation of agricultural products, including apples, citrus, strawberry, table olives, and potatoes. Using computer vision to evaluate and classify the quality of grains such as wheat, rice and corn is then discussed in Part IV. The book concludes with Part V, which is about applying computer vision technology to other food products, including pizza, cheese, bakery, noodles, and potato chips.

Computer Vision Technology for Food Quality Evaluation is written by international peers who have both academic and professional credentials, with each chapter addressing in detail one aspect of the relevant technology, thus highlighting the truly international nature of the work. The book therefore provides the engineer and technologist working in research, development, and operations in the food industry with critical, comprehensive, and readily accessible information on the art and science of computer vision technology. It should also serve as an essential reference source for undergraduate and postgraduate students and researchers in universities and research institutions.

Part I

Fundamentals of Computer Vision Technology

Outline

1 Image Acquisition Systems

2 Image Segmentation Techniques

3 Object Measurement Methods

4 Object Classification Methods

1

Image Acquisition Systems

Mohd. Zaid Abdullah,    School of Electrical and Electronic Engineering, Engineering Campus, Universiti Sains Malaysia, 14300 Penang, Malaysia

1 Introduction

In making physical assessments of agricultural materials and foodstuffs, images are undoubtedly the preferred method in representing concepts to the human brain. Many of the quality factors affecting foodstuffs can be determined by visual inspection and image analysis. Such inspections determine market price and, to some extent, the best-used-before date. Traditionally, quality inspection is performed by trained human inspectors, who approach the problem of quality assessment in two ways: seeing and feeling. In addition to being costly, this method is highly variable and decisions are not always consistent between inspectors or from day to day. This is, however, changing with the advent of electronic imaging systems and with the rapid decline in cost of computers, peripherals, and other digital devices. Moreover, the inspection of foodstuffs for various quality factors is a very repetitive task which is also very subjective in nature. In this type of environment, machine vision systems are ideally suited for routine inspection and quality assurance tasks. Backed by powerful artificial intelligence systems and state-of-the-art electronic technologies, machine vision provides a mechanism in which the human thinking process is simulated artificially. To date, machine vision has extensively been applied to solve various food engineering problems, ranging from simple quality evaluation of food products to complicated robot guidance applications (Tao et al., 1995; Pearson, 1996; Abdullah et al., 2000). Despite the general utility of machine vision images as a first-line inspection tool, their capabilities regarding more in-depth investigation are fundamentally limited. This is due to the fact that images produced by vision cameras are formed using a narrow band of radiation, extending from 10−4 m to 10−7 m in wavelength. For this reason, scientists and engineers have invented camera systems that allow patterns of energy from virtually any part of the electromagnetic spectrum to be visualized. Camera systems such as computed tomography (CT), magnetic resonance imaging (MRI), nuclear magnetic resonance (NMR), single photon emission computed tomography (SPECT) and positron emission tomography (PET) operate at shorter wavelengths ranging from 10−8 m to 10−13 m. Towards the opposite end of the electromagnetic spectrum there are infrared and radio cameras, which enable visualization to be performed at wavelengths greater than 10−6 m and 10−4 m, respectively. All these imaging modalities rely on acquisition hardware featuring an array or ring of detectors which measure the strength of some form of radiation, either following reflection or after the signal has passed transversely through the object. Perhaps one thing that these camera systems have in common is the requirement to perform digital image processing of the resulting signals using modern computing power. Whilst digital image processing is usually assumed to be the process of converting radiant energy in a three-dimensional world into a two-dimensional radiant array of numbers, this is certainly not so when the detected energy is outside the visible part of the spectrum. The reason is that the technology used to acquire the imaging signals is quite different, depending on the camera modalities. The aim of this chapter is therefore to give a brief review of the present state-of-the-art image acquisition technologies that have found many applications in the food industry.

Section 2 summarizes the electromagnetic spectrum which is useful in image formation. Section 3 describes the principles of operation of machine vision technology, along with illumination and electronics requirements. Other imaging modalities, particularly the acquisition technologies operating at the non-visible range, are briefly discussed in Section 4. In particular, technologies based on ultrasound, infrared, MRI and CT are addressed, followed by some of their successful applications in food engineering found in the literature. Section 5 concludes by addressing likely future developments in this exciting field of electronic imaging.

2 The electromagnetic spectrum

As discussed above, images are derived from electromagnetic radiation in both visible and non-visible ranges. Radiation energy travels in space at the speed of light in the form of sinusoidal waves with known wavelengths. Arranged from shorter to longer wavelengths, the electromagnetic spectrum provides information on the frequency as well as the energy distribution of the electromagnetic radiation. Figure 1.1 shows the electromagnetic spectrum of all electromagnetic waves.

Figure 1.1 The electromagnetic spectrum comprising the visible and non-visible range.

Referring to Figure 1.1, the gamma rays with wavelengths of less than 0.1 nm constitute the shortest wavelengths of the electromagnetic spectrum. Traditionally, gamma radiation is important for medical and astronomical imaging, leading to the development of various types of anatomical imaging modalities such as CT, MRI, SPECT, and PET. In CT the radiation is projected onto the target from a diametrically opposed source, whilst with others it originates from the target – by simulated emission in the case of MRI, and through the use of radiopharmaceuticals in SPECT and PET. At the other end of the spectrum, the longest waves are radio waves, which have wavelengths of many kilometers. The well-known ground-probing radar (GPR) and other microwave-based imaging modalities operate in this frequency range.

Located in the middle of the electromagnetic spectrum is the visible range, consisting of narrow portion of the spectrum with wavelengths ranging from 400 nm (blue) to 700 nm (red). The popular charge-coupled device or CCD camera operates in this range.

Infrared (IR) light lies between the visible and microwave portions of the electromagnetic band. As with visible light, infrared has wavelengths that range from near (shorter) infrared to far (longer) infrared. The latter belongs to the thermally sensitive region, which makes it useful in imaging applications that rely on the heat signature. One example of such an imaging device is the indium gallium arsenide (InGaAs)-based near-infrared (NIR) camera, which gives the optimum response in the 900–1700-nm band (Doebelin, 1996).

Ultraviolet (UV) light is of shorter wavelength than visible light. Similar to IR, the UV part of the spectrum can be divided, this time into three regions: near ultraviolet (NUV) (300 nm) (NUV), far ultraviolet (FUV) (30 nm), and extreme ultraviolet (EUV) (3 nm). NUV is closest to the visible band, while EUV is closest to the X-ray region and therefore is the most energetic of the three types. FUV, meanwhile, lies between the near and extreme ultraviolet regions, and is the least explored of the three. To date there are many types of CCD camera that provide sensitivity at the near-UV wavelength range. The sensitivity of such a camera usually peaks at around 369 nm while offering coverage down to 300 nm.

Mathematically, the wavelength (λ), the frequency (f), and the energy (E) are related by Planck’s equation:

(1.1)

where h is the Planck’s constant (6.626076 × 10−34 J s), and c is the speed of light (2.998 × 10−34 m/s). Consequently, the energy increases as the wavelength decreases. Therefore, gamma rays, which have the shortest wavelengths, have the highest energy of all the electromagnetic waves. This explains why gamma rays can easily travel through most objects without being affected. In contrast, radio waves have the longest wavelength and hence the lowest energy. Therefore, their penetrative power is at least hundreds order of magnitude lower than that of gamma or X-rays. Moreover, both gamma and X-rays travel in a straight line and their paths are not affected by the object through which these signals propagate. This is known as the hard-field effect. Conversely, radiowaves do not travel in straight lines and their paths depend strongly on the medium of propagation. This is the soft-field effect. Both the hard- and soft-field effects have a direct effect on the quality of images produced by these signals. The soft-field effect causes many undesirable artefacts, most notably, image blurring, and therefore images produced by gamma rays generally appear much clearer than do images produced by radiowaves. Another important attribute that is wavelength-dependent is image resolution. In theory, the image spatial resolution is essentially limited to half of the interrogating wavelength, and therefore the spatial resolution also increases as the wavelength decreases. Thus, the resolution of typical gamma rays is less than 0.05 nm, enabling this type of electromagnetic wave to see extremely small objects such as water molecules. In summary, these attributes, along with the physical properties of the sensor materials, establish the fundamental limits to the capability of imaging modalities and their applications.

The following sections explain the technology of image acquisition and applications for all the imaging modalities discussed, focusing on the visible modality or computer vision system, since this device has extensively been used for solving various food engineering problems. Moreover, given the progress in computer technology, computer vision hardware is now relatively inexpensive and easy to use. To date, some personal computers offer capability for a basic vision system by including a camera and its interface within the system. However, there are specialized systems for vision, offering performance in more than one aspect. Naturally, as with any specialized equipment, such systems can be expensive.

3 Image acquisition systems

In general, images are formed by incident light in the visible spectrum falling on a partially reflective, partially absorptive surface, with the scattered photons being gathered up in the camera lens and converted to electrical signals either by vacuum tube or by CCD. In practice, this is only one of many ways in which images can be generated. Generally, thermal and ultrasonic methods, X-rays, radiowaves, and other techniques can all generate an image. This section examines the methods and procedures by which images are generated for computer vision applications, including tomography.

3.1 Computer vision

The hardware configuration of computer-based machine vision systems is relatively standard. Typically, a vision system consists of:

• an illumination device, which illuminates the sample under test

• a solid-state CCD array camera, to acquire an image

• a frame-grabber, to perform the A/D (analog-to-digital) conversion of scan lines into picture elements or pixels digitized in a N row by M column image

• a personal computer or microprocessor system, to provide disk storage of images and computational capability with vendor-supplied software and specific application programs

• a high-resolution color monitor, which aids in visualizing images and the effects of various image analysis routines.

Figure 1.2 shows a typical set-up, such as an investigator needs to start experimenting with machine vision applications. All essential components are commercially available, and the price for the elementary system can be as low as £2000.00.

Figure 1.2 Essential elements of a typical computer vision system.

The set-up shown in Figure 1.2 is an example of a computer vision system that can be found in many food laboratories, mainly for research and imaging applications. In this case, the objective is ultimately to free human inspectors from undertaking tedious, laborious, time-consuming, and repetitive inspection tasks, allowing them to focus on more demanding and skilled jobs. Computer vision technology not only provides a high level of flexibility and repeatability at a relatively low cost, but also, and more importantly, it permits fairly high plant throughput without compromising accuracy. The food industry continues to be among the fastest-growing segments of machine vision application, and it ranks among the top ten industries that use machine vision systems (Gunasekaran, 1996). Currently, several commercial vendors offer automatic vision-based quality evaluation for the food industry.

Even though machine vision systems have become increasingly simple to use, the applications themselves can still be extremely complicated. A developer needs to know precisely what must be achieved in order to ensure successful implementation of a machine vision application. Key characteristics include not only the specific part dimensions and part tolerances, but also the level of measurement precision required and the speed of the production line. Virtually all manufacturing processes will produce some degree of variability and, while the best machine vision technology is robust enough to compensate automatically for minor differences over time, the applications themselves need to take major changes into account. Additional complexity arises for companies with complex lighting and optical strategies, or unusual materials-handling logistics. For these reasons, it is essential to understand the characteristics of the part and sub-assemblies of the machine system, as well as the specifications of the production line itself.

3.1.1 Illumination

The provision of correct and high-quality illumination, in many vision applications, is absolutely decisive. Despite the advances of machine vision hardware and electronics, lighting for machine vision remains the art for those involved in vision integration. Engineers and machine vision practitioners have long recognized lighting as being an important piece of the machine vision system. However, choosing the right lighting strategy remains a difficult problem because there is no specific guideline for integrating lighting into machine vision applications. In spite of this, some rules of thumb exist. In general, three areas of knowledge are required to ensure a successful level of lighting for the vision task:

1. Understanding of the role of the lighting component in machine vision applications

2. Knowledge of the behavior of light on a given surface

3. Understanding of the basic lighting techniques available that will allow the light to create the desired feature extraction.

In the vast majority of machine vision applications, image acquisition deals with reflected light, even though the use of backlit techniques can still be found. Therefore, the most important aspect of lighting is to understand what happens when light hits the surface – more specifically, to know how to control the reflection so that the image appears of a reasonably good quality.

Another major area of concern is the choice of illuminant, as this is instrumental in the capability of any form of machine vision to represent the image accurately. This is due to the fact that the sensor response of a standard imaging device is given by a spectral integration process (Matas et al., 1995). Mathematically,

(1.2)

where pxk is the response of the kth sensor at location x of the sensor array, ρk(λ) is the responsitivity function of the kth sensor, and L(λ) is the light reflected from the surface that is projected on pixel x. For a CCD camera the stimulus L(λ) is the product of the spectral power distribution S(λ) of the light that illuminates the object, and the spectral reflectance C(λ) of the camera itself, i.e.

(1.3)

Hence, two different illuminants, S1(λ) and S2(λ), may yield different stimuli using the same camera. Therefore, the illuminant is an important factor that must be taken into account when considering machine vision integration. Frequently, knowledgeable selection of an illuminant is necessary for specific vision applications.

Traditionally, the two most common illuminants are fluorescent and incandescent bulbs, even though other light sources (such as light-emitting diodes (LEDs) and electroluminescent sources) are also useful. Figure 1.3 shows the spectral distributions of three different light sources: the sun, an incandescent bulb, and standard cool white fluorescent light. Referring to Figure 1.3, the only difference between daylight and electric light is the amount of energy emitted at each wavelength. Even though the light energy itself is fundamentally the same, however, the optimum light will have more intensity than the other sources. When the light is not as intense as it should be, three possible damaging effects occur:

1. There may not be sufficient signal-to-noise ratio at the camera

2. The electrical noise tends to increase as the light gets dimmer and less intense

3. Most importantly, a less intense light will cause a significant loss in the camera depth-of-field.

Figure 1.3 Comparison in relative spectral energy distribution between daylight, incandescent, and cool white fluorescent light (Stiles and Wyszecki, 2000).

Additionally, effects from ambient light are more likely to occur under poor lighting conditions.

Referring to Figure 1.3 again, it can be seen that the incandescent source has a fairly normal distribution over the visible spectrum while the fluorescent source has sharp peaks in some regions. This means that objects under an incandescent source produce an image with a much lower signal-to-noise ratio. This is not acceptable in some cases, especially those that are concerned with color-image processing (Daley et al., 1993). In contrast, fluorescent bulbs are inherently more efficient, and produce more intense illumination at specific wavelengths. Moreover, fluorescent light provides a more uniform dispersion of light from the emitting surface, and hence does not require the use of diffusing optics to disseminate the light source over the field of view, as is the case with incandescent bulbs. For these reasons, a fluorescent bulb, particularly the cool white type, is a popular choice for many machine vision practitioners (Tao et al., 1995; Abdullah et al., 2001, 2005; Pedreschi et al., 2006). However, care must be taken when using the fluorescent light, as this source is normally AC driven. The 50-Hz fluorescent bulb usually introduces artefacts in the image resulting from the oversampling of the analog-to-digital converter. In order to reduce flickering, high-frequency fluorescent bulbs, operating at a frequency in the range of a few tens of kilohertz, are preferred rather than low-frequency ones.

Apart from the illuminant, the surface geometry is also important in the illumination design. The key factor is to determine whether the surface is specular or diffuse. Light striking a diffuse surface is scattered because of the multitude of surface angles. In comparison, light striking a glossy surface is reflected at the angle of incidence. Therefore, the position of an illuminant is very important in order to achieve high contrast in an image. There are two common geometries for the illuminators: the ring illuminator and the diffuse illuminator (see Figure 1.4).

Figure 1.4 Two possible lighting geometries: (a) the ring illuminator; (b) the diffuse illuminator.

The ring illuminator has the simplest geometry and is generally intended for general purpose applications, especially for imaging flat surfaces. The diffuse illuminator, meanwhile, delivers virtually 180° of diffuse illumination, and is used for imaging challenging reflective objects. Since most food products are basically 3D objects, the diffuse illuminator is well suited for this imaging application. However, there has been some success in using the ring illuminator to solve lighting problems in food engineering. For instance, a ring illuminator together with a 90-kHz ultra high-frequency fluorescent bulb has been found to be effective in the color- and shape-grading of star fruits (Abdullah et al., 2005). In an attempt to produce uniform lighting, Paulsen (1990) mounted a ring light in a cylindrically-shaped diffuse lighting chamber.

Such a set-up is extremely useful for visual inspection of grains and oilseed, with the success rate reaching almost 100 percent.

In spite of the general utility of the ring illuminator, however, the majority of machine vision applications are based on the diffuse illuminator. Heinemann et al. (1994) employed this type of illumination system for the shape-grading of mushrooms. The same system was investigated by Steinmetz et al. (1996) in the quality grading of melons. Both groups of authors have reported successful application of machine vision, with a grading accuracy that exceeds 95 percent. There are many other applications involving diffuse illuminator and computer vision integration. Batchelor (1985) reviewed some of the important factors to be considered when designing a good illumination system.

3.1.2 Electronics

Capturing the image electronically is the first step in digital image processing. Two key elements are responsible for this: the camera and the frame-grabber. The camera converts photons to electrical signals, and the frame-grabber then digitizes these signals to give a stream of data or bitmap image. There are many types of camera, ranging from the older pick-up tubes such as the vidicons to the most recent solid-state imaging devices, such as the Complementary Metal Oxide Silicon (CMOS) cameras. The latter is the dominant technology for cameras, and revolutionized the science of imaging with the invention of the CCD device in 1970. As CCD cameras have less noise, higher sensitivity and a greater dynamic range, they have also become the device of choice for a wide variety of food engineering applications.

In general, the CCD sensor comprises a photosensitive diode and a capacitor connected in parallel. There are two different modes in which the sensor can be operated: passive and active. Figure 1.5 shows the details of the schematics.

Figure 1.5 Sensor operation in (a) passive mode and (b) active mode.

Referring to Figure 1.5, the photodiode converts light into electrical charges, which are then stored in the capacitor. The charges are proportional to the light intensity. In passive mode, these charges are transferred to a bus line when the select signal is activated. In the active mode, charges are first amplified before being transferred to a bus line, thus compensating the limited fill factor of the photodiode. An additional reset signal allows the capacitor to be discharged when an image is rescanned.

Depending on the sensing applications, CCD imagers come in various designs. The simplest form is the linear CCD scanner, which is shown schematically in Figure 1.6a. This design is used mostly in office scanner machines. It consists of a single row of photodiodes, which capture the photons. The sensors are lined up adjacent to a CCD shift register, which does the readout. The picture or document to be scanned is moved, one line at a time, across the scanner by mechanical or optical means. Figures 1.6b and 1.6c show two-dimensional CCD area arrays, which are mostly associated with modern digital cameras. The circuit in Fig. 1.6b portrays the interline CCD architecture, while Figure 1.6c shows that of a frame-transfer imager.

Figure 1.6 Three possible CCD architectures: (a) linear, (b) interline, and (c) frame-transfer.

Basically, the interline CCD comprises of a stack of vertical linear scanners connected by an additional, horizontal shift register that collects and passes on the charge readout from linear scanners, row by row. In the case of frame-transfer architecture, the CCD elements, the entire surfaces of which are covered by photosensitive devices, form the photo-sensing area. It can be seen from Figure 1.6c that the frame-transfer design comprises integration and storage areas, forming the integration and storage frames, respectively. The storage-frame array captures an image and transfers the charge to the adjacent storage-frame array. In this way, the integration array can capture a new image while the storage array reads the previous image. Both interline and frame-transfer architectures are suitable for capturing motion images, whilst the linear scanner is best suited for scanning still pictures. Full-frame CCD cameras with four million pixels and a frame rate of more than 30 frames per second (fps) are now commercially available. Modern CCD cameras come with analog, or digital, or both outputs. The analog signals are conform with the European CCIR (Comite Consultatif International des Radiocommunication) or US RS170 video standards. In spite of a reduced dynamic range, analog cameras work well for slower applications (<20 MHz). For high-speed applications, digital cameras are preferred. These cameras have internal digitization circuitry, and usually produce a parallel digital output. Typically, data are output on a 8- to 32-bit wide parallel bus, clocking rates of up to 40 MHz and utilizing RS422 or RS644 international video standards. For the purpose of camera control and PC interfacing, analog cameras require an analog frame-grabber. Likewise, digital cameras require a digital frame-grabber.

Generally, the frame-grabber comprises signal-conditioning elements, an A/D converter, a look-up table, an image buffer and a PCI bus interface. Figure 1.7 illustrates some of the basic elements of a typical frame-grabber card, although it must be borne in mind that the internal working circuitry of modern and state-of-the-art frame-grabbers is more complex than in the one illustrated. Nevertheless, the basic elements remain relatively the same from one manufacturer to another. The latest digital frame-grabbers may feature sophisticated on-board camera control modules and on the on-the-fly data resequencing capabilities, which are intended for high-speed applications. In addition to PCI bus, some frame-grabbers feature PC104 capability and firewire plug-and-play style. In summary, the three criteria in deciding on the frame-grabber are:

1. The choice of camera

2. Speed requirements

3. Computer choice.

Figure 1.7 General structure of a frame-grabber card, showing some important elements.

3.2 Ultrasound

In addition to computer vision, there is growing interest in using ultrasound for food quality and evaluation. One reason for this is that changes in acoustic properties can be related to density changes in the food product (McClements, 1995). Furthermore, ultrasound has the ability to differentiate between the propagation velocity within various media, and the differences in acoustic impedance between different regions within a given volume. In the past, ultrasound has been used for measuring the moisture content of food products (Steele, 1974), predicting the intramuscular fat content of bovine products (Morlein et al., 2005), and studying and evaluating the turgidity and hydration of orange peel (Camarena and Martínez-Mora, 2005). In addition to these, one of the most widespread and promising ultrasonic applications is for composition measurement. Recent studies have shown that ultrasonic velocity measurements can accurately be used to predict the fat, water, protein, and other chemical compositions of meat-based products (Simal et al., 2003).

The information provided by ultrasound originates from the reflection or transmission of sound waves emitted by an external source. Refraction, absorption, and scattering also play a role, but mainly as factors that degrade the ultrasonic measurements. A typical source/detector unit is based on a piezoelectric crystal resonating between 1 MHz and 10 MHz. The basic physical parameters of importance are the frequency of the wave and the acoustical impedance of the object through which the sound wave travels. The acoustical impedance Z is itself a function of the speed of sound v and the density ρ of the object, and the relationship Z = ρv holds. When an ultrasonic plane wave propagates in a medium with Z = Z1 and is incident normally at an interface with Z = Z2, the intensity reflection and transmission coefficients (αγ and αt) are given in terms of the incident (pi), reflected (pγ), and transmitted (pt) sound pressures (Wells, 1969). Mathematically,

(1.4)

(1.5)

Clearly, from equations (1.4) and (1.5), the greater the difference in impedance at the interface, the greater the amount of energy at the interface will be. Conversely, if the impedance is similar, most of the energy is transmitted. Therefore, it may be deduced that reflections may provide the basis for digital imaging in the same way that a computer vision system generates images. McClements (1995) published useful data on impedance and the interaction processes for a wide range of human tissues, including materials of interest in food technology. For instance, the density of human muscle is typically 1000 kg/m³; the speed of sound within muscle is therefore 1500 m/s. In this case αγ at a muscle–fat interface is approximately 1 percent, and this value increases sharply to 99.9 percent at a skin–air interface. Therefore, this technique generally requires a coupling medium between the test sample and the transducer surface. Couplants such as water, gel, and oil are routinely used for particular inspection situations; however, it must be stressed that the use of a couplant may not always be suitable for some specialized applications, especially when the material property might change or contamination damage result. Gan et al. (2005) have attempted to use a non-contact-based ultrasound inspection technique for evaluating food properties. They used a pair of capacitive devices in air to deliver high ultrasonic energy, with the application of a pulse-compression technique for signal recovery and analysis. The use of such a processing technique is necessary to recover signals buried in noise, as a result of large acoustic impedance and the mismatch between air and the sample. They experimented with this technique in monitoring palm-oil crystallization, employing a chirp signal with a center frequency of 700 kHz and a bandwidth of 600 kHz. From the results, they deduced that both contact and non-contact measurements are well correlated.

The majority of the studies found in the literature rely on the measurement of ultrasonic velocity, because this is the simplest and most reliable ultrasonic measurement. It utilizes an impulse (or sequence of discrete impulses) transmitted into the medium, and the resulting interactions provide raw data for imaging. Here, the transduction can be sensed in two ways – either from the same viewpoint by the partial reflection of the ultrasonic energy; or from an opposing viewpoint by transmission, where the energy is partially attenuated. Whichever is used, the time-of-flight (TOF) and hence the velocity may be estimated from both reflection and transmission sensor data. Figure 1.8 demonstrates the general concept.

Figure 1.8 Ultrasonic measuring system showing (a) essential elements, (b) reflection, and (c) transmission measurements.

In Figure 1.8a, A represents the composite ultrasonic transducer acting as both the transmitter (Tx) and the receiver (Rx) probe; B represents the ultrasonic receiver probe. The interposing area between transducer probes consists of a homogeneous fluid of density distribution f1(x, y). Within the sample are two inhomegeneities, having densities of g1(x, y) and g2(x, y), in which g2(x, y) > g1(x, y) > f1(x, y). The interactions between ultrasound and the sample can be explained as follows. Probe A transmits the ultrasonic wave, which travels in a straight line until it reaches the f1(x, y) and g1(x, y) interface, which causes reflection. This is detected by the same probe, which now acts as a receiver. The amplified signals are fed into the y-plates of the oscilloscope, and a timebase is provided, synchronized to the transmitter pulse. Some of the energy, however, continues to travel until it reaches the f1(x, y) and g2(x, y) interface, where some energy is again reflected and hence detected by A. In similar fashion, some of the remaining energy continues to travel until it reaches probe B, where it is again detected and measured. Consequently, probe A provides a graph, detailing the echo signal, in which the height corresponds to the size of the inhomegeneity and the timebase provides its range or depth. Such a pattern is known as an A-scan (see Figure 1.8b).

Figure 1.8c shows the attenuated transmitted energy as observed by probe B. Both graphs show that information relating to the amplitude of both the transmitted and the reflected pulses can be measured, and this can also be used for imaging. As shown in Figure 1.8, the signals are usually rectified and filtered to present a simple one-dimensional picture, and the timebase can be delayed to allow for a couplant gap.

To provide a full two-dimensional image, the ultrasonic probe must be moved over the surface of the sample. The Tx/Rx probe is connected via mechanical linkages to position transducers, which measure its x and y coordinates and its orientation. In this case, the output signals determine the origin and direction of the probe, while the amplitude of the echo determines the spot brightness. As the probe is rotated and moved over the sample, an image is built and retained in a digital store. This procedure is known as a B-scan, and produces a slice through a sample, normal to the surface. In contrast, a C-scan produces an image of a slice parallel to the surface. In order to produce the C-scan image, the ultrasonic probe must again be moved but this time over the volume of the sample. The time, together with the x and y coordinates of the image displayed, represents the lateral movement of the beam across the plane. By time-gating the echo signals, only those from the chosen depth are allowed to brighten the image. C-scan images may be produced using the same equipment as for B-scanning. Most of the studies in the literature rely on the use of A-scan or B-scan methods, probably because the C-scan image does not provide any additional information which is useful for further characterization.

Regardless of the methods, the ultrasound images generally share at least three common drawbacks:

1. Low image spatial resolution – typically of a few millimeters

2. Low signal-to-noise ratio

3. Many artefacts.

The first of these is related to the wavelength and hence frequency of ultrasound, which typically ranges from 2 to 10 MHz. In order to improve the resolution, some ultrasound devices operate at frequencies higher than this; however, such devices must be used with care because the skin effect increases with increasing frequency. The factors therefore have to be balanced against each other. The second and third drawbacks are due to the coherent nature of the sound wave and the physics of reflection. Any coherent pulse will interfere with its reflected, refracted, and transmitted components, giving rise to speckle, similar to the speckle observed in laser light (Fishbane et al., 1996). On the other hand, reflection occurs when the surface has a normal component parallel to the direction of the incident wave. Interfaces between materials that are parallel to the wave will not reflect the wave, and are therefore not seen in ultrasound images; such parallel interfaces form a hole in the ultrasound image. Despite these drawbacks, the technique is safe and relatively inexpensive. Current research methods tend to eliminate artefacts, improve image contrast, and simplify the presentation of data, and many efforts are being directed towards three-dimensional data acquisition and image representation.

3.3 Infrared

When both computer vision and ultrasound systems fail to produce the desired images, food engineers and technologists could presumably resort to the use of much longer wavelengths for image acquisition. In the region of 700–1000 nm lies the infrared (IR) range, and the technique responsible for generating images with infrared light is known as thermographic photography. Thermographic imaging is based on the simple fact that all objects emit a certain amount of thermal radiation as a function of their temperature. Generally, the higher the temperature of the object, the more IR radiation it emits. A specially built camera, known as an IR camera, can detect this radiation in a way similar to that employed in an ordinary camera for visible light. However, unlike computer vision, thermal imaging does not require an illumination source for spectral reflectance, which can be affected by the varied surface color of a target or by the illumination set-up.

Thermographic signatures of food are very different for different materials, and hence IR imaging has found applications and many other uses in the food industry – such as identification of foreign bodies in food products (Ginesu et al., 2004). Moreover, many major physiological properties of foodstuffs (firmness, soluble-solid content, and acidity) appear to be highly correlated with IR signals, implying that image analysis of IR thermography is suitable for quality evaluation and shelf-life determination of a number of fruit and vegetable products (Gómez et al., 2005). Therefore, thermal imaging offers a potential alternative technology for non-destructive and non-contact image-sensing applications. Good thermographic images can be obtained by leaving the object at rest below the IR camera, applying a heat pulse produced by a flashlight, and monitoring the decreasing temperature as a function of time. Because of different thermal capacities or heat conductivities, the objects will cool down at different speeds; therefore, the thermal conductivity of an object can be measured by the decreasing temperature calculated from a sequence of IR images. Using these relatively straightforward procedures, Ginesu et al. (2004) performed experiments on objects with different thermal properties, aiming to simulate foreign-body contamination in real experiments. Both the long (500-fps) and short (80-fps) sequence modes were used to record the images, enabling the radiation patterns of objects with low and high thermal capacities, respectively, to be monitored. Temperature data were presented in terms of average gray levels computed from 10 × 10 image pixels in the neighborhood of each object. Figure 1.9 shows the results.

Figure 1.9 Decreasing temperature curves of different materials plotted as a function of time (Ginesu et al., 2004).

It can be seen from Figure 1.9 that the cardboard and the wooden stick behave quite differently from other materials, as they appear to be much hotter at the beginning but decrease in temperature rather quickly. This is due to the fact that these materials are dry and light, whereas foods contain a large quantity of water, which heats up more slowly and reaches lower temperatures, thus maintaining the heat for a longer time and cooling down slowly. By plotting and analyzing the absolute differences between radiation curves of different materials, it is possible to distinguish between food and foreign objects.

Theoretically, the existence of such unique thermal signatures of different materials is due to the concept of a black body, defined as an object that does not reflect any radiation. Planck’s law describes the radiation emission from a black body as (Gaossorgues, 1994):

1.6

where h = 6.6256 × 10−34 J s is Planck’s constant, σ = 1.38054 × 10−23 J/K is the Stefan-Boltzman’s constant, c = 2.998 × 10−8 m/s is the speed of light, θ is the absolute temperature in degrees kelvin, and λ is again the wavelength. Usually objects are not black bodies, and consequently the above law does not apply without certain corrections. Non-black bodies absorb a fraction A, reflect a fraction R, and transmit a fraction T. These fractions are selective, depending on the wavelength and on the angle of incident radiation. By introducing the spectral emissivity ε(λ) to balance the absorbance, it can be found that:

(1.7)

and

(1.8)

Using these corrections, equation (1.6) can be simplified, yielding:

(1.9)

This means that the emission coefficient ε(λ) relates the ideal radiation of a black body with real non-black bodies. In summary, an ideal black body is a material that is a perfect emitter of heat energy, and therefore has the emissivity value equal to unity. In contrast, a material with zero emissivity would be considered a perfect thermal mirror. However, most real bodies, including food objects, show wavelength-dependent emissivities. Since emissivity varies with material, this parameter is the important factor in thermographic image formation. For accurate measurement of temperature, the emissivity should be provided manually to the camera for its inclusion in temperature calculation. The function that describes the thermographic image f(x, y) can be expressed as follows:

(1.10)

where x and y are the coordinates of individual image pixels, θ(x, y) is the temperature of the target at image cooordinates (x, y), and ε(x, y) is the emissivity of the sample also at coordinates (x, y). From the computer vision viewpoint, thermographic images are a function of two variables: the temperature and emissivity variables. Contrast in thermographic images may be the result of either different temperatures of different objects on the scene, or different emissivities of different objects with the same temperature. It can also be the combination of both temperature and emissivity variations.

As mentioned previously, the infrared or thermographic cameras operate at wavelengths as long as 14 000 nm (or 14 μm). The infrared sensor array is equivalent to the CCD in the ordinary camera; sensors with a resolution of 160 × 120 pixels or higher are widely available, and their response time is sufficient to provide live thermographic video at 25 frames per second. However, unlike sensors used in conventional imaging systems, the process of image formation and acquisition in thermographic cameras is quite complex. Broadly speaking, thermographic cameras can be divided into two types: those with cooled infrared image detectors and those without cooled detectors. These are discussed in the following section.

3.3.1 Cooled infrared detectors

Cooled IR detectors are typically contained in a vacuum-sealed case and cryogenically cooled. This greatly increases their sensitivity, since their temperature is much lower than that of the objects from which they are meant to detect radiation. Typically, cooling temperatures range from −163°C to −265°C, with −193°C being the most common. In a similar way to common digital cameras, which detect and convert light to electrical charge, the IR detectors detect and convert thermal radiation to electrical signals. In the case of IR cameras, cooling is needed in order to suppress thermally emitted dark currents. A further advantage of cooling is suppression of noise from ambient radiation emitted by the apparatus. Materials used for IR detection include liquidhelium cooled bolometers, photon-counting superconducting tunnel junction arrays, and a wide range of cheaper, narrow-gap semiconductor devices. Mercury cadmium telluride (HgCdTe), indium antimonide (InSb) and indium gallium arsenide (InGaAs) are the most common types of semiconductor IR detectors, with newer compositions such as mercury manganese telluride (HgMnTe) and mercury zinc telluride (HgZnTe) currently being developed. However, the HgCdTe detector and its extension remains the most common IR detector. The principle of operation of an HgCdTe-based detector is illustrated in Figure 1.10.

Figure 1.10 Hybrid focal plane architecture for HgCdTe-based IR detector showing (a) cell structure and (b) equivalent circuit.

In Figure 1.10, the sensor is represented by a detector diode which is mechanically bonded to a silicon (Si) multiplexer for the read-out operation. An electrical connection is required between each pixel and the rest of the circuitry. This is formed by the heat- and pressure-bonding of an indium bump or solder bond. Row and column shift registers allow sequential access to each pixel. Similarly to other semiconductor devices, this type of sensor is constructed using modern fabrication technologies such as vapor deposition epitaxy (Campbell, 2001). In this method, the diode is made by depositing CdTe on sapphire followed by liquid epitaxy growth of HgCdTe. A complete HgCeTe IR detector system usually comprises a small printed circuit board (PCB), complete with a digital signal processor chip (DSP) and an optical system responsible for focusing the scene on to the plane of array. At present, large two-dimensional arrays comprising 2048 × 2048 pixels, with each pixel 18 μm in size, assembled on a 40 × 40-mm device and with a complete infrared camera system, are commercially available. They operate in the bands 3–5 μm or 8–12 μm, and need cooling at −196°C.

There are different ways to cool the detectors – mainly by using liquefied gas, a cryogenic engine, gas expansion, or the thermoelectric effect. The most common method is cryogenic cooling, employing liquefied gas stored in a vacuum called a Dewar (named after Sir James Dewar, a Scottish scientist who successfully liquefied hydrogen for the first time in 1892). Figure 1.11 shows the construction of a typical Dewar, highlighting all the important elements. Typically, the sensor is mounted directly on the cold surface, with a cold shield and infrared transparent window. Usually a protective coating such as zinc sulfide is applied on to the surface of HgCeTe in order to increase its lifespan. The most commonly used and cheapest liquefied gas is liquid nitrogen, which provides a sustainable cold temperature of −196°C without regular filling.

Figure 1.11 Schematic diagram of a typical Dewar.

Another common method of achieving cooling is through the Joule–Thompson gas expansion method. High-pressure gas such as nitrogen or argon produces droplets of liquid nitrogen at −187°C following quick expansion. Compared to the Dewar, this method is noisy and cumbersome. When refilling is not practical, such as for applications in remote areas, a cooling method using a closed Stirling cycle can be employed. This machine cools through the repetitive compression and expansion cycles of a gas piston, and is therefore again cumbersome compared to the Dewar. Another more practical approach to cooling is by thermoelectric elements, based on the Peltier–Thompson effect (Fraden, 1997). This method utilizes a junction of dissimilar metals carrying a current; the temperature rises or falls depending on the direction of the current. Current flowing in one direction results in the Peltier effect, and current flowing in the opposite direction produces the Thompson effect by the same law of physics. Unfortunately, Peltier elements are unattractive for temperatures below −73°C owing to high current consumption. In spite of this drawback, the thermoelectric cooling method involves no moving parts, and is quiet and reliable. For these reasons, it is widely used in IR cameras.

3.3.2 Uncooled IR detectors

As the name implies, uncooled thermal cameras use sensors that operate at room temperature. Uncooled IR sensors work by changes in resistance, voltage or current when exposed to IR radiation. These changes are then measured and compared with the values at the operating temperature of the sensor. Unlike cooled detectors, uncooled IR cameras can be stabilized at an ambient temperature, and thus do not require bulky, expensive cryogenic coolers. This makes such IR cameras smaller and less costly. Their main disadvantages are lower sensitivity and a longer response time, but these problems have almost been solved with the advent of surface micro-machining technology. Most uncooled detectors are based on pyroelectric materials or microbolometer technology. Pyroelectricity is the ability of some materials to generate an electrical potential when heated or cooled. It was first discovered in minerals such as quartz, tourmaline, and other ionic crystals. The first generation of uncooled thermal cameras looked very similar to the conventional cathode ray tube, apart from the face plate and target material (see Figure 1.12). As infrared signals impinge on the pyroelectric plate, the surface temperature of this plate changes. This in turn induces the charge, which accumulates on the pyroelectric material. The electron beam scans this material, and two things may happen depending on whether there is an absence or presence of charge. In the absence of charge (i.e. no radiation), the electron beam is deflected toward the mesh by the action of the x and y deflection plates. In the presence of charge, the electron beam is focused on the spot, thus causing current to flow into an amplifier circuit. In this way a video signal is built as the electron beam scans over the entire surface of the pyroelectric plate. Since the accumulation of charge only occurs when the temperature of the pyroelectric material changes, the pyroelectric tube is only suitable for imaging dynamic occurrences. This effect will benefit certain applications, such as monitoring drying process, where only the fast changes of temperature are recorded (Fito et al., 2004).

Figure 1.12 Schematic diagram of the first-generation pyroelectric tube.

With the advent of semiconductor technology, it is now possible to produce pyroelectric solid-state arrays with resolution reaching 320 × 240 pixels. This type of camera offers high detectivity, but produces images at a relatively low speed (typically 1 Hz). Furthermore, absolute temperature measurement often requires individual calibration of each element, which significantly slows down the image acquisition time. However, the main advantage lies with its ability to produce an image without the need for cooling. This makes it suitable for a wide range of non-destructive applications, especially in industry.

Another type of IR camera is based on microbolometer technology. Theoretically, a microbolometer is a monolithic sensor capable of detecting infrared radiation through the direct or indirect heating of a low-mass, temperature-dependent film. Popular materials include thermistors with high temperature coefficients of resistance, such as vanadium oxide (VOx), silicon devices such as the Schottky barrier diode and transistor, and thermoelectrics such as the silicon p–n junctions. One example of the bolometer-type uncooled infrared focal plane array (IRFPA), with a 320 × 240-pixel array and operating at a frame rate of 60 Hz, has been investigated for use in industry (Oda et al., 2003). Figure 1.13 shows the schematic structure of each bolometer pixel. The pixel is divided into two parts; a silicon readout integrated circuit (ROIC) in the lower part, and a suspended microbridge structure in the upper part. The two parts are separated by a cavity. The microbridge structure is composed of a diaphragm and supported by two beams, thereby thermally isolating the former from the latter heat sink.

Figure 1.13 Schematic representation of a bolometer detector showing (a) the cross-sectional view and (b) the plan view of each bolometer pixel.

Manufacture of microbolometers such as the one shown in Figure 1.13 uses micro-electromechanical techniques, originally developed at Bell Labs for air-bridge isolation integrated circuits. They are carefully engineered so that part of the IR radiation is absorbed by the silicon passivation layers in the diaphragm and part is transmitted. The transmitted radiation is perfectly reflected by the reflecting layer, and is again absorbed by the passivation layers. In this way, more than 80 percent of the incident IR radiation is absorbed. The absorbed radiation heats the diaphragm and changes the bolometer resistance. Supplying a bias current enables the resistance change to be converted to voltage and detected by ROIC. The analog signal voltage of the ROIC is digitized by analog-to-digital conversion of the receiving circuits. These data are first corrected for non-uniformity in bolometer responsivity, and are then adjusted for video output. The pixel size of such a detector is 37 × 37 μm, and the fill factor is about 72