NI Preprocessing

With focus on NIR

Åsmund Rinnan , Frans van den Berg,

Rasmus Bro, Søren Balling Engelsen, Lars
Nørgaard , Jonas Thygesen
 Introduction
Techniques

Introduction
Summary

NI
• How often is it used?
• When is it used?
• What methods are used?
• Why is it used?

• NIR is a measurement technique

“screaming” for preprocessing
 Introduction
Introduction
Techniques
Summary

NI
NIR

Fructose Glucose
0 100
50 50
75 25
 Introduction
Introduction
Techniques
Summary

NI
Techniques
• Correction of light • Baseline-correction
scatter • Normalization
– MSC/ ISC • Detrend
– PMSC
• Use of reference values
– EMSC/ EISC
– O-PLS
• SNV – OSC
• Derivation – OS
– Norris&Williams – SIS
– Savitsky&Golay
 Introduction
Introduction
Techniques
Summary

NI
Why

Baseline with a slope/ curve

Nonlinearity

A “fat” baseline
 Introduction

Techniques

ISC
MSC Summary

NI
• Offset
Raw spectrum
• Normalization

Reference

1. P Geladi, D MacDougal, H Martens (1985): Linearization and scatter correction for near-infrared reflectance spectra of meat, Applied Spectroscopy, 39, 491-500
 Introduction

Techniques

MSC/ ISC Summary

NI
• Offset
• Normalization
 Introduction

Techniques

EISC Summary

NI
• Offset
• Baseline
• Normalization
x new  f  x org   f  λ   f  x known   e

f  x org   a  1  b  x org  c  x org

2

f  λ   a  λ  b  λ  c  log λ  2

f  x known   h T  G
1. H. Martens and E. Stark (1991): Extended multiplicative signal correction and spectral interference subtraction: new preprocessing methods for near infrared
spectroscopy, Journal of Pharmaceutical and Biomedicinal Analysis, 9, 625-635
2. H Martens, JP Nielsen, SB Engelsen (2003): Light scattering and light absorbance separated by extended multiplicative signal correction. The application to
near-infrared transmission analysis of powder mixtures, Analytical chemistry, 75, 394-404
3. DK Pedersen, H Martens, JP Nielsen, SB Engelsen (2002): Near-Infrared Absorption and Scattering Separated by Extended Inverse Signal Correction (EISC):
Analysis of Near-Infrared Transmittance Spectra of Single Wheat, Applied Spectroscopy, 56, 1206-1214
4. Thennadil, SN, Martin, EB (2005): Empirical preprocessing methods and their impact on NIR calibrations - a simulation study, Journal of Chemometrics, 19, 77-89
 EMSC
Introduction

Techniques
Summary

NI
(w/ 2. degree & wavelength)
• Offset
• Baseline
• Normalization
 Introduction

Techniques

PxSC Summary

NI
• Offset
X = EM
EI • Baseline
M • Normalization
I

1. Isaksson, T.; Kowalski, B. R. (1993): Piece-wise multiplicative scatter correction applied to Near-Infrared diffuse transmittance data from meat products, Applied
Spectroscopy, 47, 702-09
 PMSC
Introduction

Techniques
Summary

NI
(window size = 35)
• Offset
• Normalization
 Introduction

Techniques

SNV Summary

NI
• Offset
x org  x org • Normalization
x new 
x

1. R.J. Barnes, M.S. Dhanoa, S.J. Lister (1989): Standard Normal Variate Transformation and De-trending of Near-Infrared Diffuse Reflectance Spectra, Applied
Spectroscopy, 43, 772-777
 Introduction

Techniques

SNV vs MSC Summary

NI
x org  a
x new 
b
MSC SNV
 Introduction

Techniques

Detrend Summary

NI
n • Offset
xorg  xnew   ai    i 1
• Baseline
i 1

1. R.J. Barnes, M.S. Dhanoa, S.J. Lister (1989): Standard Normal Variate Transformation and De-trending of Near-Infrared Diffuse Reflectance Spectra, Applied
Spectroscopy, 43, 772-777
 Derivation
Introduction

Techniques
Summary

NI
Savitsky-Golay
• Offset
xnew, f  x org,f m,f m  o
T • Baseline

CT
o T  d!
C C
 m ...   m i 
 i
  m  1 ...    m  1  
C   ... 0 ... 
 
 m 1 ...  m  1 i 
 m ... mi 
 
1. A Savitsky, MJE Golay (1964): Smoothing and differentiation of data by simplified least squares procedures, Analytical Chemistry, 36 (8), 1627-1639
2. P.A. Gorry (1990): General Least-Squares smoothing and differentiation by the convolution (Savitsky-Golay) method, Analytical Chemistry, 62, 570-573
 Derivation
Introduction

Techniques
Summary

NI
Savitsky-Golay
• Offset
• Baseline
 Derivation
Introduction

Techniques
Summary

NI
Savitsky-Golay
• Offset
• Baseline
 Derivation
Introduction

Techniques
Summary

NI
Norris-Williams
• Offset
xnew, f • Baseline
optional : xnew, f  n • Normalization
x
i 1
new , f , i

xorg , f  2  xorg , f 1
xnew, f ,1st  xsmooth , f 1  xsmooth , f 
3
xorg , f  2  xorg , f 1  xorg , f 1  xorg , f  2
xnew, f , 2 nd  xsmooth , f 1  xsmooth , f 1  2  xsmooth , f 
3
1

x org , f  i
xsmooth , f  i  1
3

1. Norris, KH (1983): in Food Research and Data Analysis (Eds: Martens H and Russwurm H Jr), Applied Science, London, 95-114
2. Norris KH, Williams PC (1984): Optimization of mathematical treatments of raw near-infrared signal in the measurement of protein in hard red spring wheat: I.
Influence of particle size, Cereal Chemistry, 61, 158-165
 Derivation
Introduction

Techniques
Summary

NI
Norris-Williams
• Offset
• Baseline
• Normalization
 Derivation
Introduction

Techniques
Summary

NI
Norris-Williams
• Offset
• Baseline
• Normalization
 Introduction
Techniques

Preprocessing Summary

NI
• PMSC: Sensitivite to window size
• EMSC: Small effect on baseline  Detrend
is better
• MSC and SNV are practical the same
• Savitsky-Golay is more robust than Norris-
Williams
 Introduction
Techniques

Preprocessing Summary

NI
(P)MSC/ISC/SNV
(P)EMSC/EISC
Detrend
Deriveration
Offset Baselinj Normalization

x
x
x
x
x
x
e
x
x

x
NI
Thanks for the attention

www.models.life.ku.dk