University of Basel
University of Basel
University of Basel

# Image Restoration

### Introduction to Signal and Image Processing

##### April 19th/26th, 2016
Contents Image Restoration

# Contents

• 2Abstract
• 1 Image Degradation & Restoration Model
• 4Image Degradation & Restoration Model
• 5Image Degradation & Restoration Model (2)
• 2 Noise Models
• 7Noise Models
• 8Gaussian Noise
• 9Rayleigh Noise
• 10Erlang Noise
• 11Exponential Noise
• 12Uniform Noise
• 13Impulse (Salt-and-Pepper) Noise
• 14Noise Models
• 15Visual Comparison of Different Noise Sources
• 16Periodic Noise
• 17Estimation of Noise Parameters (1)
• 18Estimation of Periodic Noise Parameters
• 3 Restoration in the Precense of Noise Only
• 20Restoration in the Precense of Noise Only
• 4 Periodic Noise Reduction by Frequency Domain Filtering
• 4.1 Bandreject Filter
• 23Bandreject Filter
• 24Bandreject Filter Example
• 4.2 Bandpass Filter
• 26Bandpass Filter
• 27Bandpass Filter Example
• 4.3 Notch Filter
• 29Notch Filter
• 30Notch Filter (2)
• 31Ideal Notch Filter D_0=16 Example
• 32Butterworth Notch Filter D_0=64 Example
• 33Gaussian Notch Filter D_0=64 Example
• 34General Notch Filter Example
• 4.4 Optimum Notch Filtering
• 36Optimum Notch Filtering
• 37Optimum Notch Filtering Principle
• 38Optimum Notch Filter Example
• 39Optimum Notch Filter Example with Pixel Increments
• 40Optimum Notch Filter Example with Pixel Increments (2)
• 5 Estimating the Degradation Function
• 43Estimating the Degradation Function (2)
• 44Estimating by Image Observation
• 45Estimating by Experimentation
• 46Estimating by Mathematical Modelling
• 47Illustration of the Atmospheric Turbulence Model
• 6 Inverse Filtering
• 6.1 Direct Inverse Filtering
• 50Direct Inverse Filtering
• 51Two Practical Approaches for Inverse Filtering
• 52Artificial Inverse Filtering Example
• 53Artificial Inverse Filtering Example (2)
• 54Inverse Filtering Example (2)
• 6.2 Wiener Filtering
• 56Wiener Filtering
• 57Wiener Filtering (2)
• 58Parametric Wiener Filter
• 59Artificial Wiener Filtering Example
• 60Wiener Filtering Example with Little Noise
• 61Wiener Filtering Example with Medium Noise
• 62Wiener Filtering Example with Heavy Noise
• 63Comparison Param. Wiener vs. Wiener Filtering
• 64Drawback of the Wiener Filter
• 6.3 Geometric Mean Filter
• 66Geometric Mean Filter
• 6.4 Constrained Least Squares Filtering
• 68Constrained Least Squares (Regularised) Filtering
• 69Constrained Least Squares Filtering (2)
• 70Constrained Least Squares Filtering (3)
• 71Optimal Selection for Gamma
• 72Artificial Constrained Least Squares Filter Example
• 73Constrained Least Squares Filter Example
Image Restoration

# (2) Abstract

As in image enhancement the goal of restoration is to improve an image for further processing. In contrast to image enhancement that was subjective and largely based on heuristics, restoration attempts to reconstruct or recover an image that has been distorted by a known degradation phenomenon. Restoration techniques thus try to model the degradation process and apply the inverse process in order to reconstruct the original image.

# Outline (Image Degradation & Restoration Model)

1. Image Degradation & Restoration Model (2 Slides)
2. Noise Models (12 Slides)
3. Restoration in the Precense of Noise Only (1 Slides)
4. Periodic Noise Reduction by Frequency Domain Filtering (15 Slides)
1. Bandreject Filter (2 Slides)
2. Bandpass Filter (2 Slides)
3. Notch Filter (6 Slides)
4. Optimum Notch Filtering (5 Slides)
5. Estimating the Degradation Function (6 Slides)
6. Inverse Filtering (21 Slides)
1. Direct Inverse Filtering (5 Slides)
2. Wiener Filtering (9 Slides)
3. Geometric Mean Filter (1 Slides)
4. Constrained Least Squares Filtering (6 Slides)
Image Degradation & Restoration Model Image Restoration

# (4) Image Degradation & Restoration Model

The degradation process is generally modelled as a degradation function $H$ and the additive noise term $\eta(x,y)$ together they yield $g(x,y)$.

Given $g(x,y)$, some knowledge about $H$, and $\eta(x,y)$ it is the objective of restoration to estimate $\hat{f}(x,y)$. Of course this estimate should be as close as possible to $f(x,y)$.

Image Degradation & Restoration Model Image Restoration

# (5) Image Degradation & Restoration Model (2)

If the degradation function $H$ is a linear, shift-invariant process, then the degraded image is given in the spatial domain by

 $$g(x,y)=h(x,y)\astf(x,y)+\eta(x,y)$$ (6.1)

where $h(x,y)$ is the spatial representation of the degradation function, and $\ast$ indicates convolution.

As we know from the Convolution Theorem this can be rewritten in the frequency domain

 $$G(u,v)=H(u,v)F(u,v)+N(u,v)$$ (6.2)

where the capital letters are the Fourier transforms of the respective function in Eq 6.1.

# Outline (Noise Models)

1. Image Degradation & Restoration Model (2 Slides)
2. Noise Models (12 Slides)
3. Restoration in the Precense of Noise Only (1 Slides)
4. Periodic Noise Reduction by Frequency Domain Filtering (15 Slides)
1. Bandreject Filter (2 Slides)
2. Bandpass Filter (2 Slides)
3. Notch Filter (6 Slides)
4. Optimum Notch Filtering (5 Slides)
5. Estimating the Degradation Function (6 Slides)
6. Inverse Filtering (21 Slides)
1. Direct Inverse Filtering (5 Slides)
2. Wiener Filtering (9 Slides)
3. Geometric Mean Filter (1 Slides)
4. Constrained Least Squares Filtering (6 Slides)
Noise Models Image Restoration

# (7) Noise Models

The principle sources of noise in digital images arise during image acquisition and/or transmission.

The performance of imaging sensors are affected by a variety of factors during acquisition, such as

• Environmental conditions during the acquisition
• Light levels (low light conditions require high gain amplification)
• Sensor temperature (higher temp implies more amplification noise)

Images can also be corrupted during transmission due to interference in the channel for example

• Lightning or other
• Atmospheric disturbances

Depending on the specific noise source, a different model must be selected that accurately reproduces the spatial characteristics of the noise.

Noise Models Image Restoration

# (8) Gaussian Noise

Because of its mathematical tractability in both the spatial and frequency domain, Gaussian noise models (aka normal distribution) are used frequently in practice.

The PDF (Probability density function) of a Gaussian random variable $z$ is given by

 $$p_G(z)=\frac{1}{\sqrt{2\pi}\sigma}e^{-(z-\mu)^2/(2\sigma^2)}$$ (6.3)

where $z$ represents the grey level, $\mu$ the mean value and $\sigma$ the standard deviation.

Fig 6.2: Gaussian probability density function $p_G(z)$
The application of the Gaussian model is so convenient that it is often used in situations in which they are marginally applicable at best.

Noise Models Image Restoration

# (9) Rayleigh Noise

The PDF of Rayleigh noise is defined by

 $$p_R(z)=\begin{cases}\frac{2}{b}(z-a)e^{-\frac{(z-a)^2}{b}}&\mbox{for}z\gea\\[2ex]0&\mbox{for}z (6.4) the mean and variance are given by  $\begin{eqnarray*}\mu&=&a+\sqrt{\pib/4}\\[2ex]\sigma^2&=&\frac{b(4-\pi)}{4}\end{eqnarray*}$ (6.5) Fig 6.3: Rayleigh probability density function $p_R(z)$ As the shape of the Rayleigh density function is skewed it is useful for approximating skewed histograms. Noise Models Image Restoration # (10) Erlang Noise The PDF of the Erlang noise is given by  $\[p_E(z)=\begin{cases}\frac{a^bz^{b-1}}{(b-1)!}e^{-az}&\mbox{for}z\ge0\\[2ex]0&\mbox{for}z<0\end{cases}$$ (6.6)

where $a>0$, $b$ is a positive integer. The mean and variance are then given by

 $\begin{eqnarray*}\mu&=&\frac{b}{a}\$2ex]\sigma^2&=&\frac{b}{a^2}\end{eqnarray*}$ (6.7) Fig 6.4: Erlang probability density function $p_E(z)$ Noise Models Image Restoration # (11) Exponential Noise The PDF of the exponential noise is given by  $\[p_{exp}(z)=\begin{cases}ae^{-az}&\mbox{for}z\ge0\\[2ex]0&\mbox{for}z<0\end{cases}$$ (6.8)

where $a>0$. The mean and variance of the density function are

 $\begin{eqnarray*}\mu&=&\frac{1}{a}\$2ex]\sigma^2&=&\frac{1}{a^2}\end{eqnarray*}$ (6.9) Fig 6.5: Probability density function $p_{exp}(z)$ of exponential noise The exponential noise model is a special case of the Erlang noise model with $b=1$. Noise Models Image Restoration # (12) Uniform Noise The PDF of the uniform noise is given by  $\[p_U(z)=\begin{cases}\frac{1}{b-a}&\mbox{for}a\lez\leb\\[2ex]0&\mbox{forotherwise}\end{cases}$$ (6.10)

The mean and variance of the density function are given by

 $\begin{eqnarray*}\mu&=&\frac{a+b}{2}\$2ex]\sigma^2&=&\frac{(b-a)^2}{12}\end{eqnarray*}$ (6.11) Fig 6.6: Probability density function $p_U(z)$ of uniform noise Noise Models Image Restoration # (13) Impulse (Salt-and-Pepper) Noise The PDF of bipolar impulse noise model is given by  $\[p_I(z)=\begin{cases}P_a&\mbox{for}z=a\\[2ex]P_b&\mbox{for}z=b\\[2ex]0&\mbox{otherwise}\end{cases}$$ (6.12)

Fig 6.7: Probability density function $p_I(z)$ of the bipolar impulse noise model

If $b>a$, grey-level $b$ appears as a light dot (salt) in the image. Conversely, $a$ will appear as dark dot (pepper). If either $P_a,P_b$ is zero, the PDF is called unipolar.

Because impulse corruption is generally large compared to the signal strength, the assumption is usually that $a$ and $b$ are digitised as saturated values thus black (pepper) and white (salt).

Noise Models Image Restoration

# (14) Noise Models

The presented noise models provide a useful tool for approximating a broad range of noise corruption situations found in practice. For example:

• Gaussian noise: arises in images due to sensor noise caused by poor illumination and/or high temperature, and electronic circuit noise
• The Rayleigh noise model is used to characterise noise phenomena in range images
• Exponential and Erlang noise models find their application in Laser imaging
• Impulse noise takes place in situations where high transients (faulty switching) occurs

Noise Models Image Restoration

# (15) Visual Comparison of Different Noise Sources

Although the corresponding histograms show close resemblance to the PDF of the respective noise model, it is virtually impossible to select the type of noise causing the degradation from the image alone.

 Gaussian noise model Rayleigh noise model Erlang noise model Exponential noise model Uniform noise model Impulse noise model (Salt & Pepper)

Noise Models Image Restoration

# (16) Periodic Noise

 Periodic noise typically arises from electrical or electromechanical interference during image acquisition and is spatial dependent. Fig 6.8: Periodic noise on a satellite image of Pompeii Fig 6.9: Spectrum of the Pompeii image with periodic noise (peaks of the periodic noise visually enhanced)

Noise Models Image Restoration

# (17) Estimation of Noise Parameters (1)

 In rare cases the parameters of noise PDFs may be known from sensor specifications, but it is often necessary to estimate them for a particular imaging arrangement. If the imaging system is available, one simple way to study noise characteristics would be to capture multiple images of homogeneous environments, such as solid grey cards. Fig 6.10: Sample histogram of a small patch of Erlang noise

In many cases the noise parameters have to be estimated from already captured images. The simplest way is to estimate the mean and variance of the grey levels from small patches of reasonably constant grey level.

Noise Models Image Restoration

# (18) Estimation of Periodic Noise Parameters

 The parameters of periodic noise are typically estimated by inspection of the Fourier spectrum. The frequency spikes can often be detected by visual analysis. Automatic analysis is possible if the noise spikes are either exceptionally pronounced or a priori knowledge about their location is known. Fig. 6.11: Fig. 6.12:

# Outline (Restoration in the Precense of Noise Only)

1. Image Degradation & Restoration Model (2 Slides)
2. Noise Models (12 Slides)
3. Restoration in the Precense of Noise Only (1 Slides)
4. Periodic Noise Reduction by Frequency Domain Filtering (15 Slides)
1. Bandreject Filter (2 Slides)
2. Bandpass Filter (2 Slides)
3. Notch Filter (6 Slides)
4. Optimum Notch Filtering (5 Slides)
5. Estimating the Degradation Function (6 Slides)
6. Inverse Filtering (21 Slides)
1. Direct Inverse Filtering (5 Slides)
2. Wiener Filtering (9 Slides)
3. Geometric Mean Filter (1 Slides)
4. Constrained Least Squares Filtering (6 Slides)
Restoration in the Precense of Noise Only Image Restoration

# (20) Restoration in the Precense of Noise Only

When noise is the only image degradation present in an image, thus $H(u,v)=1$, Eqs 6.1 & 6.2 become

 $$g(x,y)=f(x,y)+\eta(x,y)$$ (6.13)

and

 $$G(u,v)=F(u,v)+N(u,v)$$ (6.14)

In case of periodic noise, it is usually possible to estimate $N(u,v)$ from the spectrum of $G(u,v)$ and subtract it to obtain an estimate of the original image $f(x,y)$.

But estimating the noise terms $\eta(x,y)$ is unreasonable, so subtracting them from $g(x,y)$ is impossible. Spatial filtering is the method of choice in situations when only additive noise is present.

# Outline (Periodic Noise Reduction by Frequency Domain Filtering)

1. Image Degradation & Restoration Model (2 Slides)
2. Noise Models (12 Slides)
3. Restoration in the Precense of Noise Only (1 Slides)
4. Periodic Noise Reduction by Frequency Domain Filtering (15 Slides)
1. Bandreject Filter (2 Slides)
2. Bandpass Filter (2 Slides)
3. Notch Filter (6 Slides)
4. Optimum Notch Filtering (5 Slides)
5. Estimating the Degradation Function (6 Slides)
6. Inverse Filtering (21 Slides)
1. Direct Inverse Filtering (5 Slides)
2. Wiener Filtering (9 Slides)
3. Geometric Mean Filter (1 Slides)
4. Constrained Least Squares Filtering (6 Slides)

# Bandreject Filter

Bandreject Filter Image Restoration

# (23) Bandreject Filter

Bandreject filters remove or attenuate a band of frequencies around the origin in the Fourier domain.

Ideal Bandreject Filter
 $$H(u,v)=\begin{cases}1&\mbox{if}D(u,v)D_0+\frac{W}{2}\\\end{cases}$$ (6.15)

Fig 6.13: Ideal bandreject filter
Butterworth Bandreject Filter
 $$H(u,v)=\frac{1}{1+\left[\frac{D(u,v)W}{D^2(u,v)-D_0^2}\right]^{2n}}$$ (6.16)

Fig 6.14: Butterworth bandreject filter
Gaussian Bandreject Filter
 $$H(u,v)=1-e^{-\frac{1}{2}\left[\frac{D^2(u,v)-D_0^2}{D(u,v)W}\right]^2}$$ (6.17)

Fig 6.15: Gaussian bandreject filter

Bandreject Filter Image Restoration

# (24) Bandreject Filter Example

 Fig 6.16: Aerial image of Pompeii with periodic noise Fig 6.17: Ideal bandreject filter $D0=100,W=6$ Fig 6.18: Butterworth bandreject filter $D0=100,W=6$ Fig 6.19: Gaussian bandreject filter $D0=100,W=6$

# Bandpass Filter

Bandpass Filter Image Restoration

# (26) Bandpass Filter

The Bandpass filter performs the opposite of a bandreject filter and thus lets pass the frequencies in a narrow band of width $W$ around $D_0$. The transfer function $H(u,v)$ can be obtained from a corresponding bandreject filter by using the equation

 $$H_{bp}=1-H_{br}$$ (6.18)

The perspective plots of the corresponding filters are depicted in Fig 6.20.

 (a) (b) (c) Fig 6.20 (a) Ideal bandpass filter, (b) Butterworth bandpass filter, and (c) Gaussian bandpass filter

Bandpass Filter Image Restoration

# (27) Bandpass Filter Example

 Fig 6.21: Aerial image of Pompeii with periodic noise Fig 6.22: Ideal bandpass filter $D0=100,W=6$ Fig 6.23: Butterworth bandpass filter $D0=100,W=6$ Fig 6.24: Gaussian bandpass filter $D0=100,W=6$

# Notch Filter

Notch Filter Image Restoration

# (29) Notch Filter

A Notch filter rejects or passes frequencies in predefined neighbourhoods about a centre frequency. Due to the symmetry of the Fourier transform, notch filters must appear in symmetric pairs about the origin (except the one at the origin).

The transfer functions of the ideal, Butterworth, and Gaussian notch reject filter of radius (width) $D_0$ with centres at $(u_0,v_0)$ and, by symmetry, at $(-u_0,-v_0)$, are

Ideal Notch Filter
 $$H(u,v)=\begin{cases}0&\mbox{if}D_1(u,v)\leD_0\mbox{or}D_2(u,v)\leD_0\\[2ex]1&\mbox{otherwise}\end{cases}$$ (6.19)

Fig 6.25: Ideal notch filter
Butterworth Notch Filter
 $$H(u,v)=\frac{1}{1+\left[\frac{D_0^2}{D_1(u,v)D_2(u,v)}\right]^{n}}$$ (6.20)

Fig 6.26: Butterworth notch filter
Gaussian Notch Filter
 $$H(u,v)=1-e^{-\frac{1}{2}\left[\frac{D_1(u,v)D_2(u,v)}{D_0^2}\right]}$$ (6.21)

Fig 6.27: Gaussian notch filter

where

 $\begin{eqnarray*}D_1(u,v)&=&\left[(u-M/2-u_0)^2+(v-N/2-v_0)^2\right]^\frac{1}{2}\$2ex]D_2(u,v)&=&\left[(u-M/2+u_0)^2+(v-N/2+v_0)^2\right]^\frac{1}{2}\end{eqnarray*}$ (6.22) Notch Filter Image Restoration # (30) Notch Filter (2) Similar to the bandpass/bandreject filters, the Notch reject filters can be turned into Notch pass filters with the relation  $\[H_{np}(u,v)=1-H_{nr}(u,v)$$ (6.23)

where $H_{np}(u,v)$ is the transfer function of the notch pass filter, and $H_{nr}(u,v)$ the corresponding Notch reject filter.

If $u_0=v_0=0$
• the Notch reject filter becomes a highpass filter and
• the Notch pass filter becomes a lowpass filter.

Notch Filter Image Restoration

# (31) Ideal Notch Filter D_0=16 Example

 Fig 6.28: Pompeii aerial image with periodic noise Fig 6.29: Notch reject filtered image Fig 6.30: Notch pass filtered image (contrast enhanced)

Notch Filter Image Restoration

# (32) Butterworth Notch Filter D_0=64 Example

 Fig 6.31: Pompeii aerial image with periodic noise Fig 6.32: Notch reject filtered image Fig 6.33: Notch pass filtered image (contrast enhanced)

Notch Filter Image Restoration

# (33) Gaussian Notch Filter D_0=64 Example

 Fig 6.34: Pompeii aerial image with periodic noise Fig 6.35: Notch reject filtered image Fig 6.36: Notch pass filtered image (contrast enhanced)

Notch Filter Image Restoration

# (34) General Notch Filter Example

 Fig 6.37: Original Mariner 6 martian image Fig 6.38: Log Fourier spectra of the image Fig 6.39: Notch filtered log spectra Fig 6.40: Notch filtered image

# Outline (Optimum Notch Filtering)

1. Image Degradation & Restoration Model (2 Slides)
2. Noise Models (12 Slides)
3. Restoration in the Precense of Noise Only (1 Slides)
4. Periodic Noise Reduction by Frequency Domain Filtering (15 Slides)
1. Bandreject Filter (2 Slides)
2. Bandpass Filter (2 Slides)
3. Notch Filter (6 Slides)
4. Optimum Notch Filtering (5 Slides)
5. Estimating the Degradation Function (6 Slides)
6. Inverse Filtering (21 Slides)
1. Direct Inverse Filtering (5 Slides)
2. Wiener Filtering (9 Slides)
3. Geometric Mean Filter (1 Slides)
4. Constrained Least Squares Filtering (6 Slides)
Optimum Notch Filtering Image Restoration

# (36) Optimum Notch Filtering

Images derived from electro-optical scanners, such as those used in space and aerial imaging, are sometimes corrupted by coupling and amplification of low-level signals in the scanners' circuitry. The resulting images tend to contain substantial periodic noise. The interference pattern are, however, in practice not always as clearly defined as in the samples shown thus far.

The first step in Optimum notch filtering is to find the principal frequency components and placing notch pass filters at the location of each spike, yielding $H(u,v)$. The Fourier transform of the interference pattern is thus given by

 $$N(u,v)=H(u,v)G(u,v)$$ (6.24)

where $G(u,v)$ is the Fourier transform of the corrupted image. The corresponding interference pattern in the spatial domain is obtained with the inverse Fourier transform

 $$\eta(x,y)={\calF}^{-1}\{H(u,v)G(u,v)\}$$ (6.25)

As the corrupted image $g(x,y)$ is assumed to be formed by the addition of the uncorrupted image $f(x,y)$ and the interference noise $\eta(x,y)$

 $$g(x,y)=f(x,y)+\eta(x,y)$$ (6.26)

If the noise term $\eta(x,y)$ were known completely, subtracting it from $g(x,y)$ would yield $f(x,y)$. The problem is of course that the estimated interference pattern $\eta(x,y)$ is only an approximation. The effect of components not present in the estimate $\eta(x,y)$ can be minimised be subtracting from $g(x,y)$ a weighted portion of $\eta(x,y)$ to obtain an estimate for $f(x,y)$

 $$\hat{f}(x,y)=g(x,y)-\omega(x,y)\eta(x,y)$$ (6.27)

where $\hat{f}(x,y)$ is the estimate of $f(x,y)$, and $\omega(x,y)$ is to be determined and optimises $\hat{f}(x,y)$ in some meaningful way.

One common approach is to select $\omega(x,y)$ so that the variance of $\hat{f}(x,y)$ is minimised over a specified neighbourhood of size $(2a+1)\times(2b+1)$ about every point $(x,y)$.

The additional assumption that $\omega(x,t)$ remains constant over the neighbourhood yields the approximation

 $$\omega(x+s,y+t)=\omega(x,y)$$ (6.28)

and we can solve the minimisation problem with

 $$\sigma^2(x,y)=\frac{1}{(2a+1)(2b+1)}\sum_{s=-a}^{a}\sum_{t=-b}^{b}\left[\hat{f}(x+s,y+t)-\bar{\hat{f}}(x,y)\right]^2$$ (6.29)

where $\bar{\hat{f}}(x,y)$ is the average value of $\hat{f}(x,y)$ in the neighbourhood; that is

 $$\overline{\hat{f}}(x,y)=\frac{1}{(2a+1)(2b+1)}\sum_{s=-a}^{a}\sum_{t=-b}^{b}\hat{f}(x+s,y+t)$$ (6.30)

Substituting Eq 6.27 into Eq 6.29 yields

 $\begin{eqnarray*}\sigma^2(x,y)&=&\frac{1}{(2a+1)(2b+1)}\sum_{s=-a}^{a}\sum_{t=-b}^{b}([g(x+s,y+t)\\&&-\omega(x+s,y+t)\eta(x+s,y+t)]\\&&-[\bar{g}(x,y)-\overline{\omega(x,y)\eta(x,y)}])^2\end{eqnarray*}$ (6.31)

Assuming that $\omega(x,y)$ remains constant over the neighbourhood gives the approximation

 $$\omega(x+s,y+t)=\omega(x,y)$$ (6.32)

for $-a\les\lea$ and $-b\let\leb$. This assumption results in the expression

 $$\overline{\omega(x,y)\eta(x,y)}=\omega(x,y)\bar{\eta}(x,y)$$ (6.33)

in the neighbourhood. With these approximations Eq 6.30 becomes

 $\begin{eqnarray*}\sigma^2(x,y)&=&\frac{1}{(2a+1)(2b+1)}\sum_{s=-a}^{a}\sum_{t=-b}^{b}([g(x+s,y+t)\\&&-\omega(x,y)\eta(x+s,y+t)]-[\bar{g}(x,y)-\omega(x,y)\overline{\eta}(x,y)])^2\end{eqnarray*}$ (6.34)

To minimise $\sigma^2(x,y)$, we solve

 $$\frac{\partial\sigma^2(x,y)}{\partial\omega(x,y)}=0$$ (6.35)

for $\omega(x,y)$. The result is

 $$\omega(x,y)=\frac{\overline{g(x,y)\eta(x,y)}-\bar{g}(x,y)\bar{\eta}(x,y)}{\overline{\eta^2}(x,y)-\bar{\eta}^2(x,y)}$$ (6.36)

To obtain an estimated restored image $\hat{f}(x,y)$, $\omega(x,y)$ must be computed from Eq 6.36 and substituted into Eq 6.27 to determine $\hat{f}(x,y)$.

Optimum Notch Filtering Image Restoration

# (37) Optimum Notch Filtering Principle

 Fig 6.41 Principle of the optimum notch filter

Optimum Notch Filtering Image Restoration

# (38) Optimum Notch Filter Example

This example shows an aerial image of Pompeii that is distorted with a high frequency sinusoidal pattern that is modulated with a low frequency sinus. It is often difficult to catch the low frequency components of the periodic noise (buried in the strong low frequency part of the spectrum). Ordinary notch filtering can thus not compensate for the distortions. It often even worsens the result. Optimum notch filtering can to some extent compensate.
 Fig 6.42 Aerial image of Pompeii with heavy periodic noise distortions Fig 6.43 Noise estimation with Gaussian notch pass filter Fig 6.44 Notch reject filtered image Fig 6.45 Result filtered with an optimum notch filter of size $20\times20$

Optimum Notch Filtering Image Restoration

# (39) Optimum Notch Filter Example with Pixel Increments

 → pompeii_mod_periodic_noise_opt_notch_filtered_pixel_steps.png [06-Restoration-media/figs/pompeii_mod_periodic_noise_opt_notch_filtered_pixel_steps.png]→ pompeii_mod_periodic_noise_opt_notch_filtered.png [06-Restoration-media/figs/pompeii_mod_periodic_noise_opt_notch_filtered.png]

Fig 6.46 Aerial image of Pompeii corrected with the optimal notch filter of size $50\times50$ with the window shifted pixelwise (mouseover) shows the result when shifting the window by 50

Optimum Notch Filtering Image Restoration

# (40) Optimum Notch Filter Example with Pixel Increments (2)

Fig 6.47 $\omega(x,y)$ of the example image in Fig 6.46

# Outline (Estimating the Degradation Function)

1. Image Degradation & Restoration Model (2 Slides)
2. Noise Models (12 Slides)
3. Restoration in the Precense of Noise Only (1 Slides)
4. Periodic Noise Reduction by Frequency Domain Filtering (15 Slides)
1. Bandreject Filter (2 Slides)
2. Bandpass Filter (2 Slides)
3. Notch Filter (6 Slides)
4. Optimum Notch Filtering (5 Slides)
5. Estimating the Degradation Function (6 Slides)
6. Inverse Filtering (21 Slides)
1. Direct Inverse Filtering (5 Slides)
2. Wiener Filtering (9 Slides)
3. Geometric Mean Filter (1 Slides)
4. Constrained Least Squares Filtering (6 Slides)
Estimating the Degradation Function Image Restoration

# (42) Estimating the Degradation Function

So far we concentrated mainly at the additive noise term $\eta(x,y)$. In the remainder of this presentation we concentrate on ways to remove the degradation function $H$.

Estimating the Degradation Function Image Restoration

# (43) Estimating the Degradation Function (2)

There are three principle methods to estimate the degradation function $H$

1. Observation
2. Experimentation
3. Mathematical modelling

The process of restoring a corrupted image using the estimated degradation function is sometimes called blind deconvolution, as the true degradation is seldom known completely.

In recent years a forth method based on maximum-likelihood estimation (MLE) often called blind deconvolution has been proposed.
Beware: All multiplications/divisions in the equations in this part of the lecture are componentwise.

Estimating the Degradation Function Image Restoration

# (44) Estimating by Image Observation

Given

A degraded image without any information about the degradation function $H$.

Solution
1. Select a subimage with strong signal content → $g_s(x,y)$
2. By estimating the sample grey levels of the object and background in $g_s(x,y)$ we can construct an unblurred image of the same size and characteristics as the observed subimage → $\hat{f}_s(x,y)$
3. Assuming that the effect of noise is negligible (thus the choice of a strong signal area), it follows from Eq 6.2 that

$$H_s(u,v)=\frac{G_s(u,v)}{\hat{F}_s(u,v)}$$

4. Thanks to the shift invariance we can deduce $H(u,v)$ from $H_s(u,v)$

Estimating the Degradation Function Image Restoration

# (45) Estimating by Experimentation

Given

Equipment similar to the equipment used to acquire the degraded image

Solution
1. Images similar to the degraded image can be acquired with various system settings until it closely matches the degraded image
2. The system can then be characterised by measuring the impulse (small bright dot of light) response. A linear, shift invariant system can be fully described by its impulse response.
3. As the Fourier transform of an impulse is a constant, it follows from Eq 6.2

$$H(u,v)=\frac{G(u,v)}{A}$$

where $G(u,v)$ is the Fourier transform of the observed image, and $A$ is a constant describing the strength of the impulse.

Estimating the Degradation Function Image Restoration

# (46) Estimating by Mathematical Modelling

Degradation modelling has been used for many years. It, however, requires an indepth understanding of the involved physical phenomenon. In some cases it can even incorporate environmental conditions that cause degradations. Hufnagel and Stanly proposed in 1964 a model to characterise atmospheric turbulence.

 $$H(u,v)=e^{-k(u^2+v^2)^{5/6}}$$ (6.37)

where $k$ is a constant that depends on the nature of the turbulence.

Except of the 5/6 power the above equation has the same form as a Gaussian lowpass filter.

Estimating the Degradation Function Image Restoration

# (47) Illustration of the Atmospheric Turbulence Model

 Fig 6.49: Aerial image of Pompeii Fig 6.50: Pompeii with mild turbulence $k=0.00025$ Fig 6.51: Pompeii with medium turbulence $k=0.001$ Fig 6.52 Pompeii with heavy turbulence $k=0.0025$

# Outline (Inverse Filtering)

1. Image Degradation & Restoration Model (2 Slides)
2. Noise Models (12 Slides)
3. Restoration in the Precense of Noise Only (1 Slides)
4. Periodic Noise Reduction by Frequency Domain Filtering (15 Slides)
1. Bandreject Filter (2 Slides)
2. Bandpass Filter (2 Slides)
3. Notch Filter (6 Slides)
4. Optimum Notch Filtering (5 Slides)
5. Estimating the Degradation Function (6 Slides)
6. Inverse Filtering (21 Slides)
1. Direct Inverse Filtering (5 Slides)
2. Wiener Filtering (9 Slides)
3. Geometric Mean Filter (1 Slides)
4. Constrained Least Squares Filtering (6 Slides)

# Outline (Direct Inverse Filtering)

1. Image Degradation & Restoration Model (2 Slides)
2. Noise Models (12 Slides)
3. Restoration in the Precense of Noise Only (1 Slides)
4. Periodic Noise Reduction by Frequency Domain Filtering (15 Slides)
1. Bandreject Filter (2 Slides)
2. Bandpass Filter (2 Slides)
3. Notch Filter (6 Slides)
4. Optimum Notch Filtering (5 Slides)
5. Estimating the Degradation Function (6 Slides)
6. Inverse Filtering (21 Slides)
1. Direct Inverse Filtering (5 Slides)
2. Wiener Filtering (9 Slides)
3. Geometric Mean Filter (1 Slides)
4. Constrained Least Squares Filtering (6 Slides)
Direct Inverse Filtering Image Restoration

# (50) Direct Inverse Filtering

The simplest approach to restore a degraded image is to form an estimate of the form

 $$\hat{F}(u,v)=\frac{G(u,v)}{H(u,v)}$$ (6.38)

and then obtain the corresponding image by inverse Fourier transform. This method is called direct inverse filtering. With the degradation model Equation 6.2 we get

 $$\hat{F}(u,v)=F(u,v)+\frac{N(u,v)}{H(u,v)}$$ (6.39)

This simple equation tells us that, even if we knew $H(u,v)$ exactly we could not recover the undegraded image $f(x,y)$ completely because:

1. the noise component is a random function whose Fourier transform $N(u,v)$ is not known, and
2. in practice $H(u,v)$ can contain numerous zeros.
Direct Inverse filtering is seldom a suitable approach in practical applications

Direct Inverse Filtering Image Restoration

# (51) Two Practical Approaches for Inverse Filtering

1. Low Pass Filter
 $$\hat{F}(u,v)=\frac{G(u,v)}{H(u,v)}L(u,v)$$ (6.40)

where $L(u,v)$ is a low-pass filter that should eliminiate the very low (or even zero) values often experienced in the high frequencies.

2. Constrained Division
 $$\hat{F}(u,v)=\begin{cases}\frac{G(u,v)}{H(u,v)}&\mbox{if}|H(u,v)|\ged\\G(u,v)&\mbox{if}|H(u,v)| (6.41) gives values, where $H(u,v)$ is lower than a threshold $d$ a special treatment. These appraoches are also known as Pseudo-Inverse Filtering. Direct Inverse Filtering Image Restoration # (52) Artificial Inverse Filtering Example  Fig 6.53: Checkerboard sample Fig 6.54: Motion blur of $\unit{7}{pixel}$ in $45^\circ$ direction Fig 6.55: Additive noise with $\mu=0,\sigma=0.1$ Fig 6.56: Motion and noise degraded checkerboard Direct Inverse Filtering Image Restoration # (53) Artificial Inverse Filtering Example (2) Although the degradation function $H(u,v)$ was known exactly when calculating this example, the degradation could not be undone. Zero values and small values in $H(u,v)$ dominate and render the reconstruction useless. Although very tempting from its simplicity, the direct inverse filtering approach does not work for practical applications. Even if the degradation $H$ is known exactly.  Fig 6.57: Direct inverse filtering result Direct Inverse Filtering Image Restoration # (54) Inverse Filtering Example (2) Inverse filtering with all frequencies is useless (topleft). Limiting the inverse filtering to the lower frequencies yields useful results as can be seen in the top/right and lower/left image. The optimum radius seems to be around 80 with lower radiuses losing to much of the high frequencies and higher radiuses introducing too much artefacts (lower/right).  Fig 6.58: Inverse filtering of Figure 6.52 Fig 6.59: Pseudo Inverse filtering for freq. radius $\le40$ Fig 6.60: Pseudo Inverse filtering for frequencies radius $\le80$ Fig 6.61: Pseudo Inverse filtering for frequencies radius $\le120$ # Wiener Filtering Wiener Filtering Image Restoration # (56) Wiener Filtering The best known improvement to inverse filtering is the Wiener Filter. The Wiener filter seeks an estimate $\hat{f}$ that minimises the statistical error function  $\[e^2=E\{(f-\hat{f})^2\}$$ (6.42)

where $E$ is the expected value and $f$ the undegraded image. The solution to this equation in the frequency domain is

 $$\hat{F}(u,v)=\left[\frac{1}{H(u,v)}\frac{|H(u,v)|^2}{|H(u,v)|^2+S_\eta(u,v)/S_f(u,v)}\right]G(u,v)$$ (6.43)

where

• $H(u,v)$ is the degradation function
• $|H(u,v)|^2=H^\ast(u,v)H(u,v)$ with $H^\ast(u,v)$ the complex conjugate of $H$
• $S_\eta(u,v)=|N(u,v)|^2$ is the power spectrum of the noise
• $S_f(u,v)=|F(u,v)|^2$ is the power spectrum of the undegraded image

Wiener Filtering Image Restoration

# (57) Wiener Filtering (2)

If the noise spectrum $|N(u,v)|^2$ is zero, the noise-to-signal power ratio $S_\eta(u,v)/S_f(u,v)=|N(u,v)|^2/|F(u,v)|^2=0$ vanishes and the Wiener Filter reduces to the inverse filter.

 $\begin{eqnarray*}\hat{F}(u,v)&=&\left[\frac{1}{H(u,v)}\frac{|H(u,v)|^2}{|H(u,v)|^2+{\color{red}S_\eta(u,v)}/S_f(u,v)}\right]G(u,v)\$2ex]&=&\left[\frac{1}{H(u,v)}\frac{|H(u,v)|^2}{|H(u,v)|^2+0}\right]G(u,v)\\[2ex]&=&\frac{1}{H(u,v)}G(u,v)\end{eqnarray*}$ (6.44) This is no problem, as the inverse filter works fine if no noise is present. Wiener Filtering Image Restoration # (58) Parametric Wiener Filter The main problem with the Wiener filter is that the power spectrum $|F(u,v)|^2$ of the undegraded image is seldom known. A frequently used approach when these quantities cannot be estimated is to approximate Eq 6.43 by the expression  $\[\hat{F}(u,v)=\left[\frac{1}{H(u,v)}\frac{|H(u,v)|^2}{|H(u,v)|^2+K}\right]G(u,v)$$ (6.45)

where $K$ is a user selected constant.

This simplification can be partly justified when dealing with spectrally white noise, where the noise spectrum $S_\eta(u,v)=|N(u,v)|^2$ is constant. However, the problem still remains that the power spectrum $S_f(u,v)=|F(u,v)|^2$ of the undegraded image is unknown and must be estimated.

Even if the actual ratio $K$ is not known, it becomes a simple matter to experiment interactively varying the constant and viewing the results.

Wiener Filtering Image Restoration

# (59) Artificial Wiener Filtering Example

This example shows the performance of the Wiener filter on the example Fig 6.56 using (1) the approximation of Eq 6.45 with $K=0.04$ in Fig 6.62 and (2) the Wiener filter using full knowledge of the noise and undegraded image's power spectra in Fig 6.63.

 Fig 6.62: Parametric Wiener filter using a constant ratio Fig 6.63: Wiener filter knowing the noise and undegraded signal power spectra

Wiener Filtering Image Restoration

# (60) Wiener Filtering Example with Little Noise

With only little to no noise the three methods seem to be comparable.
 Fig 6.64: Pompeii blurred with Gaussian kernel $\sigma=5$, and Gaussian noise $\mu=0,\sigma=0.001$ Fig 6.65: Result of direct inverse filtering Fig 6.66: Result with Parametric Wiener filtering $K=0.0001$ Fig 6.67: Result with Wiener filtering

Wiener Filtering Image Restoration

# (61) Wiener Filtering Example with Medium Noise

With only mild noise the direct inverse filtering approach starts to fail, while the parametric Wiener and pure Wiener filtering can still cope with images.
 Fig 6.68: Pompeii blurred with Gaussian kernel $\sigma=5$, and Gaussian noise $\mu=0,\sigma=0.01$ Fig 6.69: Result of direct inverse filtering Fig 6.70: Result with Parametric Wiener filtering $K=0.0001$ Fig 6.71: Result with Wiener filtering

Wiener Filtering Image Restoration

# (62) Wiener Filtering Example with Heavy Noise

The direct inverse filtering approach completely fails. The parametric Wiener does a decent job, but has some ghosting artefacts. The pure Wiener filtering still does a very good job.
 Fig 6.72: Pompeii blurred with Gaussian kernel $\sigma=5$, and Gaussian noise $\mu=0,\sigma=0.1$ Fig 6.73: Result of direct inverse filtering Fig 6.74: Result with Parametric Wiener filtering $K=0.0001$ Fig 6.75: Result with Wiener filtering

Wiener Filtering Image Restoration

# (63) Comparison Param. Wiener vs. Wiener Filtering

 → pompeii_gaussblur=5_noise=0.1_param_wiener.png [06-Restoration-media/figs/pompeii_gaussblur=5_noise=0.1_param_wiener.png]→ pompeii_gaussblur=5_noise=0.1_wiener.png [06-Restoration-media/figs/pompeii_gaussblur=5_noise=0.1_wiener.png]
 (a) (b)

Fig 6.76 Aerial image of Pompeii with Gaussian blur ($\sigma=5$) and noise ($\mu=0,\sigma=0.1$) (a) restored with the Parametric Wiener ($K=0.0001$), (b) (mouseover) same image restored with Wiener filter.

Wiener Filtering Image Restoration

# (64) Drawback of the Wiener Filter

The problem of having to know something about the degradation function $H(u,v)$ is common to all methods discussed in this part.

The Wiener filter has the additional difficulty, that

• the power spectra of the undegraded image and
• the power spectra of the noise

must be known.

Although an approximation by a constant is possible as in the Parametric Wiener filter, see Fig 6.62, this approach is not always suitable.

# Outline (Geometric Mean Filter)

1. Image Degradation & Restoration Model (2 Slides)
2. Noise Models (12 Slides)
3. Restoration in the Precense of Noise Only (1 Slides)
4. Periodic Noise Reduction by Frequency Domain Filtering (15 Slides)
1. Bandreject Filter (2 Slides)
2. Bandpass Filter (2 Slides)
3. Notch Filter (6 Slides)
4. Optimum Notch Filtering (5 Slides)
5. Estimating the Degradation Function (6 Slides)
6. Inverse Filtering (21 Slides)
1. Direct Inverse Filtering (5 Slides)
2. Wiener Filtering (9 Slides)
3. Geometric Mean Filter (1 Slides)
4. Constrained Least Squares Filtering (6 Slides)
Geometric Mean Filter Image Restoration

# (66) Geometric Mean Filter

The Geometric Mean Filter is a generalisation of inverse filtering and the Wiener filtering. Through its parameters $\alpha,\beta$ it allows access to an entire family of filters.

 $$\hat{F}(u,v)=\left[\frac{H^\ast(u,v)}{|H(u,v)|^2}\right]^\alpha\left[\frac{H^\ast(u,v)}{|H(u,v)|^2+\beta\frac{S_\eta(u,v)}{S_f(u,v)}}\right]^{1-\alpha}G(u,v)$$ (6.46)

Depending on the parameters $\alpha,\beta$ the filter characteristics can be adjusted

• $\alpha=1$: inverse filtering
• $\alpha=1/2,\beta=1$: spectrum equalisation filter
• $\alpha=0$: parametric Wiener filter → $\beta=1$ Wiener

# Outline (Constrained Least Squares Filtering)

1. Image Degradation & Restoration Model (2 Slides)
2. Noise Models (12 Slides)
3. Restoration in the Precense of Noise Only (1 Slides)
4. Periodic Noise Reduction by Frequency Domain Filtering (15 Slides)
1. Bandreject Filter (2 Slides)
2. Bandpass Filter (2 Slides)
3. Notch Filter (6 Slides)
4. Optimum Notch Filtering (5 Slides)
5. Estimating the Degradation Function (6 Slides)
6. Inverse Filtering (21 Slides)
1. Direct Inverse Filtering (5 Slides)
2. Wiener Filtering (9 Slides)
3. Geometric Mean Filter (1 Slides)
4. Constrained Least Squares Filtering (6 Slides)
Constrained Least Squares Filtering Image Restoration

# (68) Constrained Least Squares (Regularised) Filtering

The Constrained Least Squares Filtering approach only requires knowledge of the mean and variance of the noise. As shown in previous lectures these parameters can be usually estimated from the degraded image.

The definition of the 2D discrete convolution is

 $$h(x,y)\astf(x,y)=\frac{1}{MN}\sum_{m=0}^{M-1}\sum_{n=0}^{N-1}f(m,n)h(x-m,y-n)$$ (6.47)

Using this equation we can express the linear degradation model $g(x,y)=h(x,y)\astf(x,y)+\eta(x,y)$ in vector notation as

 $${\bfg=Hf+\eta}$$ (6.48)

Constrained Least Squares Filtering Image Restoration

# (69) Constrained Least Squares Filtering (2)

It seems obvious that the restoration problem is now reduced to simple matrix manipulations. Unfortunately this is not the case. The problem with the matrix formulation is that the $H$ matrix is very large $MN\timesMN$ and that its inverse is firstly very sensitive to noise and secondly does not necessarily exist.

One way to deal with these issues is to base optimality of restoration on a measure of smoothness, such as the second derivative of the image, e.g. the Laplacian.

 $$C=\sum_{x=0}^{M-1}\sum_{y=0}^{N-1}\left[\nabla^2f(x,y)\right]$$ (6.49)

subject to the constraint

 $$||{\bfg-H\hat{f}}||^2=||{\bf\eta}^2||$$ (6.50)

where $||\cdot||$ is the Euclidian norm, and $\hat{f}$ an estimate of the undegraded image.

Constrained Least Squares Filtering Image Restoration

# (70) Constrained Least Squares Filtering (3)

The Frequency domain solution to this optimisation problem is given by

 $$\hat{F}(u,v)=\left[\frac{H^\ast(u,v)}{|H(u,v)|^2+\gamma|P(u,v)|^2}\right]G(u,v)$$ (6.51)

where $\gamma$ is a parameter that must be adjusted so that the constraint in Eq 6.50 is satisfied, and $P(u,v)$ is the Fourier transform of the Laplace operator

 $$p(x,y)=\begin{bmatrix}0&-1&0\\-1&4&-1\\0&-1&0\end{bmatrix}$$ (6.52)

Note: $p(x,y)$ must be properly padded with zeros prior to computing the Fourier transform.

It is possible to adjust the parameter $\gamma$ interactively until acceptable results are obtained.

Constrained Least Squares Filtering Image Restoration

# (71) Optimal Selection for Gamma

If we are interested in optimality, the parameter $\gamma$ must be adjusted so that the constraint $||{\bfg-H\hat{f}}||^2=||{\bf\eta}^2||$ in Eq 6.50 is satisfied.

It turns out that

 $$\gamma=MN[\sigma^2_\eta+m_\eta^2]$$ (6.53)

is the optimal selection given the above optimisation equation where $\sigma_\eta,m_\eta$ are the mean and variance of the noise.

It is important to understand, that optimum restoration in the sense of constrained least squares does not necessarily imply best in the visual sense. In general, automatic determined restoration filters yield inferior results to manual adjustment of filter parameters.

Constrained Least Squares Filtering Image Restoration

# (72) Artificial Constrained Least Squares Filter Example

The example on the left clearly shows that our chosen optimisation criterion (laplace) indeed resulted in a smooth image. The noise seems to have been entirely removed. However, as the manually selected $\gamma$ on the right side shows there are settings for gamma that provide from a visual standpoint a better solution.

This example shows the performance of the Constrained Least Squares Filter on the example Fig 6.56 using (1) the theoretical optimal value for $\gamma=40.96$ in Fig 6.77 and (2) the constrained least squares filter with a manually selected $\gamma$ of $10.1861$ 6.78.

 Fig 6.77: Result with optimal $\gamma$ Fig 6.78: Results with manually selected $\gamma$

Constrained Least Squares Filtering Image Restoration

# (73) Constrained Least Squares Filter Example

The CLS filter shows good performance although only one single parameter has to be adapted (ideally manually).
 Fig 6.79: Pompeii blurred with Gaussian kernel $\sigma=5$ Fig 6.80: Result with low Gaussian noise of $\sigma=0.001$ Fig 6.81: Result with medium Gaussian noise of $\sigma=0.01$ Fig 6.82: Result with high Gaussian noise of $\sigma=0.1$

April 19th/26th, 2016 Introduction to Signal and Image Processing