University of Basel
University of Basel
University of Basel

Image Restoration

Introduction to Signal and Image Processing

MIAC, University of Basel
April 29th, 2014
Contents Ph. Cattin: Image Restoration

Contents

Ph. Cattin: Image Restoration

(2) Abstract

As in image enhancement the goal of restoration is to improve an image for further processing. In contrast to image enhancement that was subjective and largely based on heuristics, restoration attempts to reconstruct or recover an image that has been distorted by a known degradation phenomenon. Restoration techniques thus try to model the degradation process and apply the inverse process in order to reconstruct the original image.



Image Degradation & Restoration Model

Outline (Image Degradation & Restoration Model)

  1. Image Degradation & Restoration Model (2 Slides)
  2. Noise Models (12 Slides)
  3. Restoration in the Precense of Noise Only (1 Slides)
  4. Periodic Noise Reduction by Frequency Domain Filtering (15 Slides)
    1. Bandreject Filter (2 Slides)
    2. Bandpass Filter (2 Slides)
    3. Notch Filter (6 Slides)
    4. Optimum Notch Filtering (5 Slides)
  5. Estimating the Degradation Function (6 Slides)
  6. Inverse Filtering (21 Slides)
    1. Direct Inverse Filtering (5 Slides)
    2. Wiener Filtering (9 Slides)
    3. Geometric Mean Filter (1 Slides)
    4. Constrained Least Squares Filtering (6 Slides)
Image Degradation & Restoration Model Ph. Cattin: Image Restoration

(4) Image Degradation & Restoration Model

The degradation process is generally modelled as a degradation function H and the additive noise term \eta(x,y) together they yield g(x,y).

Given g(x,y), some knowledge about H, and \eta(x,y) it is the objective of restoration to estimate \hat{f}(x,y). Of course this estimate should be as close as possible to f(x,y).



Fig 6.1 Image degradation model


Image Degradation & Restoration Model Ph. Cattin: Image Restoration

(5) Image Degradation & Restoration Model (2)

If the degradation function H is a linear, shift-invariant process, then the degraded image is given in the spatial domain by

\[g(x,y)=h(x,y)\astf(x,y)+\eta(x,y)\](6.1)

where h(x,y) is the spatial representation of the degradation function, and \ast indicates convolution.

As we know from the Convolution Theorem this can be rewritten in the frequency domain

\[G(u,v)=H(u,v)F(u,v)+N(u,v)\](6.2)

where the capital letters are the Fourier transforms of the respective function in Eq 6.1.



Noise Models

Outline (Noise Models)

  1. Image Degradation & Restoration Model (2 Slides)
  2. Noise Models (12 Slides)
  3. Restoration in the Precense of Noise Only (1 Slides)
  4. Periodic Noise Reduction by Frequency Domain Filtering (15 Slides)
    1. Bandreject Filter (2 Slides)
    2. Bandpass Filter (2 Slides)
    3. Notch Filter (6 Slides)
    4. Optimum Notch Filtering (5 Slides)
  5. Estimating the Degradation Function (6 Slides)
  6. Inverse Filtering (21 Slides)
    1. Direct Inverse Filtering (5 Slides)
    2. Wiener Filtering (9 Slides)
    3. Geometric Mean Filter (1 Slides)
    4. Constrained Least Squares Filtering (6 Slides)
Noise Models Ph. Cattin: Image Restoration

(7) Noise Models

The principle sources of noise in digital images arise during image acquisition and/or transmission.

The performance of imaging sensors are affected by a variety of factors during acquisition, such as

Images can also be corrupted during transmission due to interference in the channel for example

Depending on the specific noise source, a different model must be selected that accurately reproduces the spatial characteristics of the noise.



Noise Models Ph. Cattin: Image Restoration

(8) Gaussian Noise

Because of its mathematical tractability in both the spatial and frequency domain, Gaussian noise models (aka normal distribution) are used frequently in practice.

The PDF (Probability density function) of a Gaussian random variable z is given by

\[p_G(z)=\frac{1}{\sqrt{2\pi}\sigma}e^{-(z-\mu)^2/(2\sigma^2)}\](6.3)

where z represents the grey level, \mu the mean value and \sigma the standard deviation.


Fig 6.2: Gaussian probability density function p_G(z)
The application of the Gaussian model is so convenient that it is often used in situations in which they are marginally applicable at best.


Noise Models Ph. Cattin: Image Restoration

(9) Rayleigh Noise

The PDF of Rayleigh noise is defined by

\[p_R(z)=\begin{cases}\frac{2}{b}(z-a)e^{-\frac{(z-a)^2}{b}}&\mbox{for}z\gea\\[2ex]0&\mbox{for}z<a\end{cases}\](6.4)

the mean and variance are given by

\begin{eqnarray*}\mu&=&a+\sqrt{\pib/4}\\[2ex]\sigma^2&=&\frac{b(4-\pi)}{4}\end{eqnarray*}(6.5)

Fig 6.3: Rayleigh probability density function p_R(z)

As the shape of the Rayleigh density function is skewed it is useful for approximating skewed histograms.



Noise Models Ph. Cattin: Image Restoration

(10) Erlang Noise

The PDF of the Erlang noise is given by

\[p_E(z)=\begin{cases}\frac{a^bz^{b-1}}{(b-1)!}e^{-az}&\mbox{for}z\ge0\\[2ex]0&\mbox{for}z<0\end{cases}\](6.6)

where a>0, b is a positive integer. The mean and variance are then given by

\begin{eqnarray*}\mu&=&\frac{b}{a}\\[2ex]\sigma^2&=&\frac{b}{a^2}\end{eqnarray*}(6.7)

Fig 6.4: Erlang probability density function p_E(z)



Noise Models Ph. Cattin: Image Restoration

(11) Exponential Noise

The PDF of the exponential noise is given by

\[p_{exp}(z)=\begin{cases}ae^{-az}&\mbox{for}z\ge0\\[2ex]0&\mbox{for}z<0\end{cases}\](6.8)

where a>0. The mean and variance of the density function are

\begin{eqnarray*}\mu&=&\frac{1}{a}\\[2ex]\sigma^2&=&\frac{1}{a^2}\end{eqnarray*}(6.9)

Fig 6.5: Probability density function p_{exp}(z) of exponential noise

The exponential noise model is a special case of the Erlang noise model with b=1.



Noise Models Ph. Cattin: Image Restoration

(12) Uniform Noise

The PDF of the uniform noise is given by

\[p_U(z)=\begin{cases}\frac{1}{b-a}&\mbox{for}a\lez\leb\\[2ex]0&\mbox{forotherwise}\end{cases}\](6.10)

The mean and variance of the density function are given by

\begin{eqnarray*}\mu&=&\frac{a+b}{2}\\[2ex]\sigma^2&=&\frac{(b-a)^2}{12}\end{eqnarray*}(6.11)

Fig 6.6: Probability density function p_U(z) of uniform noise



Noise Models Ph. Cattin: Image Restoration

(13) Impulse (Salt-and-Pepper) Noise

The PDF of bipolar impulse noise model is given by

\[p_I(z)=\begin{cases}P_a&\mbox{for}z=a\\[2ex]P_b&\mbox{for}z=b\\[2ex]0&\mbox{otherwise}\end{cases}\](6.12)

Fig 6.7: Probability density function p_I(z) of the bipolar impulse noise model

If b>a, grey-level b appears as a light dot (salt) in the image. Conversely, a will appear as dark dot (pepper). If either P_a,P_b is zero, the PDF is called unipolar.

Because impulse corruption is generally large compared to the signal strength, the assumption is usually that a and b are digitised as saturated values thus black (pepper) and white (salt).



Noise Models Ph. Cattin: Image Restoration

(14) Noise Models

The presented noise models provide a useful tool for approximating a broad range of noise corruption situations found in practice. For example:



Noise Models Ph. Cattin: Image Restoration

(15) Visual Comparison of Different Noise Sources

Although the corresponding histograms show close resemblance to the PDF of the respective noise model, it is virtually impossible to select the type of noise causing the degradation from the image alone.

Gaussian noise model

Rayleigh noise model

Erlang noise model

Exponential noise model

Uniform noise model

Impulse noise model

(Salt & Pepper)





Noise Models Ph. Cattin: Image Restoration

(16) Periodic Noise

Periodic noise typically arises from electrical or electromechanical interference during image acquisition and is spatial dependent.


Fig 6.8: Periodic noise on a satellite image of Pompeii

Fig 6.9: Spectrum of the Pompeii image with periodic noise (peaks of the periodic noise visually enhanced)


Noise Models Ph. Cattin: Image Restoration

(17) Estimation of Noise Parameters (1)

In rare cases the parameters of noise PDFs may be known from sensor specifications, but it is often necessary to estimate them for a particular imaging arrangement.

If the imaging system is available, one simple way to study noise characteristics would be to capture multiple images of homogeneous environments, such as solid grey cards.


Fig 6.10: Sample histogram of a small patch of Erlang noise

In many cases the noise parameters have to be estimated from already captured images. The simplest way is to estimate the mean and variance of the grey levels from small patches of reasonably constant grey level.



Noise Models Ph. Cattin: Image Restoration

(18) Estimation of Periodic Noise Parameters

The parameters of periodic noise are typically estimated by inspection of the Fourier spectrum. The frequency spikes can often be detected by visual analysis. Automatic analysis is possible if the noise spikes are either exceptionally pronounced or a priori knowledge about their location is known.


Fig. 6.11:

Fig. 6.12:


Restoration in the Precense of Noise Only

Outline (Restoration in the Precense of Noise Only)

  1. Image Degradation & Restoration Model (2 Slides)
  2. Noise Models (12 Slides)
  3. Restoration in the Precense of Noise Only (1 Slides)
  4. Periodic Noise Reduction by Frequency Domain Filtering (15 Slides)
    1. Bandreject Filter (2 Slides)
    2. Bandpass Filter (2 Slides)
    3. Notch Filter (6 Slides)
    4. Optimum Notch Filtering (5 Slides)
  5. Estimating the Degradation Function (6 Slides)
  6. Inverse Filtering (21 Slides)
    1. Direct Inverse Filtering (5 Slides)
    2. Wiener Filtering (9 Slides)
    3. Geometric Mean Filter (1 Slides)
    4. Constrained Least Squares Filtering (6 Slides)
Restoration in the Precense of Noise Only Ph. Cattin: Image Restoration

(20) Restoration in the Precense of Noise Only

When noise is the only image degradation present in an image, thus H(u,v)=1, Eqs 6.1 & 6.2 become

\[g(x,y)=f(x,y)+\eta(x,y)\](6.13)

and

\[G(u,v)=F(u,v)+N(u,v)\](6.14)

In case of periodic noise, it is usually possible to estimate N(u,v) from the spectrum of G(u,v) and subtract it to obtain an estimate of the original image f(x,y).

But estimating the noise terms \eta(x,y) is unreasonable, so subtracting them from g(x,y) is impossible. Spatial filtering is the method of choice in situations when only additive noise is present.



Periodic Noise Reduction by Frequency Domain Filtering

Outline (Periodic Noise Reduction by Frequency Domain Filtering)

  1. Image Degradation & Restoration Model (2 Slides)
  2. Noise Models (12 Slides)
  3. Restoration in the Precense of Noise Only (1 Slides)
  4. Periodic Noise Reduction by Frequency Domain Filtering (15 Slides)
    1. Bandreject Filter (2 Slides)
    2. Bandpass Filter (2 Slides)
    3. Notch Filter (6 Slides)
    4. Optimum Notch Filtering (5 Slides)
  5. Estimating the Degradation Function (6 Slides)
  6. Inverse Filtering (21 Slides)
    1. Direct Inverse Filtering (5 Slides)
    2. Wiener Filtering (9 Slides)
    3. Geometric Mean Filter (1 Slides)
    4. Constrained Least Squares Filtering (6 Slides)

Bandreject Filter

Bandreject Filter Ph. Cattin: Image Restoration

(23) Bandreject Filter

Bandreject filters remove or attenuate a band of frequencies around the origin in the Fourier domain.

Ideal Bandreject Filter
\[H(u,v)=\begin{cases}1&\mbox{if}D(u,v)<D_0-\frac{W}{2}\\[2ex]%0&\mbox{if}D_0-\frac{W}{2}\leD(u,v)\leD_0+\frac{W}{2}\\[2ex]0&\mbox{otherwise}\\[2ex]1&\mbox{if}D(u,v)>D_0+\frac{W}{2}\\\end{cases}\](6.15)

Fig 6.13: Ideal bandreject filter
Butterworth Bandreject Filter
\[H(u,v)=\frac{1}{1+\left[\frac{D(u,v)W}{D^2(u,v)-D_0^2}\right]^{2n}}\](6.16)

Fig 6.14: Butterworth bandreject filter
Gaussian Bandreject Filter
\[H(u,v)=1-e^{-\frac{1}{2}\left[\frac{D^2(u,v)-D_0^2}{D(u,v)W}\right]^2}\](6.17)

Fig 6.15: Gaussian bandreject filter


Bandreject Filter Ph. Cattin: Image Restoration

(24) Bandreject Filter Example


Fig 6.16: Aerial image of Pompeii with periodic noise

Fig 6.17: Ideal bandreject filter D0=100,W=6

Fig 6.18: Butterworth bandreject filter D0=100,W=6

Fig 6.19: Gaussian bandreject filter D0=100,W=6


Bandpass Filter

Bandpass Filter Ph. Cattin: Image Restoration

(26) Bandpass Filter

The Bandpass filter performs the opposite of a bandreject filter and thus lets pass the frequencies in a narrow band of width W around D_0. The transfer function H(u,v) can be obtained from a corresponding bandreject filter by using the equation

\[H_{bp}=1-H_{br}\](6.18)

The perspective plots of the corresponding filters are depicted in Fig 6.20.


(a)

(b)

(c)
Fig 6.20 (a) Ideal bandpass filter, (b) Butterworth bandpass filter, and (c) Gaussian bandpass filter


Bandpass Filter Ph. Cattin: Image Restoration

(27) Bandpass Filter Example


Fig 6.21: Aerial image of Pompeii with periodic noise

Fig 6.22: Ideal bandpass filter D0=100,W=6

Fig 6.23: Butterworth bandpass filter D0=100,W=6

Fig 6.24: Gaussian bandpass filter D0=100,W=6


Notch Filter

Notch Filter Ph. Cattin: Image Restoration

(29) Notch Filter

A Notch filter rejects or passes frequencies in predefined neighbourhoods about a centre frequency. Due to the symmetry of the Fourier transform, notch filters must appear in symmetric pairs about the origin (except the one at the origin).

The transfer functions of the ideal, Butterworth, and Gaussian notch reject filter of radius (width) D_0 with centres at (u_0,v_0) and, by symmetry, at (-u_0,-v_0), are

Ideal Notch Filter
\[H(u,v)=\begin{cases}0&\mbox{if}D_1(u,v)\leD_0\mbox{or}D_2(u,v)\leD_0\\[2ex]1&\mbox{otherwise}\end{cases}\](6.19)

Fig 6.25: Ideal notch filter
Butterworth Notch Filter
\[H(u,v)=\frac{1}{1+\left[\frac{D_0^2}{D_1(u,v)D_2(u,v)}\right]^{n}}\](6.20)

Fig 6.26: Butterworth notch filter
Gaussian Notch Filter
\[H(u,v)=1-e^{-\frac{1}{2}\left[\frac{D_1(u,v)D_2(u,v)}{D_0^2}\right]}\](6.21)

Fig 6.27: Gaussian notch filter

where

\begin{eqnarray*}D_1(u,v)&=&\left[(u-M/2-u_0)^2+(v-N/2-v_0)^2\right]^\frac{1}{2}\\[2ex]D_2(u,v)&=&\left[(u-M/2+u_0)^2+(v-N/2+v_0)^2\right]^\frac{1}{2}\end{eqnarray*}(6.22)


Notch Filter Ph. Cattin: Image Restoration

(30) Notch Filter (2)

Similar to the bandpass/bandreject filters, the Notch reject filters can be turned into Notch pass filters with the relation

\[H_{np}(u,v)=1-H_{nr}(u,v)\](6.23)

where H_{np}(u,v) is the transfer function of the notch pass filter, and H_{nr}(u,v) the corresponding Notch reject filter.

If u_0=v_0=0
  • the Notch reject filter becomes a highpass filter and
  • the Notch pass filter becomes a lowpass filter.


Notch Filter Ph. Cattin: Image Restoration

(31) Ideal Notch Filter D_0=16 Example


Fig 6.28: Pompeii aerial image with periodic noise

Fig 6.29: Notch reject filtered image

Fig 6.30: Notch pass filtered image (contrast enhanced)


Notch Filter Ph. Cattin: Image Restoration

(32) Butterworth Notch Filter D_0=64 Example


Fig 6.31: Pompeii aerial image with periodic noise

Fig 6.32: Notch reject filtered image

Fig 6.33: Notch pass filtered image (contrast enhanced)


Notch Filter Ph. Cattin: Image Restoration

(33) Gaussian Notch Filter D_0=64 Example


Fig 6.34: Pompeii aerial image with periodic noise

Fig 6.35: Notch reject filtered image

Fig 6.36: Notch pass filtered image (contrast enhanced)


Notch Filter Ph. Cattin: Image Restoration

(34) General Notch Filter Example


Fig 6.37: Original Mariner 6 martian image

Fig 6.38: Log Fourier spectra of the image

Fig 6.39: Notch filtered log spectra

Fig 6.40: Notch filtered image


Optimum Notch Filtering

Optimum Notch Filtering Ph. Cattin: Image Restoration

(36) Optimum Notch Filtering

Images derived from electro-optical scanners, such as those used in space and aerial imaging, are sometimes corrupted by coupling and amplification of low-level signals in the scanners' circuitry. The resulting images tend to contain substantial periodic noise. The interference pattern are, however, in practice not always as clearly defined as in the samples shown thus far.

The first step in Optimum notch filtering is to find the principal frequency components and placing notch pass filters at the location of each spike, yielding H(u,v). The Fourier transform of the interference pattern is thus given by

\[N(u,v)=H(u,v)G(u,v)\](6.24)

where G(u,v) is the Fourier transform of the corrupted image. The corresponding interference pattern in the spatial domain is obtained with the inverse Fourier transform

\[\eta(x,y)={\calF}^{-1}\{H(u,v)G(u,v)\}\](6.25)

As the corrupted image g(x,y) is assumed to be formed by the addition of the uncorrupted image f(x,y) and the interference noise \eta(x,y)

\[g(x,y)=f(x,y)+\eta(x,y)\](6.26)

If the noise term \eta(x,y) were known completely, subtracting it from g(x,y) would yield f(x,y). The problem is of course that the estimated interference pattern \eta(x,y) is only an approximation. The effect of components not present in the estimate \eta(x,y) can be minimised be subtracting from g(x,y) a weighted portion of \eta(x,y) to obtain an estimate for f(x,y)

\[\hat{f}(x,y)=g(x,y)-\omega(x,y)\eta(x,y)\](6.27)

where \hat{f}(x,y) is the estimate of f(x,y), and \omega(x,y) is to be determined and optimises \hat{f}(x,y) in some meaningful way.

One common approach is to select \omega(x,y) so that the variance of \hat{f}(x,y) is minimised over a specified neighbourhood of size (2a+1)\times(2b+1) about every point (x,y).

The additional assumption that \omega(x,t) remains constant over the neighbourhood yields the approximation

\[\omega(x+s,y+t)=\omega(x,y)\](6.28)

and we can solve the minimisation problem with

\[\sigma^2(x,y)=\frac{1}{(2a+1)(2b+1)}\sum_{s=-a}^{a}\sum_{t=-b}^{b}\left[\hat{f}(x+s,y+t)-\bar{\hat{f}}(x,y)\right]^2\](6.29)

where \bar{\hat{f}}(x,y) is the average value of \hat{f}(x,y) in the neighbourhood; that is

\[\overline{\hat{f}}(x,y)=\frac{1}{(2a+1)(2b+1)}\sum_{s=-a}^{a}\sum_{t=-b}^{b}\hat{f}(x+s,y+t)\](6.30)

Substituting Eq 6.27 into Eq 6.29 yields

\begin{eqnarray*}\sigma^2(x,y)&=&\frac{1}{(2a+1)(2b+1)}\sum_{s=-a}^{a}\sum_{t=-b}^{b}([g(x+s,y+t)\\&&-\omega(x+s,y+t)\eta(x+s,y+t)]\\&&-[\bar{g}(x,y)-\overline{\omega(x,y)\eta(x,y)}])^2\end{eqnarray*}(6.31)

Assuming that \omega(x,y) remains constant over the neighbourhood gives the approximation

\[\omega(x+s,y+t)=\omega(x,y)\](6.32)

for -a\les\lea and -b\let\leb. This assumption results in the expression

\[\overline{\omega(x,y)\eta(x,y)}=\omega(x,y)\bar{\eta}(x,y)\](6.33)

in the neighbourhood. With these approximations Eq 6.30 becomes

\begin{eqnarray*}\sigma^2(x,y)&=&\frac{1}{(2a+1)(2b+1)}\sum_{s=-a}^{a}\sum_{t=-b}^{b}([g(x+s,y+t)\\&&-\omega(x,y)\eta(x+s,y+t)]-[\bar{g}(x,y)-\omega(x,y)\overline{\eta}(x,y)])^2\end{eqnarray*}(6.34)

To minimise \sigma^2(x,y), we solve

\[\frac{\partial\sigma^2(x,y)}{\partial\omega(x,y)}=0\](6.35)

for \omega(x,y). The result is

\[\omega(x,y)=\frac{\overline{g(x,y)\eta(x,y)}-\bar{g}(x,y)\bar{\eta}(x,y)}{\overline{\eta^2}(x,y)-\bar{\eta}^2(x,y)}\](6.36)

To obtain an estimated restored image \hat{f}(x,y), \omega(x,y) must be computed from Eq 6.36 and substituted into Eq 6.27 to determine \hat{f}(x,y).



Optimum Notch Filtering Ph. Cattin: Image Restoration

(37) Optimum Notch Filtering Principle


Fig 6.41 Principle of the optimum notch filter


Optimum Notch Filtering Ph. Cattin: Image Restoration

(38) Optimum Notch Filter Example

This example shows an aerial image of Pompeii that is distorted with a high frequency sinusoidal pattern that is modulated with a low frequency sinus. It is often difficult to catch the low frequency components of the periodic noise (buried in the strong low frequency part of the spectrum). Ordinary notch filtering can thus not compensate for the distortions. It often even worsens the result. Optimum notch filtering can to some extent compensate.

Fig 6.42 Aerial image of Pompeii with heavy periodic noise distortions

Fig 6.43 Noise estimation with Gaussian notch pass filter

Fig 6.44 Notch reject filtered image

Fig 6.45 Result filtered with an optimum notch filter of size 20\times20


Optimum Notch Filtering Ph. Cattin: Image Restoration

(39) Optimum Notch Filter Example with Pixel Increments

pompeii_mod_periodic_noise_opt_notch_filtered_pixel_steps.pngpompeii_mod_periodic_noise_opt_notch_filtered.png

Fig 6.46 Aerial image of Pompeii corrected with the optimal notch filter of size 50\times50 with the window shifted pixelwise (mouseover) shows the result when shifting the window by 50



Optimum Notch Filtering Ph. Cattin: Image Restoration

(40) Optimum Notch Filter Example with Pixel Increments (2)

pompeii_mod_periodic_noise_opt_notch_filtered_pixel_steps_omega.png

Fig 6.47 \omega(x,y) of the example image in Fig 6.46



Estimating the Degradation Function

Outline (Estimating the Degradation Function)

  1. Image Degradation & Restoration Model (2 Slides)
  2. Noise Models (12 Slides)
  3. Restoration in the Precense of Noise Only (1 Slides)
  4. Periodic Noise Reduction by Frequency Domain Filtering (15 Slides)
    1. Bandreject Filter (2 Slides)
    2. Bandpass Filter (2 Slides)
    3. Notch Filter (6 Slides)
    4. Optimum Notch Filtering (5 Slides)
  5. Estimating the Degradation Function (6 Slides)
  6. Inverse Filtering (21 Slides)
    1. Direct Inverse Filtering (5 Slides)
    2. Wiener Filtering (9 Slides)
    3. Geometric Mean Filter (1 Slides)
    4. Constrained Least Squares Filtering (6 Slides)
Estimating the Degradation Function Ph. Cattin: Image Restoration

(42) Estimating the Degradation Function

So far we concentrated mainly at the additive noise term \eta(x,y). In the remainder of this presentation we concentrate on ways to remove the degradation function H.


Fig 6.48 Image degradation model


Estimating the Degradation Function Ph. Cattin: Image Restoration

(43) Estimating the Degradation Function (2)

There are three principle methods to estimate the degradation function H

  1. Observation
  2. Experimentation
  3. Mathematical modelling

The process of restoring a corrupted image using the estimated degradation function is sometimes called blind deconvolution, as the true degradation is seldom known completely.

In recent years a forth method based on maximum-likelihood estimation (MLE) often called blind deconvolution has been proposed.
Beware: All multiplications/divisions in the equations in this part of the lecture are componentwise.


Estimating the Degradation Function Ph. Cattin: Image Restoration

(44) Estimating by Image Observation

Given

A degraded image without any information about the degradation function H.

Solution
  1. Select a subimage with strong signal content → g_s(x,y)
  2. By estimating the sample grey levels of the object and background in g_s(x,y) we can construct an unblurred image of the same size and characteristics as the observed subimage → \hat{f}_s(x,y)
  3. Assuming that the effect of noise is negligible (thus the choice of a strong signal area), it follows from Eq 6.2 that

    \[H_s(u,v)=\frac{G_s(u,v)}{\hat{F}_s(u,v)}\]

  4. Thanks to the shift invariance we can deduce H(u,v) from H_s(u,v)


Estimating the Degradation Function Ph. Cattin: Image Restoration

(45) Estimating by Experimentation

Given

Equipment similar to the equipment used to acquire the degraded image

Solution
  1. Images similar to the degraded image can be acquired with various system settings until it closely matches the degraded image
  2. The system can then be characterised by measuring the impulse (small bright dot of light) response. A linear, shift invariant system can be fully described by its impulse response.
  3. As the Fourier transform of an impulse is a constant, it follows from Eq 6.2

    \[H(u,v)=\frac{G(u,v)}{A}\]

    where G(u,v) is the Fourier transform of the observed image, and A is a constant describing the strength of the impulse.


Estimating the Degradation Function Ph. Cattin: Image Restoration

(46) Estimating by Mathematical Modelling

Degradation modelling has been used for many years. It, however, requires an indepth understanding of the involved physical phenomenon. In some cases it can even incorporate environmental conditions that cause degradations. Hufnagel and Stanly proposed in 1964 a model to characterise atmospheric turbulence.

\[H(u,v)=e^{-k(u^2+v^2)^{5/6}}\](6.37)

where k is a constant that depends on the nature of the turbulence.

Except of the 5/6 power the above equation has the same form as a Gaussian lowpass filter.



Estimating the Degradation Function Ph. Cattin: Image Restoration

(47) Illustration of the Atmospheric Turbulence Model


Fig 6.49: Aerial image of Pompeii

Fig 6.50: Pompeii with mild turbulence k=0.00025

Fig 6.51: Pompeii with medium turbulence k=0.001

Fig 6.52 Pompeii with heavy turbulence k=0.0025


Inverse Filtering

Outline (Inverse Filtering)

  1. Image Degradation & Restoration Model (2 Slides)
  2. Noise Models (12 Slides)
  3. Restoration in the Precense of Noise Only (1 Slides)
  4. Periodic Noise Reduction by Frequency Domain Filtering (15 Slides)
    1. Bandreject Filter (2 Slides)
    2. Bandpass Filter (2 Slides)
    3. Notch Filter (6 Slides)
    4. Optimum Notch Filtering (5 Slides)
  5. Estimating the Degradation Function (6 Slides)
  6. Inverse Filtering (21 Slides)
    1. Direct Inverse Filtering (5 Slides)
    2. Wiener Filtering (9 Slides)
    3. Geometric Mean Filter (1 Slides)
    4. Constrained Least Squares Filtering (6 Slides)

Direct Inverse Filtering

Direct Inverse Filtering Ph. Cattin: Image Restoration

(50) Direct Inverse Filtering

The simplest approach to restore a degraded image is to form an estimate of the form

\[\hat{F}(u,v)=\frac{G(u,v)}{H(u,v)}\](6.38)

and then obtain the corresponding image by inverse Fourier transform. This method is called direct inverse filtering. With the degradation model Equation 6.2 we get

\[\hat{F}(u,v)=F(u,v)+\frac{N(u,v)}{H(u,v)}\](6.39)

This simple equation tells us that, even if we knew H(u,v) exactly we could not recover the undegraded image f(x,y) completely because:

  1. the noise component is a random function whose Fourier transform N(u,v) is not known, and
  2. in practice H(u,v) can contain numerous zeros.
Direct Inverse filtering is seldom a suitable approach in practical applications


Direct Inverse Filtering Ph. Cattin: Image Restoration

(51) Two Practical Approaches for Inverse Filtering

  1. Low Pass Filter
    \[\hat{F}(u,v)=\frac{G(u,v)}{H(u,v)}L(u,v)\](6.40)

    where L(u,v) is a low-pass filter that should eliminiate the very low (or even zero) values often experienced in the high frequencies.

  2. Constrained Division
    \[\hat{F}(u,v)=\begin{cases}\frac{G(u,v)}{H(u,v)}&\mbox{if}|H(u,v)|\ged\\G(u,v)&\mbox{if}|H(u,v)|<d\end{cases}\](6.41)

    gives values, where H(u,v) is lower than a threshold d a special treatment.

These appraoches are also known as Pseudo-Inverse Filtering.


Direct Inverse Filtering Ph. Cattin: Image Restoration

(52) Artificial Inverse Filtering Example


Fig 6.53: Checkerboard sample

Fig 6.54: Motion blur of \unit{7}{pixel} in 45^\circ direction

Fig 6.55: Additive noise with \mu=0,\sigma=0.1

Fig 6.56: Motion and noise degraded checkerboard


Direct Inverse Filtering Ph. Cattin: Image Restoration

(53) Artificial Inverse Filtering Example (2)

Although the degradation function H(u,v) was known exactly when calculating this example, the degradation could not be undone. Zero values and small values in H(u,v) dominate and render the reconstruction useless.

Although very tempting from its simplicity, the direct inverse filtering approach does not work for practical applications. Even if the degradation H is known exactly.


Fig 6.57: Direct inverse filtering result


Direct Inverse Filtering Ph. Cattin: Image Restoration

(54) Inverse Filtering Example (2)

Inverse filtering with all frequencies is useless (topleft). Limiting the inverse filtering to the lower frequencies yields useful results as can be seen in the top/right and lower/left image. The optimum radius seems to be around 80 with lower radiuses losing to much of the high frequencies and higher radiuses introducing too much artefacts (lower/right).

Fig 6.58: Inverse filtering of Figure 6.52

Fig 6.59: Inverse filtering for freq. radius \le40

Fig 6.60: Inverse filtering for frequencies radius \le80

Fig 6.61: Inverse filtering for frequencies radius \le120


Wiener Filtering

Wiener Filtering Ph. Cattin: Image Restoration

(56) Wiener Filtering

The best known improvement to inverse filtering is the Wiener Filter. The Wiener filter seeks an estimate \hat{f} that minimises the statistical error function

\[e^2=E\{(f-\hat{f})^2\}\](6.42)

where E is the expected value and f the undegraded image. The solution to this equation in the frequency domain is

\[\hat{F}(u,v)=\left[\frac{1}{H(u,v)}\frac{|H(u,v)|^2}{|H(u,v)|^2+S_\eta(u,v)/S_f(u,v)}\right]G(u,v)\](6.43)

where

  • H(u,v) is the degradation function
  • |H(u,v)|^2=H^\ast(u,v)H(u,v) with H^\ast(u,v) the complex conjugate of H
  • S_\eta(u,v)=|N(u,v)|^2 is the power spectrum of the noise
  • S_f(u,v)=|F(u,v)|^2 is the power spectrum of the undegraded image


Wiener Filtering Ph. Cattin: Image Restoration

(57) Wiener Filtering (2)

If the noise spectrum |N(u,v)|^2 is zero, the noise-to-signal power ratio S_\eta(u,v)/S_f(u,v)=|N(u,v)|^2/|F(u,v)|^2=0 vanishes and the Wiener Filter reduces to the inverse filter.

\begin{eqnarray*}\hat{F}(u,v)&=&\left[\frac{1}{H(u,v)}\frac{|H(u,v)|^2}{|H(u,v)|^2+{\color{red}S_\eta(u,v)}/S_f(u,v)}\right]G(u,v)\\[2ex]&=&\left[\frac{1}{H(u,v)}\frac{|H(u,v)|^2}{|H(u,v)|^2+0}\right]G(u,v)\\[2ex]&=&\frac{1}{H(u,v)}G(u,v)\end{eqnarray*}(6.44)

This is no problem, as the inverse filter works fine if no noise is present.



Wiener Filtering Ph. Cattin: Image Restoration

(58) Parametric Wiener Filter

The main problem with the Wiener filter is that the power spectrum |F(u,v)|^2 of the undegraded image is seldom known.

A frequently used approach when these quantities cannot be estimated is to approximate Eq 6.43 by the expression

\[\hat{F}(u,v)=\left[\frac{1}{H(u,v)}\frac{|H(u,v)|^2}{|H(u,v)|^2+K}\right]G(u,v)\](6.45)

where K is a user selected constant.

This simplification can be partly justified when dealing with spectrally white noise, where the noise spectrum S_\eta(u,v)=|N(u,v)|^2 is constant. However, the problem still remains that the power spectrum S_f(u,v)=|F(u,v)|^2 of the undegraded image is unknown and must be estimated.

Even if the actual ratio K is not known, it becomes a simple matter to experiment interactively varying the constant and viewing the results.


Wiener Filtering Ph. Cattin: Image Restoration

(59) Artificial Wiener Filtering Example

This example shows the performance of the Wiener filter on the example Fig 6.56 using (1) the approximation of Eq 6.45 with K=0.04 in Fig 6.62 and (2) the Wiener filter using full knowledge of the noise and undegraded image's power spectra in Fig 6.63.


Fig 6.62: Parametric Wiener filter using a constant ratio

Fig 6.63: Wiener filter knowing the noise and undegraded signal power spectra


Wiener Filtering Ph. Cattin: Image Restoration

(60) Wiener Filtering Example with Little Noise

With only little to no noise the three methods seem to be comparable.

Fig 6.64: Pompeii blurred with Gaussian kernel \sigma=5, and Gaussian noise \mu=0,\sigma=0.001

Fig 6.65: Result of direct inverse filtering

Fig 6.66: Result with Parametric Wiener filtering K=0.0001

Fig 6.67: Result with Wiener filtering


Wiener Filtering Ph. Cattin: Image Restoration

(61) Wiener Filtering Example with Medium Noise

With only mild noise the direct inverse filtering approach starts to fail, while the parametric Wiener and pure Wiener filtering can still cope with images.

Fig 6.68: Pompeii blurred with Gaussian kernel \sigma=5, and Gaussian noise \mu=0,\sigma=0.01

Fig 6.69: Result of direct inverse filtering

Fig 6.70: Result with Parametric Wiener filtering K=0.0001

Fig 6.71: Result with Wiener filtering


Wiener Filtering Ph. Cattin: Image Restoration

(62) Wiener Filtering Example with Heavy Noise

The direct inverse filtering approach completely fails. The parametric Wiener does a decent job, but has some ghosting artefacts. The pure Wiener filtering still does a very good job.

Fig 6.72: Pompeii blurred with Gaussian kernel \sigma=5, and Gaussian noise \mu=0,\sigma=0.1

Fig 6.73: Result of direct inverse filtering

Fig 6.74: Result with Parametric Wiener filtering K=0.0001

Fig 6.75: Result with Wiener filtering


Wiener Filtering Ph. Cattin: Image Restoration

(63) Comparison Param. Wiener vs. Wiener Filtering

pompeii_gaussblur=5_noise=0.1_param_wiener.pngpompeii_gaussblur=5_noise=0.1_wiener.png

(a)

(b)

Fig 6.76 Aerial image of Pompeii with Gaussian blur (\sigma=5) and noise (\mu=0,\sigma=0.1) (a) restored with the Parametric Wiener (K=0.0001), (b) (mouseover) same image restored with Wiener filter.



Wiener Filtering Ph. Cattin: Image Restoration

(64) Drawback of the Wiener Filter

The problem of having to know something about the degradation function H(u,v) is common to all methods discussed in this part.

The Wiener filter has the additional difficulty, that

  • the power spectra of the undegraded image and
  • the power spectra of the noise

must be known.

Although an approximation by a constant is possible as in the Parametric Wiener filter, see Fig 6.62, this approach is not always suitable.



Geometric Mean Filter

Geometric Mean Filter Ph. Cattin: Image Restoration

(66) Geometric Mean Filter

The Geometric Mean Filter is a generalisation of inverse filtering and the Wiener filtering. Through its parameters \alpha,\beta it allows access to an entire family of filters.

\[\hat{F}(u,v)=\left[\frac{H^\ast(u,v)}{|H(u,v)|^2}\right]^\alpha\left[\frac{H^\ast(u,v)}{|H(u,v)|^2+\beta\frac{S_\eta(u,v)}{S_f(u,v)}}\right]^{1-\alpha}G(u,v)\](6.46)

Depending on the parameters \alpha,\beta the filter characteristics can be adjusted

  • \alpha=1: inverse filtering
  • \alpha=1/2,\beta=1: spectrum equalisation filter
  • \alpha=0: parametric Wiener filter → \beta=1 Wiener


Constrained Least Squares Filtering

Outline (Constrained Least Squares Filtering)

  1. Image Degradation & Restoration Model (2 Slides)
  2. Noise Models (12 Slides)
  3. Restoration in the Precense of Noise Only (1 Slides)
  4. Periodic Noise Reduction by Frequency Domain Filtering (15 Slides)
    1. Bandreject Filter (2 Slides)
    2. Bandpass Filter (2 Slides)
    3. Notch Filter (6 Slides)
    4. Optimum Notch Filtering (5 Slides)
  5. Estimating the Degradation Function (6 Slides)
  6. Inverse Filtering (21 Slides)
    1. Direct Inverse Filtering (5 Slides)
    2. Wiener Filtering (9 Slides)
    3. Geometric Mean Filter (1 Slides)
    4. Constrained Least Squares Filtering (6 Slides)
Constrained Least Squares Filtering Ph. Cattin: Image Restoration

(68) Constrained Least Squares (Regularised) Filtering

The Constrained Least Squares Filtering approach only requires knowledge of the mean and variance of the noise. As shown in previous lectures these parameters can be usually estimated from the degraded image.

The definition of the 2D discrete convolution is

\[h(x,y)\astf(x,y)=\frac{1}{MN}\sum_{m=0}^{M-1}\sum_{n=0}^{N-1}f(m,n)h(x-m,y-n)\](6.47)

Using this equation we can express the linear degradation model g(x,y)=h(x,y)\astf(x,y)+\eta(x,y) in vector notation as

\[{\bfg=Hf+\eta}\](6.48)


Constrained Least Squares Filtering Ph. Cattin: Image Restoration

(69) Constrained Least Squares Filtering (2)

It seems obvious that the restoration problem is now reduced to simple matrix manipulations. Unfortunately this is not the case. The problem with the matrix formulation is that the H matrix is very large MN\timesMN and that its inverse is firstly very sensitive to noise and secondly does not necessarily exist.

One way to deal with these issues is to base optimality of restoration on a measure of smoothness, such as the second derivative of the image, e.g. the Laplacian.

\[C=\sum_{x=0}^{M-1}\sum_{y=0}^{N-1}\left[\nabla^2f(x,y)\right]\](6.49)

subject to the constraint

\[||{\bfg-H\hat{f}}||^2=||{\bf\eta}^2||\](6.50)

where ||\cdot|| is the Euclidian norm, and \hat{f} an estimate of the undegraded image.



Constrained Least Squares Filtering Ph. Cattin: Image Restoration

(70) Constrained Least Squares Filtering (3)

The Frequency domain solution to this optimisation problem is given by

\[\hat{F}(u,v)=\left[\frac{H^\ast(u,v)}{|H(u,v)|^2+\gamma|P(u,v)|^2}\right]G(u,v)\](6.51)

where \gamma is a parameter that must be adjusted so that the constraint in Eq 6.50 is satisfied, and P(u,v) is the Fourier transform of the Laplace operator

\[p(x,y)=\begin{bmatrix}0&-1&0\\-1&4&-1\\0&-1&0\end{bmatrix}\](6.52)

Note: p(x,y) must be properly padded with zeros prior to computing the Fourier transform.

It is possible to adjust the parameter \gamma interactively until acceptable results are obtained.


Constrained Least Squares Filtering Ph. Cattin: Image Restoration

(71) Optimal Selection for Gamma

If we are interested in optimality, the parameter \gamma must be adjusted so that the constraint ||{\bfg-H\hat{f}}||^2=||{\bf\eta}^2|| in Eq 6.50 is satisfied.

It turns out that

\[\gamma=MN[\sigma^2+m_\eta^2]\](6.53)

is the optimal selection given the above optimisation equation.

It is important to understand, that optimum restoration in the sense of constrained least squares does not necessarily imply best in the visual sense. In general, automatic determined restoration filters yield inferior results to manual adjustment of filter parameters.


Constrained Least Squares Filtering Ph. Cattin: Image Restoration

(72) Artificial Constrained Least Squares Filter Example

The example on the left clearly shows that our chosen optimisation criterion (laplace) indeed resulted in a smooth image. The noise seems to have been entirely removed. However, as the manually selected \gamma on the right side shows there are settings for gamma that provide from a visual standpoint a better solution.

This example shows the performance of the Constrained Least Squares Filter on the example Fig 6.56 using (1) the theoretical optimal value for \gamma=40.96 in Fig 6.77 and (2) the constrained least squares filter with a manually selected \gamma of 10.1861 6.78.


Fig 6.77: Result with optimal \gamma

Fig 6.78: Results with manually selected \gamma


Constrained Least Squares Filtering Ph. Cattin: Image Restoration

(73) Constrained Least Squares Filter Example

The CLS filter shows good performance although only one single parameter has to be adapted (ideally manually).

Fig 6.79: Pompeii blurred with Gaussian kernel \sigma=5

Fig 6.80: Result with low Gaussian noise of \sigma=0.001

Fig 6.81: Result with medium Gaussian noise of \sigma=0.01$

Fig 6.82: Result with high Gaussian noise of \sigma=0.1


April 29th, 2014 Introduction to Signal and Image Processing