Real-time image acquisition and deblurring for underwater gravel extraction by smartphone

Real-time image acquisition and deblurring for underwater gravel extraction by smartphone

Ming-Fu Chen1,*, Yi-Chiun Kung2, Shih-Jie Chou1, Wei-Shen Lo3,and Chi-Kuei Wang3

1 Instrument Technology Research Center (ITRC), NARLabs, Taiwan
2 Department of Electrical and Control Engineering, National Chiao Tung University, Taiwan
3 Department of Geomatics, National Cheng Kung University, Taiwan

(Received 13 September 2013; Accepted 23 December 2013; Published on line 1 March 2014)
*Corresponding author: mfchen@itrc.narl.org.tw
DOI: 10.5875/ausmt.v4i1.292

Abstract:

Gravel size distribution is an important aspect of stream investigation. Using water photography to determine such distribution is a simple and cost-effective approach for gathering instream gravel information. However, good-quality images of underwater gravels in shallow areas are difficult to acquire because of the flow- and wind-induced perturbation at water surface. Thus, two Lucy–Richardson iterations are applied on an averaged image to obtain a deblurred image for gravel extraction.

A Matlab code for multi-frame image averaging and image deblurring is implemented on a laptop computer. Underwater gravel images are acquired using a video camera and processed offline. Thus, the usability of the images acquired during field investigation cannot be determined immediately. However, returning to the investigated streams for additional data gathering would be costly, and the cameras may accidentally be dropped into the water.

This paper presents multi-frame image averaging and image deblurring smartphone-based approaches for underwater gravel extraction. A waterproof smartphone is used to acquire the images, on which image deblurring is immediately conducted to test whether the images can be used for gravel extraction. The averaged image of using mean-based filter is derived during real-time image acquisition. The deblurred image is derived block-by-block because of limited memory capacity of smartphones. The time consumed for acquiring 1500 frame images with size of 1280 × 720 pixels is approximately 6 min by Sony Xperia smartphones. Image averaging can be performed in real time during image acquisition. Image deblurring is accomplished accurately and is consistent with results of the Matlab code. The processing time for image deblurring is approximately 12 min.

A compact system for underwater gravel investigation using smartphones is successfully developed in this study. Image acquisition and deblurring are completed in real time at the investigated fields. Thus, we can immediately test whether the acquired images are usable for gravel extraction, thereby improving investigation efficiency significantly.

Keywords: deblurred; Richardson–Lucy algorithm; underwater gravel, smartphone

Introduction

Underwater physical features are essential to stream investigation. The geometry, size, spread type, and other characteristics of underwater gravels affect the flow and sand transporting capability of streams. Although water photography is an effective method for investigating the underwater features in shallow regions (depth < 40 cm), flow- and wind-induced perturbations at water surface distort gravel geometry and blur the acquired images. Thus, underwater gravels are difficult to recognize from the acquired images. A number of methods[1-4] for processing acquired images have been proposed. Lo [5] proposed the application of two Lucy–Richardson iterations on an averaged (blurred) image to obtain a deblurred image of underwater gravels. An averaged image is derived by applying mean-based filter on consecutive through-water images. With this method, distorted images of underwater gravels in shallow areas can be corrected effectively. Lo conducted field investigation by using a system composed of Matlab code, a digital camera, and a laptop. However, such system has the following drawbacks: (1) the usability of the acquired images for gravel extraction cannot be confirmed instantly; (2) the camera may be lost or damaged in the stream; and (3) researchers must return to the investigated streams for additional data gathering if some of the acquired images cannot be used.

To overcome these limitations and to ensure the convenient and safe operation of the investigation system, Lo’s system is ported to waterproof smartphones, which are compact and portable devices. A smartphone is used to conduct real-time consecutive image acquisition and image deblurring to immediately ensure that the acquired images can be used.

This paper presents smartphone-based approaches for real-time image acquisition, multi-frame image averaging, and image deblurring to improve the efficiency of underwater gravel investigation.

Methodology

In this study, the smartphone-based approach consists of two tasks: (1) simultaneous real-time multi-frame image acquisition and averaging; and (2) two iterations of Lucy–Richardson algorithm on the averaged image for image deblurring. The subsequent paragraphs detailedly describe the algorithms and procedures of image averaging and deblurring for underwater gravel extraction.

Image Acquisition and Averaging

An averaged image can be derived from a consecutive image averaging of frame-by-frame calculation. The scheme is described by pseudo code as follows:

IMGAVE=IMG1, averaged image for 1 images,
IMGAVE=(IMG2 + IMGAVE)/2, averaged image for 2 images,
IMGAVE=(IMG3 + 2*IMGAVE)/3, averaged image for 3 images,
……
IMGAVE=(IMGN + (N-1)*IMGAVE)/N, averaged image for N images.

where image IMGi is the i-th acquired image, and IMGAVE is the averaged image calculated by N sets of acquired images. This method is usually implemented on laptop and personal computer platforms. Thus, the computing effort should be reduced because of the limited resources of smartphones. Image averaging based on a referred image with floating point data is selected and expressed by pseudo code as follows. The averaged image can be computed by determining the gray level summation for each corresponding pixel of acquired multi-frame images.

IMGAVE = IMG1
IMGAVE = IMGAVE + IMG2
……
IMGAVE = IMGAVE + IMGN
IMGAVE = IMGAVE / N

The final calculated image IMGAVE is the averaged image derived by N sets of acquired images. In terms of computing effort, the latter method required less effort than the former.

Image Deblurring

• Blurred Image of Through-Water Photographs

Objects in through-water images are distorted by water surface perturbation. Images manipulated by time-based mean filter are prone to motion blurring and long-time exposure [1]. Probability Density Function (PDF) can describe the water surface slop variation at a position during a time section, as shown in Figures 1 and 2 [5]. The corresponding underwater position differs for each slope because of light refraction. The gray level of an image pixel is regarded as the recorded radiance of its neighboring pixels, and an image would blur if time mean-based filter is applied. Blurred through-water images can be described by time mean-based filter and spatial mean-based filter as follows:

  1. Time mean-based filter: The gray level of a pixel on a blurred (averaged) image is derived by applying this filter on the recorded gray level of the same pixel from time i to j, as shown in Figure 1. The blurred image can be described by the following equation:
\[IM{{G}_{B}}=\frac{1}{j-i+1}\sum\nolimits_{t=i}^{t=j}{IM{{G}_{t}}},\tag{1}\]

where IMGB is the blurred image, and IMGt represents the consecutive images acquired from time i to j.

  1. Spatial mean-based filter: For each pixel on an image, the probability that the pixel can be affected by its neighboring pixels can be expressed by a function (h). Therefore, the blurred image can be regarded as the convolution of the clear image and the Point Spread Function (PSF) induced by water surface perturbation and described by following equation:
\[IM{{G}_{B}}=f*h,\tag{2}\]

where $f$ is the clear (deblurred) image without water perturbation, and h is the PSF and the probability of the neighboring pixels affecting the central pixel. Thus, the averaged image of using time mean-based filter processing is equivalent to the convolution of the clear image and $h$. Therefore, we can derive the deblurred image $f$ on the basis of following relationship between time mean-based filter and PSF:

\[\frac{1}{j-i+1}\sum\nolimits_{t=i}^{t=j}{IM{{G}_{t}}=f*h}.\tag{3}\]

• Methodology of Image Deblurring

A blurred image can be derived from the convolution of a clear image and PSF. Thus, the clear image can be calculated by applying deconvolution algorithms on the blurred image. Deconvolution algorithms have two categories: non-iterative and iterative processing. The most representative methods are Wiener filter [6] and Richardson–Lucy (R–L) algorithm [2, 4] for non-iterative and iterative deconvolution processing, respectively. R–L algorithm is described by the following equation:

\[{{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{f}}^{(r+1)}}={{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{f}}^{(r)}}\left[h*\frac{g}{h*{{{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{f}}}^{(r)}}}\right],\tag{4}\]

where ${{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{f}}^{(r+1)}}$ and ${{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{f}}^{(r)}}$ are the estimated deblurred images of $r+1$ and $r$ iterations, respectively; $g$ is the blurred image; and $h$ is PSF. R–L algorithm is an iterative method based on Bayes' theorem or on a maximum-likelihood formulation under the assumption of Poisson distribution. By given an initial estimated deblurred image, this estimated image would be improved iteratively to be close to the original unblurred image. The purpose of iteration is to find the maximum probability of similarity between the original blurred image ($g$) and the blurred image (${{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{f}}^{(r)}}$*$h$) convoluted with the deblurred image and PSF at the ${{r}^{th}}$ iteration. The result of the square bracket in Equation (4) would converge to 1 when r increases accordingly.

Al-amri et al. [7] used the two aforementioned methods to compare deblurred images and found Richardson–Lucy algorithm has better anti-noise capability and error tolerance of using worse PSF than Wiener filter. Through-water images are acquired under outdoor environment conditions, which have slight lighting variations, underwater impurities, and inaccurately estimated PSF. Thus, Richardson–Lucy algorithm is a more suitable image deblurring technique for extracting underwater gravels [5]. Accelerated Richardson–Lucy algorithm [8] is therefore adopted to achieve efficient image processing.

Blind deconvolution is a process for deriving unknown PSF and unknown clear image from a blurred image. A deblurred image can be derived by using two Richardson–Lucy iterations on a presumed clear image[2]. The first R–L processing is for obtaining the estimated PSF, whereas the second R–L processing is for deriving the estimated deblurred image. The flowchart of the Two Richardson–Lucy iterations is shown in Figure 3.

The iteration number of the R–L algorithm is an important parameter in image deblurring. If too many iterations are applied, the image noise would be magnified because of the ring effect [8]. Lo [5] found that for gravel extraction, ring effect occurred if the iteration number exceeded 20.

The converging mechanism of the R–L algorithm is of oscillating type [9]. Thus, the optimal iteration number can be found within 20 iterations, as indicated by the red point mark (Figure 4) and the mean square error between the estimated and original blurred image for underwater gravel extraction [5]. PSF of a through-water blurred image is the probability distribution of PDF of the water surface slope. Therefore, the summation of PSF is equal to 1.

• Software Porting from Matlab on a Laptop to Java/C on Smartphone

The image deblurring code is originally developed by Lo using Matlab on a laptop [5]. The Matlab code is ported to application software (APP) with Java/C on smartphone. Several instructions and functions in Matlab code are simple and lacking of detail subroutines. Therefore, great effort is exerted in developing a new code and in searching, modifying, and verifying the code from open sources to complete the APP.

The processes are automatically initialized after the smartphone powers on. Only 64 MB of memory is supported for each process because of the OS limitations, as the Sony Xperia LT26w has a total of RAM of 1 GB. The designing of the APP is limited by memory allocation; thus, the efficient management of resources of the smartphone is very important.

Experiments and Discussion

Equipment and Devices for Experiments

The following equipment and devices are prepared for embedded system development and tests on a smartphone. (1) A red point mark with a size of 7.5 cm × 9.5 cm is painted on an acrylic board with black background (Figure 5). This red point mark is fixed underwater for PSF derivation of the blurred image. (2) Waterproof smartphones of Sony Xperia LT26w and Xperia Z with dual-core CPU (1GB RAM) and four-core CPU (2GB RAM) respectively are used in this study. A smartphone is used for image acquisition and averaging, image deblurring, and display. The smartphone would play music when image acquisition or deblurring is accomplished to notify the investigators.

Image Acquisition and Averaging

The frame rate of a digital camera reaches up to 30 frames/sec (FPS), whereas that of Sony Xperia LT26w and Xperia Z smartphones ranges from 4 to 5 FPS. With this frame rate limitation, the smartphone imaging time reached approximately 6 min to acquire 1500 frame images with resolution of 1280 × 720 pixels. Figures 6 and 7 show the averaged images of two test cases obtained by Sony Xperia LT26w. Images are blurred because of water surface waves and lighting variations.

Figure 8 shows the operation flowchart for image acquisition (and image averaging). Image averaging is performed during image acquisition to reduce processing time.

Image Deblurring

The deblurred image has to be derived block-by-block, each with pixel size of 512 × 512, because of the limited storage of the smartphone. In addition, the image blocks have an overlap of 64 pixels with one another. Figure 9 shows the operation flowchart of image deblurring. Several tests are conducted to verify the deblurring function on the smartphone. Figures 10 and 11 show the deblurred images from the averaged images in Figures 6 and 7, respectively. These images indicate that underwater gravels in deblurred images are clear and suitable for gravel extraction. Figure 12 shows the PSF of the blurred averaged image in Figure 6.

By comparing the derived PSF and image deblurring results of Matlab code on a laptop with that of a smartphone firmware, we found that the accuracy and consistency of both platforms are excellent. The difference in derived PSF between Matlab code and firmware is less than 1%. The time consumed by a Sony Xperia LT26w smartphone for image deblurring is approximately 12 min. We have also ported the APP on Sony Xperia Z smartphone which has four-core CPU and 2GB RAM for trying to improve the deblurring efficiency.

Four sets of averaged images were derived from the field image acquisition provided by Lo [5]. According to the flow condition of stream water, two cases belong to the condition of unbroken standing wave (USW) which has distinct wave on water surface, and others belong to that of broken standing wave (BSW) which has distinct wave and broken spray. Table 1 is the single shot, averaged and deblurred images, and the time consumed for image deblurring on a Sony Xperia Z smartphone for each case. It’s very hard to recognize the underwater gravels by using single shot images. An investigator can determine whether the acquired images are usable or not in the field based on deblurred images as shown in Table 1. Comparing the time consumed for image deblurring, we found the efficiency of using Sony Xperia Z smartphone is not definitely better than that of using Xperia LT26w. It’s because the OS of Xperia Z smartphone is almost same as Xperia LT26. And the memory limitation of 64 MB for each process is the same.

Conclusion

The proposed approach and a compact operation system for underwater gravel extraction are successfully developed on a smartphone. The hardware platform of smartphones is characterized by lightness, portability, water resistance, and high mobility. With this method, risks of hardware damage in the stream are significantly reduced. Image acquisition and deblurring are accomplished in real time during field investigation. Thus, images can immediately be reacquired if the acquired images are unusable for gravel extraction. Consequently, investigation efficiency is substantially improved, and a great amount of time, cost, and effort is conserved.

At present, total time required for image acquisition and deblurring with a Sony Xperia LT26w is approximately 18 min. From a user point of view, smartphones with better performance in image acquisition and deblurring are expected to have better operation efficiency. But better hardware performance is no guarantee for improving the processing efficiency. Efficiency is also dependent on the characteristics of smartphone OS and the firmware. Therefore, we aim to port the firmware to other smartphones with superior performance (higher frame rate of image acquisition) and resolution (e.g., 1920 × 1080 pixels), and to reconstruct the firmware with the multi-thread architecture in the future to extend software capabilities and values.

References

  1. Z. X. Chen, "Reconstruction of through water image by using continuous photographs," M. S. Thesis, Department of Geomatics, National Cheng Kung University, Taiwan, 2010.
  2. L. B. Lucy, "An iterative technique for the rectification of observed distributions," Astronomical Journal, vol. 79, no. 6, pp. 745-754, 1974.
    doi: 10.1086/111605
  3. D. A. Fish, A. M. Brinicombe, E. R. Pike, and J. G. Walker, "Blind deconvolution by means of the Richardson-Lucy algorithm," Journal of the Optical Society of America A, vol. 12, no. 1, pp. 58-65, 1995.
    doi: 10.1364/JOSAA.12.000058
  4. W. H. Richardson, "Bayesian-based iterative method of image restoration," Journal of the Optical Society of America, vol. 62, no. 1, pp. 55-59, 1972.
    doi: 10.1364/JOSA.62.000055
  5. W. S. Lo, "Extraction of gravel size information from deblurred through water image," M. S. Thesis, Department of Geomatics, National Cheng-Kung University, Taiwan, 2012.
  6. N. Wiener, "The interpolation, extrapolation and smoothing of stationary time series," Report of the Services 19, Research Project DIC-6037 MIT, February 1942.
  7. S. S. Al-amri and N. V. Kalyankar, "A comparative study for deblurred average blurred images," International Journal on Computer Science and Engineering, vol. 2, no. 3, pp. 731-733, 2010.
  8. L. Yuan, J. Sun, L. Quan, and H. Y. Shum, "Image deblurring with blurred/noisy image pairs," ACM Transaction on Graphics, vol. 26, no. 3, p. 1, 2007.
    doi: 10.1145/1276377.1276379
  9. D. S. C. Biggs and M. Andrews, "Acceleration of iterative image restoration algorithms," Applied Optics, vol. 36, no. 8, pp. 1766-1775, 1997.
    doi: 10.1364/AO.36.001766

Refbacks

  • There are currently no refbacks.

Comments on this article

View all comments


Copyright © 2011-2017  AUSMT   ISSN: 2223-9766