Image Inpainting – Filling in the gaps.
Abstract
Inpainting is the process of rebuilding lost or damaged parts of images and videos. Image inpainting, also called hole filling, is the technique of image reconstruction by filling or replacing the region where there is damage or intentional removal of objects. The outcome is to make the observer unaware of the restoration process. In ancient times, restoration of paintings was done by hand and was time-consuming. With the invention of computers and mathematics, image inpainting has become automated and faster for primarily digital imagery. There are three main documented methods for performing image inpainting using mathematical algorithms. Namely, partial differentiation equations, exemplar-based, and convolution-filter based equations. In this proposal, we recommend reintroducing and enhancing a novel way of performing image inpainting that uses scene matching with a larger source of image dataset to perform image inpainting.
Introduction
Image inpainting plays an essential role in the restoration of digital images by way of filling in holes or gaps left by damage. This reconstruction is for replacing regions where distortions or undesired objects. With a high interest in this area, there has been a considerable amount of research carried out with several reviews done on the plethora of proposed algorithms. Most literature surveyed identified three main types of image inpainting algorithms. Namely, partial differentiation equation-based (PDE), exemplar-based, and convolution-filter based algorithms. In one review (Vreja & Brad, 2014), five different methods were evaluated using a standard benchmark by measuring the peak signal-to-noise ratio (PSNR). That is PSNR shows how much an algorithm can enhance a deteriorated image to more closely resemble the original (Hore & Ziou, 2010). Thus, accurately concluding the effectiveness of the equation. There are also hybrid methods that use a combination of the three methods listed above.
Literature Review
PDE algorithms work by propagating or diffusing a target region with data from a known region pixel by pixel. With several variants including linear, non-linear, isotropic (parallel diffusion), anisotropic (diffusion varies with direction), this method works well for the completion of lines and curves (Telea, 2004). Bertalmio, Sapiro, Caselles, and Ballester (2000) used an algorithm based on fluid dynamics theories. Derived from the Navier-Stokes equation, images are restored using this approach. PDE is suitable for filling in small, nontextured target regions but does not fare well with more significant regions as it produces blurring.
Get Help With Your Essay
If you need assistance with writing your essay, our professional essay writing service is here to help!
Essay Writing Service
Exemplar based methods use a texture synthesis process whereby unknown patches in the target regions are patched using similar patches in known regions (Criminisi, Perez, & Toyama, 2004). This method has two critical phases: determining the fill-in order and choosing good exemplars (Deng, Huang, & Zhao, 2015). Exemplar based algorithms perform exceptionally well with restoring large regions; however, the only drawback is the high computational cost it incurs (Criminisi et al., 2004).
Convolution-filter based algorithm inpaints a target image by convolving a proper kernel with the neighbourhoods damaged pixels. Oliveira Richard and Chang (2001) presented a fast image algorithm that works by convolving an image with a Gaussian kernel (i.e., computing the weighted averages of pixels’ neighbourhoods). Equivalent to isotropic diffusion (linear heat equation). The algorithm uses a weighted average matrix that only considers contributions from the neighbour pixels. Hadhoud, Moustafa, and Shenoda (2005) modified the Oliveira algorithm by implementing time reduction. Both the above convolution- based algorithms are quick inpainting techniques. However, they produce poor results in damaged edges with high contrast.
Hybrid methods take the strengths of both the PDE and exemplar-based algorithms. Bertalmio, Vese, Sapiro, and Osher (2003) proposed an algorithm using both PDE and exemplar inpainting missing regions on images. The technique decomposes the main image into two images, image structure, and image texture. The combination of these two processed images is the result of both images inpainted by their methods. Another method proposed filled in smaller regions by copying blocks from the outside, which was preceded by matching contours that cross the edge of the occluded area in its interior (Efros & Leung, 1999). All methods described above involve complex algorithms with each having benefits over another. Although the results produced were sometimes impressive, each method has its drawbacks and generally all take longer times to perform the restoration.
Methodology
We propose to build and enhance a model first introduced by Hays and Efros (2007). Their research involved a method that patches up holes in images based on searching for and finding similar images in a dataset of 2 million images. The dataset used was a collection of images from Flickr groups. A low dimensional scene descriptor is then used to find similar images (Oliva & Torralba, 2006), which was relatively fast to find the nearest scenes. After finding a match or matches, using a graph cut (Boykov, Veksler, & Zabih, 1999), they added the source images to the target image then the Poisson solver of (Agarwala et al., 2004) is used to blend the edges. Their method falls under the classification of learning-based imaging inpainting techniques. The method they employed produced believable results; however, due to the limited number of images used, some did not produce realistic results. We also replace their scene matching algorithm with a faster and more accurate one.
To improve the accuracy of matching images required for the region in a target image that needs inpainting, we will firstly download a more extensive collection of images from the ImageNet dataset which are then indexed by assigning a unique signature to every entry (Wong et al., 2002). These images get saved to a backend MySQL database. For scene matching images, we employ the Image-Match technique (EdjoLabs, 2018), which compares the source with the database collection. Once we identify a match, the area matching the region required for inpainting on the target image is cut and patched onto this area. The added patch is then blended using the Poisson solver.
Timetable for Completion
Task
Deadline
The final decision on the topic, create research questions
February 1st, 2020
Literature review
March 1st, 2020
Experimentation
April 1st, 2020
Analyse, collate and compare new results to previous ones
May 30th, 2020
Present final results and report
July 4th, 2020
References
Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., . . . Cohen, M. (2004). Interactive digital photomontageACM. Symposium conducted at the meeting of the ACM Transactions on Graphics (TOG)
Bertalmio, M., Sapiro, G., Caselles, V., & Ballester, C. (2000). Image inpainting. presented at the meeting of the Proceedings of the 27th annual conference on Computer graphics and interactive techniques, https://doi.org/10.1145/344779.344972
Bertalmio, M., Vese, L., Sapiro, G., & Osher, S. (2003). Simultaneous structure and texture image inpainting. IEEE Transactions on Image Processing, 12(8), 882-889.
Boykov, Y., Veksler, O., & Zabih, R. (1999). Fast approximate energy minimization via graph cutsIEEE. Symposium conducted at the meeting of the Proceedings of the Seventh IEEE International Conference on Computer Vision
Criminisi, A., Perez, P., & Toyama, K. (2004). Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on Image Processing, 13(9), 1200-1212. https://doi.org/10.1109/TIP.2004.833105
Deng, L.-J., Huang, T.-Z., & Zhao, X.-L. (2015). Exemplar-Based Image Inpainting Using a Modified Priority Definition. PLoS ONE, 10(10), e0141199. https://doi.org/10.1371/journal.pone.0141199
Efros, A. A., & Leung, T. K. (1999). Texture synthesis by non-parametric samplingIEEE. Symposium conducted at the meeting of the Proceedings of the seventh IEEE international conference on computer vision
Hadhoud, M. M., Moustafa, K. A., & Shenoda, S. Z. (2005). Digital images inpainting using modified convolution based method. Int. J. Signal Process. Image Process. Pattern Recogn, 1-10.
Hays, J., & Efros, A. A. (2007). Scene completion using millions of photographs. ACM Transactions on Graphics (TOG), 26(3), 4.
Hore, A., & Ziou, D. (2010, 23-26 Aug. 2010). Image Quality Metrics: PSNR vs. SSIM Symposium conducted at the meeting of the 2010 20th International Conference on Pattern Recognition https://doi.org/10.1109/ICPR.2010.579
Oliva, A., & Torralba, A. (2006). Building the gist of a scene: The role of global image features in recognition. Progress in brain research, 155, 23-36.
Richard, M. M. O. B. B., & Chang, M. Y.-S. (2001). Fast digital image inpainting Symposium conducted at the meeting of the Appeared in the Proceedings of the International Conference on Visualization, Imaging and Image Processing (VIIP 2001), Marbella, Spain
Telea, A. (2004). An Image Inpainting Technique Based on the Fast Marching Method. Journal of Graphics Tools, 9(1), 23-34. https://doi.org/10.1080/10867651.2004.10487596
Vreja, R., & Brad, R. (2014). Image inpainting methods evaluation and improvement. TheScientificWorldJournal, 2014, 937845-937845. https://doi.org/10.1155/2014/937845
Essay Writing Service Features
Our Experience
No matter how complex your assignment is, we can find the right professional for your specific task. Contact Essay is an essay writing company that hires only the smartest minds to help you with your projects. Our expertise allows us to provide students with high-quality academic writing, editing & proofreading services.Free Features
Free revision policy
$10Free bibliography & reference
$8Free title page
$8Free formatting
$8How Our Essay Writing Service Works
First, you will need to complete an order form. It's not difficult but, in case there is anything you find not to be clear, you may always call us so that we can guide you through it. On the order form, you will need to include some basic information concerning your order: subject, topic, number of pages, etc. We also encourage our clients to upload any relevant information or sources that will help.
Complete the order formOnce we have all the information and instructions that we need, we select the most suitable writer for your assignment. While everything seems to be clear, the writer, who has complete knowledge of the subject, may need clarification from you. It is at that point that you would receive a call or email from us.
Writer’s assignmentAs soon as the writer has finished, it will be delivered both to the website and to your email address so that you will not miss it. If your deadline is close at hand, we will place a call to you to make sure that you receive the paper on time.
Completing the order and download