Intuitionistic Fuzzy Sets Based Inpainting for Reconstruction of Heritage Images

Image inpainting is a process of reconstructing damaged or missing part of an image. Initially the work of restoration was performed by skilled artists and was limited to paintings and other artwork of eminence but this process was very tedious and strenuous and therefore digital image inpainting was introduced. The application of inpainting in reconstruction of heritage images is garnering a lot of attention from researchers. The reconstruction of heritage images using inpainting poses a challenging task because of its very high resolution and high meaning full structure content. In traditional exemplar based algorithms the patch with the minimum distance or high similarity may not always be the best match patch. One of the characteristic functions of the Intuitionistic fuzzy set is its definition of degree of hesitancy. In the proposed paper this concept is used to find best exemplars. The mathematical model of intuitionistic fuzzy set is studied and a modified exemplar algorithm based on Intuitionistic fuzzy sets is proposed to solve image inpainting problem. The proposed intuitionistic based inpainting algorithm is applied to heritage images. The results are compared with existing and earlier proposed algorithms on the basis of subjective inpainting quality assessment method


Introduction
Image inpainting is a process of reconstructing damaged or missing part of an image. Initially the work of restoration was performed by skilled artists and was limited to painting and other artwork of eminence but this process was very tedious and strenuous and therefore digital image inpainting was introduced. Image inpainting can be thought of as an interpolation problem in which information from known portion of image is interpolated to unknown part of image.
In recent years, research in image inpainting is been very active due to its immense importance where damaged images are restored from scratches, text overlays on images with reference to impaired image transmission or disocclusion in image based rendering (IBR), red-eye correction etc. This technique can also be used to interpolate the lost blocks in the coding and transmission of images. Image inpainting has also found importance in areas such as visual reconstruction of heritage sites, special effects in films, editing, medical image processing, image compression etc. Image inpainting can be thought as a unique problem without particular solution. It is therefore necessary to consider image priors. The inpainting technique can be categorized broadly as Diffusion based and Exemplar based on the basis of image priors such as smoothness priors, self similarity prior and sparse priors. In these methods an assumption is made that the known pixels and unknown pixels of the images have same geometrical structures or statistical properties.
Diffusion based inpainting works well for smaller damaged area but it creates blur for larger damaged area [1]- [4]. In order to deal with this blurring effect, exemplar based inpainting was introduced. In exemplar based inpainting approach, the region to be inpainted is filled by most similar patches from source region. The exemplar based inpainting approach considers both structural and textural information of image while filling up target region. A filling algorithm is said to be better when it gives higher priority of filling to those regions of the target or unknown area which lie on the continuation of image structures. The main advantages of this approach are balanced region filling, speed efficiency, accurate propagation of linear structure and accuracy in synthesis of texture. The priority of the patch is decided by two terms namely confidence term and data term. Priority computation is biased towards those patches which are on the continuation of strong edges and are supposed to be surrounded by high confidence pixels.
The Exemplar based inpainting method [5] for inpainting bigger patches has received vast acceptance because of its simplicity and effectiveness through computation of priority value. But as the algorithm progresses towards Research Article the center of the unknown region with more number of unknown pixels, the overall confidence of the patch drops significantly, thus dropping the priority value. This results into less visually acceptable inpainting. In this paper the crisp nature of confidence values is converted into vagueness using the concepts of intutionistic fuzzy sets to give more visibly acceptable outputs.
The inpainting of heritage images where high amount of uncertainty is present is a challenging task. These images consist of complex textures and structures with greater amount of uncertainty. An intuitionistic fuzzy set based inpainting approach is applied to reconstruct heritage images.

2.1.Introduction to inpainting problem
The image to be inpainted is denoted by "I". This image is divided in two parts where undamaged region is called known region and damaged region or region to be filled is unknown region. Let the region to be inpainted be called as target region and is denoted as "θ". The known part of image also called as source region be denoted as "ϕ". The target region boundary is given by "δθ" as shown in Figure 1(a). The task is to fill a patch ᴪ y , of a given window size, where y∊ δθ , with the best matching patch, ᴪ x , from the source region.

2.2.Exemplar based inpainting algorithm
Criminisi et al. in [5] proposed an algorithm which simultaneously propagates texture and structure information is single algorithm. The inpainting algorithm based on exemplar approach includes the following four main steps: a) Initialization of the target Region "θ": The damaged region is selected. A mask is created by extracting the target region and representing it with proper data structure. b) Identifying the fill front "δθ" and computing filling priority: c) Finding the best exemplar. d) The algorithm is used to search for the most matching patch from the source region. The distance metric, d, is used to compute the similarity between the patches in source region and target region.

2.3.Selection of damaged region
The first step in inpainting is to select damage region or region to be removed. The damaged region can be selected manually or through segmentation techniques. The process of mask selection mostly done manually and require user interference. However, Rong-Chi Change [6] developed a new mechanism which can automatically detect defective portion in photo including damage by color in spray and scratch drawing. Hongying Zhang [7] proposed new algorithm for detection of arbitrary scratches for image inpainting and is based on uniform gray structural information of scratches. The scratches and other linear structures can also be detected using various edge detection techniques. Selecting damaged region manually makes inpainting technique semi-automatic. The process of object selection, removal and mask creation is shown in Figure 2. The image to be inpainted is given as Eq. no (1) Where "I n+1 " denotes the image to be inpainted after applying mask as shown in Figure 2(d) and "M" is the mask of the damaged region. The value n = 2 for grayscale images (2-dimentional) or n = 3 for RGB color images (3-dimentional). Once the mask is created the user has to select patch size for processing. The patch size should be greater than the size of the texel for successful inpainting. Lizarraga-Morales et al. in [8] provides a method to find texel size.

2.4.Computing the patch priority
The initial fill front is obtained through boundary of the target region. A priority value is computed for a patch ᴪ y of given window size around every pixel on the boundary. The priority value is computed using predefined priority function given as Eq. (2), P y = C y * D y WhereC(y) is the confidence term and the data term is given as D(y). For the pixel "y" on the boundary, the confidence value is computed as, ᴪ y , which is the area of the patch surrounding pixel y. The data term is given as, D y = ∇I y ⊥ .n y α (3) ∇I y ⊥ -is direction perpendicular to the gradient called as isophotes n y -Unit vector perpendicular to each pixel on boundary.
The isophote direction and gradient direction is shown in Figure 3. "α" is used for normalization and for grayscale images its value is 255. The confidence of a patch is given as, During initialization, C(y) has value 0 when y∊ θ (unknown region) and C(y) is 1 when y∊ ϕ that is source region. The confidence map is constructed by taking complement of mask. A search for matching exemplar is conducted using different similarity priors and the most similar is copied in the damaged region according to the priority value assigned.

Finding the best exemplar
The algorithm searches for best matching patch for the patch ᴪ y in the target region with the patch ᴪ x from the source region. The similarity between two patches is computed through distance metric "d" as given in Eq. (5) Where, ᴪ x -is the best matched patch.
A quantitative measure of the degree of matching or similarity between two images, or image patches, A and B is provided by the distance metric "d". The similarity measures using distance metrics are broadly classified as, pixel based similarity measures and statistical based similarity measures. The distance function refers to difference between samples that are regarded as points in high dimensional space. The similarity measure is complement of distance measure. The minimum distance between two patches is considered as best matching for which similarity index is highest. In earlier work of exemplar based image inpainting [5], sum of squared difference (SSD) is used for similarity measurement. This similarity measure is based on pixel to pixel intensity differences between two images. The matching point can be determined by considering the location of minimum value in the image matrices. The sum of squared difference between two patches ᴪ y and ᴪ x can be given as Eq. (6) The minimum value of difference will give the best matching patch. But, in some cases it may occur that multiple patches may have numerically close SSD and the patch with the lowest value of SSD may not necessarily be the most visually plausible one. It is observed that SSD is biased towards uniform textures. This problem is answered in [9] where mean SSD distance is proposed to better reflect the average of similarity between the two patches. The mean SSD is given as, Where, , and denotes mean of image intensity of individual color channel. Sum of squared absolute difference (SSAD) of the two feature vectors is calculated for finding matching patches in [10] as given in Eq. [8] which is computationally less complex.

2.6.Updating the fill front
The fill front "δθ" is updated for each iteration after copying the matching patches to the target region. The confidence value of the new fill front is computed. As the algorithm propagates from outside to the inside of the region being inpainted the fill front goes on reducing in a onion peel fashion.
The steps are iterated until all the pixels in target area are filled.

Intuitionistic fuzzy sets for image inpainting 3.`1.Introduction to intuitionistic fuzzy sets
Different variations of higher order fuzzy sets have been proposed by many researchers motivated from Zadeh"s proposed fuzzy sets (FSs) theory [11]. Intuitionistic fuzzy sets (IFSs) are among one such notion proposed by  which has turned out to be a suitable tool for modeling the hesitancy which arises from imprecise information. IFSs are defined by the characteristic functions, namely the membership and the non-membership, describing the degree of belongingness or non-belongingness of an element of the universe to the IFS respectively and degree of hesitancy.
In exemplar based inpainting a distance metric is used for finding the best matching patch from the known region. In cases where two more patches are located at same minimum distance with respect to the target patch, hesitancy is created. In traditional confidence term as in [5] the pixel value for absence of data = 0. This results into fast dropping of confidence value when large amount of damage is present. This uncertainty is defined by using degree of hesitancy. It has been observed that the proposed IFS based inpainting approach gives visually acceptable results as compared to the most popular algorithm in [5].

3.2.Mathematical model of intuitionistic fuzzy sets
Let "A" be the fuzzy set defined over finite set X = {x 1 , x 2 ,x 3 …..x n } be mathematically represented as given in Eq. (9) A = X, μ A x |xϵX (9) Where the degree of membership of an element x ∈ X is given by the function as given by Eq. (10) v A (x)is the non membership is considered as complement of the membership and can be given as Eq. (11) The intuitionistic fuzzy sets proposed in [12] are generalized representation of fuzzy sets where each element is characterized by a membership and a non membership value.
An intuitionistic fuzzy set (IFS) in X is represented mathematically as given in Eq. (12) With the necessary condition The second parameter which is hesitancy degree is can be defined as Eq. (13) AA Where the necessary condition of hesitancy degree is A 0 (x) 1   

3.3.Proposed image representation using Intuitionistic fuzzy set
The intutionistic fuzzy generator is used for image reconstruction of intuitionistic fuzzy images [17], [18], [19]. In the proposed method instead of complete image, a patch ᴪ y and patch ᴪ x formed around the pixel y and x are considered as set A and set B respectively , where y ∈ δθ (boundary of target region) and x ∈ ϕ (source region). The patch ᴪ y which represents set A is IFS with membership function given as Eq. (14) 2 2 (y m) 2 Where, c -is mean of patch centered around pixel y on boundary σ -is standard deviation ( selected as 0.7) The membership of set B is represented by Eq. (15) 2 2 (x m) 2 The non membership of set A and B is represented by Sugeno non-membership function and is given as Eq. (16) and Eq. (17) where the value of λ = 1.
The hesitancy value for set A and B is defined by using Eq. (13) .
A similarity measure is crucial for measuring the similarity degree between two objects. Similarly, distance measure is a vital tool which is used to describe the differences between two sets and is duality of similarity measure. Yingjie Yang et al in [20] compared 2D and 3D distances as extended to IFS. Szmidt et al in [21] provided a generalization of existing 2D and 3D distance metrics as applied to IFS Computational complexity of the algorithm should also be considered while selecting a distance metric where the quality is also not compromised. In this paper a generalization of 3D Sum of Absolute Difference (SAD) is used as a distance measure to find the similarity between IFS A and IFS B as given in Eq. (18).
The heritage sites all around the world have undergone damage due to a number of natural calamities, destruction due to war and damage from visitors, resulting in their damages. Thus many sites have been restricted for visitors and are declared as UNESCO world heritage sites.
An effort is made here to reconstruct the damaged images of Bishnupur heritage sites. A database is available in [22] where images of different temples of Bishnupur are catalogued. The heritage images are a challenge for reconstruction because of its high structure contents and uncertainties. The traditional exemplar based algorithm is structure driven and gives priority to the structures lying on the boundaries on region being inpainted. But the patch with highest priority may not be the suitable candidate for inpainting . This uncertainty is considered in the proposed IFS based inpainting algorithm .

3.4.Simulation and result
The result of the proposed algorithm in reconstruction of "Jorbangla" images is shown in Figure 4. The images jb1, jb6 and jb5 are obtained from "Jorbangla" temple image from BHID database. Various portions of the "Jorbangla" image are cropped to create the images for inpainting.
Patch size of all three images is different.  Wang et al. in [28], assessment is done using modified SSIM index. This index is composed of three aspects: luminance, definition and gradient similarity which evaluates the blur artifacts produced during inpainting. This approach is suitable only when the reference image is available.
In this paper, a subjective analysis is conducted for the quality assessment of the inpainted region. A survey was conducted on 16 observers out of which 8 were female. All the observers had normal or corrected vision. The observers were selected on the basis of gender, age, profession and knowledge. The observers were provided with the inpainted results using existing and proposed method along with the original image to compare. The time limit was not set and they were allowed to observe for any duration of time before responding. This is done so as to remove existing lacunas in the subjective assessment such as fatigue of the observer, attention span and other environmental conditions. The questionnaire for observer"s survey is shown is Table 1. The Qualitative analysis through subjective method is shown in Figure 5.

4.Conclusion
The problem of reconstructing heritage images is analyzed and a possible solution is proposed using intuitionistic fuzzy sets. The heritage images have high structure content with non uniform backgrounds. The use of IFS has been found to give better results than the existing algorithms.
The quality analysis of the existing and proposed algorithm is done using qualitative approach. The quantitative IIQA measures were give insufficient measures and were found to be inaccurate hence a subjective analysis is conducted. The subjective assessment is obtained through observer score and is more accurate as human perception is found to be the best judge. In cases where the object being removed is overlapping with another Instance and semantic segmentation can be used to separately label the object and remove it. Deep learning algorithms and artificial intelligence can be used to build human perception model to produce quantitative methods for IIQA. Table1: Observer"s survey for inpainted result Sr.no.