Hands-On Guide To Image Extrapolation With Boundless-GAN
Suppose that given a sub-image (a part of the original image) of the wild scene, you are asked to draw the entire image beyond its boundary without having the entire image. We humans usually first imagine the entire image according to prior knowledge, then we draw the details outwards from inside progressively based on sub-image and imaginary image. This technique is called extrapolation. Image extrapolation is such a task in computer vision that aims to fill the surrounding region of a sub-image, e.g. completing the object appearing in the image or predicting the unseen view from the scene picture. This task is extremely challenging since the extrapolated image must be realistic with reasonable and meaningful context. Moreover, the extrapolated region should be consistent in structure and texture with the original sub-image. Recently, extrapolating an image is challenging for humans, but thanks to advanced machines and development in the GAN networks. A lot of effort has been made on this task to step forward, achieving good accuracy. However, existing GAN based models for image extrapolation mainly generate a whole image and paste the given part onto it, making the final image look blurry or jarring. In addition, due to distant contextual generation problems, directly applying inpainting methods tends to generate blurry or repetitive pixels with inconsistent semantics. Boundless GAN is a network developed by the Google Research team which has done semantic conditioning to the discriminator of GAN and achieved strong results on image extension with coherent semantics and visually pleasing colors and textures. In this article, we will see how we can use this network for our extrapolation task. Code Implementation: Image Extrapolation With Boundless-GAN Import all dependencies: # for numeric operations import numpy as np # for visualization import matplotlib.pyplot as plt # for image handling from PIL import Image # to handle tensors import tensorflow as tf # to load the model from hub import tensorflow_hub as hub # to handle web address for images from six.moves.urllib.request import urlopen Helper Functions: Helper functions are the user defined functions so that we can make use of these functions for repetitive tasks. To read images: def image_read(filename): file = None if (filename.startswith('http')): file = urlopen(filename) else: file = tf.io.gfile.GFile(filename, 'rb') pil_image = Image.open(file) width, height = pil_image.size pil_image = pil_image.crop((0,0, width, height)) pil_image = pil_image.resize((257,257),Image.ANTIALIAS) unscaled_image = np.array(pil_image) image_np = np.expand_dims(unscaled_image.astype(np.float32)/255.,axis=0) return image_np The above function takes a filename as an argument and checks whether the image is being taken from the web or a local machine; based on that, we will have our image in binary format as a variable file. Next, we are formatting the image to handle it; the model takes 257 x 257 with 3 channels. Additionally, we are cropping and scaling (between 0 and 1) the image into squares to avoid distortion. Visualization function: Here we have used matplotlib for the comparison purpose, comparing the result against the original image. def comparison(img_orignal, img_masked, img_filled): plt.figure(figsize=(24,12)) plt.subplot(131) plt.imshow((np.squeeze(img_orignal))) plt.title('Orignal',fontsize=20) plt.axis('off') plt.subplot(132) plt.imshow((np.squeeze(img_masked))) plt.title('masked',fontsize=20) plt.axis('off') plt.subplot(133) plt.imshow((np.squeeze(img_filled))) plt.title('Generated',fontsize = 20) plt.axis('off') plt.show() Load the images: Here well will load sample images from internet but you can also try with your own image but try to avoid image of humans, the model has certain limitation, if you still insist you can try #image_path = 'https://media.gettyimages.com/photos/pune-railway-station-india-picture-id1186355743?s=612x612' image_path = "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSCZlf5lc5tX-0gY-y94pGS0mQdL-D0lCH2OQ&usqp=CAU" #image_path = "https://orthostudio.ca/wp-content/uploads/2016/11/image-3.jpg" # image_path = "https://upload.wikimedia.org/wikipedia/commons/thumb/9/9d/Aegina_sunset.jpg/640px-Aegina_sunset.jpg" image = image_read(image_path) Select the model from Tensorflow-Hub: According to the selected version, there are three versions of models: Half, Quarters and Three Quarters, which take the input image and mask the portion of the image. model_map = {'Boundless Half' : 'https://tfhub.dev/google/boundless/half/1', 'Boundless Quarter' : 'https://tfhub.dev/google/boundless/quarter/1', 'Boundless Three Quarters' : 'https://tfhub.dev/google/boundless/three_quarter/1'} model_name = 'Boundless Quarter' model_handle = model_map[model_name] print('Loading model {} ({})'.format(model_name,model_handle)) model = hub.load(model_handle) Let’s take a look at our sampled images; plt.figure(figsize=(7,7)) plt.imshow(image_read(image_path)[0]) plt.axis('off') Doing the inference: The model has two outputs one is a masked image, and the other is the generated image by the model. result = model.signatures['default'](tf.constant((image))) generated_img = result['default'] masked_img = result['masked_image'] comparison(image_read(image_path),masked_img,generated_img) 1st sample: 2nd sample: 3rd sample: 4th sample: Conclusion: We have seen the Novel generative framework Boundless GAN to extrapolate an image. The model takes the image mask by 25%, 50%, and 75% while inferring it maintains the semantic conditioning to the discriminator of GAN, which makes reconstruction of the masked portion nearly to the original portion and above results is the evidence for this statement. References: Link for Google Colab Tensorflow-Hub for Boundless GANMore about the architecture
Image extrapolation is such a task in computer vision that aims to fill the surrounding region of a sub-image, e.g. completing the object appearing in the image or predicting the unseen view from the scene picture. This task is extremely challenging since the extrapolated image must be realistic with reasonable and meaningful context. Moreover, the extrapolated region should be consistent in structure and texture with the original sub-image.
Image extrapolation is such a task in computer vision that aims to fill the surrounding region of a sub-image, e.g. completing the object appearing in the image or predicting the unseen view from the scene picture. This task is extremely challenging since the extrapolated image must be realistic with reasonable and meaningful context. Moreover, the extrapolated region should be consistent in structure and texture with the original sub-image.
["GANs", "Google", "Guide", "image processing", "Machine Learning"]
["NumPy", "TPU", "AI", "image processing", "Machine Learning", "computer vision", "Colab", "Ray", "Aim", "Matplotlib", "Google", "GANs", "TensorFlow", "R", "Guide"]
["AI", "computer vision", "Aim", "Ray", "TensorFlow", "Colab", "NumPy", "Matplotlib", "TPU", "R"]
Vijaysinh is an enthusiast in machine learning and deep learning. He is skilled in ML algorithms, data manipulation, handling and visualization, model building.