RestoreIt- Restore your old and damaged photos

Source: Deep Learning on Medium

RestoreIt- Restore your old and damaged photos

Photographs have always been a part of our life whether it be the old times where the hard copies were preserved or the modern era where everything is digital. We all love to click and preserve pictures but with time pictures deteriorate. This deterioration can be of many forms like with time the hard copy of photos loses color and even gain a yellow tinge. In some other cases when folded and kept for a long time the photos lose the color along the lines of the fold, this is called fold damage. The type of damage that is common to both hard, as well as digital copies of photographs, is the image corruption damage. In hard copies, some of the parts of the images can get torn off and thus corrupt the image and in cases of digital copies there are a number of ways in which it can get corrupted, it can get corrupted during sharing or upload or any such process during which transfer of information bits (i.e. pixels) takes place. Using various technologies namely, Computer Vision (OpenCV) and Deep Learning, all the damages mentioned above can be solved and hence the photo can be restored. OpenCV offers a lot of services for manual restoration of photographs whereas using Deep Learning we can automate the process. In the process of restoration, there is one model namely the Deep Convolutional Generative Adversarial Network (DCGAN) that has multipurpose usage, it can be used for re-colorization, image in-painting, and even image re-construction. This project makes use of OpenCV and Generative Adversarial Networks during the process of restoration.

It is said that a photograph can say a thousand words. It is even said to capture memories and valuable moments. This might be the reason why they are considered such treasured possessions. Ever since the invention of the camera, photographs have become an integral part of our lives. The fact that people can relive certain memories if they have a certain moment captured has always been a very alluring factor for people towards photographs. We all have albums filled with photographs of our childhood, our parents’ childhood and many such more memories. One thing that must be common with almost every physical copy of the photograph is the yellowish tinge that is accumulated over time. This is just one of many types of abrasion that a photo suffers. Some other types of disfigurement that a photo suffers is the fold damage, silverfish damage, spoil damage, etc. The following is the detailed discussion of the types of damages and their proposed solution

· Yellow tint- Color dyes in film fade and change over time, usually becoming more yellow and also the print paper of older times wasn’t nearly as good as the ones right now. That is primarily the reason that old photographs generally gained a yellowish tint over time as seen in Figure 2. The solution that is proposed for this re-colorization of image. The image re-colorization can be achieved in many ways. Primarily, this can be achieved by using OpenCV and Deep Learning. This technique was discussed by Richard Zhang in his Colourful Image Colourization paper. Zhang et al. decided to attack the problem of image colorization by using Convolutional Neural Networks to “hallucinate” what an input grayscale image would look like when colorized.

Yellow Tint damaged and after restoration

· Silverfish damage — Silverfish are bugs that are wicked fast and incredibly ugly. The National Pesticide Information Center describes silverfish as “wingless insects with flat, carrot-shaped bodies.” Silverfish damage is one of the most common types of photodamage that we see at the studio. Most people don’t know that they have silverfish issues because they assume the photo has faded or bits and pieces of the photo have fallen off over time. The bugs eat the emulsion and leave holes in the image. The solution that is proposed for this damage is Image Inpainting.

Silverfish damage closeup

· Fold damage- Fold damage is falling off of bits of images along the lines of the fold. This happens in cases of photographs which have stayed folded for a long time. These are white lines that appear due to the fold.

Fold damaged photo and restored

How will I do it?

The system that we propose a generative adversarial network which fills in the missing part of an image using inpainting and re-colorize the image using deep learning technique. The Generative Adversarial Network are generally used for generating parts of images. If a deep convolutional neural network is convolved with a GAN we get deep convolutional generative adversarial neural networks and that can be used for multiple purposes since it has characteristics of both convolutional neural networks and generative adversarial networks.

First of all, a DCGAN is constructed. A DCGAN mainly composes of convolution layers without max pooling or fully connected layers. It uses convolutional stride and transposed convolution for the downsampling and the upsampling. The steps followed for developing a DCGAN is:

· Replace all max pooling with convolutional stride

· Use transposed convolution for upsampling.

· Eliminate connected layers.

· Use Batch normalization except the output layer for the generator and the input layer of the discriminator.

· Use ReLU in the generator except for the output which uses tanh.

· Use LeakyReLU in the discriminator.

The tuning tips as stated in one of the research papers are as follows:

All models were trained with mini-batch stochastic gradient descent (SGD) with a mini-batch size of 128. All weights were initialized from a zero-centered Normal distribution with standard deviation 0.02. In the LeakyReLU, the slope of the leak was set to 0.2 in all models. While previous GAN work has used momentum to accelerate training, we used Adam optimizer with tuned hyperparameters. We found the suggested learning rate of 0.001, to be too high, using 0.0002 instead. Additionally, we found leaving the momentum term β1 at the suggested value of 0.9 resulted in training oscillation and instability while reducing it to 0.5 helped stabilize training.

For training, I fed random values to our generator, which will learn to create outputs out of this noise. We also take care that neither the generator nor the discriminator becomes too strong by balancing their losses — otherwise, this would inhibit the learning of either part and could even stop the network from learning anything at all. The DCGAN after development, when trained on a dataset, generates the same output. Thus, when trained on LFW (Labeled Faces in the Wild) dataset it started giving average results of faces. OpenCV also provides an inbuilt function for painting images so making use of that the re-colorization of the process would take place. That is the basic gist of the algorithm that goes into the program.

Future Scope

One thing that was incomplete during the course of the project was the proprietary model that could detect the type of damage that a particular image has suffered. Also, the amalgamation of all the scripts into one without compromising the simplicity of using the service is one of the major hurdles that still remains to be surpassed.