Image Forgery Detection

Original article was published by Kishan on Deep Learning on Medium


plt.figure(figsize = (10,10))k = dataset[‘class_label’].value_counts()k.plot(kind = ‘bar’)plt.xlabel(“Class of images”)plt.ylabel(“Number of Images”)plt.title(“Images belonging to each class”)

We can conclude that the number of pristine images in the dataset is more then the number of fake images. Hence the images have different number of channels, so we need to convert every images in the same number of channels that we’ll do it using tensorflow data pipelines.

Implementing tensorflow data pipeline:

Now we are converting each images into same number of channels. Also we are doing a train test split into 80:20

def get_label(file_path):
parts = tf.strings.split(file_path, os.path.sep)
one_hot = parts[-2] == class_names
return tf.argmax(tf.cast(one_hot, tf.int32))
def decode_img(img):
img = tf.image.decode_jpeg(img, channels=3)
return tf.image.resize(img, [img_height, img_width])
def process_path(file_path):label = get_label(file_path)img = tf.io.read_file(file_path)img = decode_img(img)return img, label

Now we are going to predict the accuracy using 3 different models:

  1. Convolutional Model
  2. ResNet50
  3. VGG16
model=tf.keras.Sequential(
[ tf.keras.layers.InputLayer(input_shape=(128, 128, 3)),
tf.keras.layers.Conv2D(filters=32, kernel_size=3, strides= (2, 2), activation=’relu’), tf.keras.layers.Conv2D(filters=64, kernel_size=3, strides=(2, 2), activation=’relu’), tf.keras.layers.Dropout(0.5) tf.keras.layers.Flatten(), tf.keras.layers.Dense(6) ])#Compile the modeloptimizer = Adam()model.compile(optimizer=optimizer, loss=’binary_crossentropy’,metrics=[‘accuracy’])#Fitting the modelhistory = model.fit(train_ds,validation_data=val_ds,epochs=5,steps_per_epoch = 10,callbacks=[tensorboard_callback,mcp_save])

Using the CNN model, we are getting Precision score of 0.63, Recall score of 0.69 and F1 score: 0.60. In this CNN model. we have used CONV2d with dropout value of 0.5. After this we’ve proceed with Resnet and vgg16 models.

model=tf.keras.Sequential([tf.keras.layers.InputLayer(input_shape=(128, 128, 3)),tf.keras.applications.VGG16(filters=32, kernel_size=3, strides=(2, 2), activation=’relu’),tf.keras.layers.Dropout(0.5),tf.keras.layers.Flatten(),tf.keras.layers.Dense(6)])#Compile the modeloptimizer = Adam()model.compile(optimizer=optimizer, loss=’binary_crossentropy’,metrics=[‘accuracy’])#Fitting the modelhistory = model.fit(train_ds,validation_data=val_ds,epochs=5,steps_per_epoch = 10,callbacks=[tensorboard_callback,mcp_save])

Using this vgg16 model, we are getting Precision score of 0.75, Recall score of 0.78 and F1 score of 0.73 . Here we have used vgg16 models with 32 filters and kernel size as 3. Also, we’ve used relu as the activation unit. Now we are trying with resnet50 model.

model=tf.keras.Sequential([tf.keras.layers.InputLayer(input_shape=(128, 128, 3)),tf.keras.applications.ResNet50(filters=32, kernel_size=3, strides=(2, 2), activation=’relu’),tf.keras.layers.Dropout(0.5),tf.keras.layers.Flatten(),tf.keras.layers.Dense(6)])#Compile the modeloptimizer = Adam()model.compile(optimizer=optimizer, loss=’binary_crossentropy’,metrics=[‘accuracy’])#Fitting the modelhistory = model.fit(train_ds,validation_data=val_ds,epochs=5,steps_per_epoch = 10,callbacks=[tensorboard_callback,mcp_save])

Using this resnet model, we are getting Precision score of 0.71, Recall score of 0.79 and F1 score of 0.70 . Here we have used vgg16 models with 32 filters and kernel size as 3. Also, we’ve used relu as the activation unit. Now we are trying with resnet50 model.

Now, we are writing a function to predict whether the image is fake or not.

def func(image_directory):img1 = image.load_img(image_directory,target_size=(128, 128))img = image.img_to_array(img1)img = img/255img = np.expand_dims(img, axis=0)prediction = model.predict(img, batch_size=None,steps=1)if(prediction[:,:]>0.5):print(“Fake Image”)else:print(“Pristine Image”)x=Image.open(image_directory)plt.imshow(x)

Importing images from fake images folder and predicting their class label

func(‘/content/drive/My Drive/training/fake/0908dafde12041540b70d688315df6e9.png’)
func(‘/content/drive/My Drive/training/fake/010543abfbd0db1e9aa1b24604336e0c.png’)

Importing images from Pristine images folder and predicting their class label

func(‘/content/drive/My Drive/training/Real/00d56bffe2d757a4e4625cc2d0bd0de9.png’)
func(‘/content/drive/My Drive/traning/Real/01e6ccb1d207ce31033251643aa55f9d.png’)

Detecting portion of forged images:

The RGB stream models visual tampering artifacts, such as unusually high contrast along object edges. The noise stream first obtains the noise feature map by passing input RGB image through an SRM filter layer, and leverages the noise features to provide additional evidence for manipulation classification. A bilinear pooling layer after it enables the network to combine the spatial co-occurrence features from the two streams. Finally, passing the results through a fully connected layer, the network produces required mask output.

import numpy as npq = [4.0, 12.0, 2.0]filter1 = [[0, 0, 0, 0, 0],[0, -1, 2, -1, 0],[0, 2, -4, 2, 0],[0, -1, 2, -1, 0],[0, 0, 0, 0, 0]]filter2 = [[-1, 2, -2, 2, -1],[2, -6, 8, -6, 2],[-2, 8, -12, 8, -2],[2, -6, 8, -6, 2],[-1, 2, -2, 2, -1]]filter3 = [[0, 0, 0, 0, 0],[0, 0, 0, 0, 0],[0, 1, -2, 1, 0],[0, 0, 0, 0, 0],[0, 0, 0, 0, 0]]filter1 = np.asarray(filter1, dtype=float) / q[0]filter2 = np.asarray(filter2, dtype=float) / q[1]filter3 = np.asarray(filter3, dtype=float) / q[2]filters = filter1+filter2+filter3for x in fakes:image = imread(fake_path+x)processed_image = cv2.filter2D(image,-1,filters)plt.imsave(‘processed_images/’+x,processed_image)input_img = Input((512, 512, 3), name='img1')n_filters=16batchnorm=Truedropout=0.5# contracting pathc1 = conv2d_block(input_img, n_filters=n_filters*1, kernel_size=3, batchnorm=batchnorm)p1 = MaxPooling2D((2, 2)) (c1)p1 = Dropout(dropout*0.5)(p1)c2 = conv2d_block(p1, n_filters=n_filters*2, kernel_size=3, batchnorm=batchnorm)p2 = MaxPooling2D((2, 2)) (c2)p2 = Dropout(dropout)(p2)c3 = conv2d_block(p2, n_filters=n_filters*4, kernel_size=3, batchnorm=batchnorm)p3 = MaxPooling2D((2, 2)) (c3)p3 = Dropout(dropout)(p3)c4 = conv2d_block(p3, n_filters=n_filters*8, kernel_size=3, batchnorm=batchnorm)p4 = MaxPooling2D(pool_size=(2, 2)) (c4)p4 = Dropout(dropout)(p4)c5 = conv2d_block(p4, n_filters=n_filters*16, kernel_size=3, batchnorm=batchnorm)#Expanding pathu6 = Conv2DTranspose(n_filters*8, (3, 3), strides=(2, 2), padding='same') (c5)#skip_connectionsu6 = concatenate([u6, c4])u6 = Dropout(dropout)(u6)c6 = conv2d_block(u6, n_filters=n_filters*8, kernel_size=3, batchnorm=batchnorm)u7 = Conv2DTranspose(n_filters*4, (3, 3), strides=(2, 2), padding='same') (c6)u7 = concatenate([u7, c3])u7 = Dropout(dropout)(u7)c7 = conv2d_block(u7, n_filters=n_filters*4, kernel_size=3, batchnorm=batchnorm)u8 = Conv2DTranspose(n_filters*2, (3, 3), strides=(2, 2), padding='same') (c7)u8 = concatenate([u8, c2])u8 = Dropout(dropout)(u8)c8 = conv2d_block(u8, n_filters=n_filters*2, kernel_size=3, batchnorm=batchnorm)u9 = Conv2DTranspose(n_filters*1, (3, 3), strides=(2, 2), padding='same') (c8)u9 = concatenate([u9, c1], axis=3)u9 = Dropout(dropout)(u9)c9 = conv2d_block(u9, n_filters=n_filters*1, kernel_size=3, batchnorm=batchnorm)output = Conv2D(3, (1, 1), activation='sigmoid') (c9)#model1 = Model(inputs=[input_img], outputs=[outputs])input_img_filter = Input((512, 512, 3), name='img2')n_filters=16batchnorm=Truedropout=0.5c1 = conv2d_block(input_img_filter, n_filters=n_filters*1, kernel_size=3, batchnorm=batchnorm)p1 = MaxPooling2D((2, 2)) (c1)p1 = Dropout(dropout*0.5)(p1)c2 = conv2d_block(p1, n_filters=n_filters*2, kernel_size=3, batchnorm=batchnorm)p2 = MaxPooling2D((2, 2)) (c2)p2 = Dropout(dropout)(p2)c3 = conv2d_block(p2, n_filters=n_filters*4, kernel_size=3, batchnorm=batchnorm)p3 = MaxPooling2D((2, 2)) (c3)p3 = Dropout(dropout)(p3)c4 = conv2d_block(p3, n_filters=n_filters*8, kernel_size=3, batchnorm=batchnorm)p4 = MaxPooling2D(pool_size=(2, 2)) (c4)p4 = Dropout(dropout)(p4)c5 = conv2d_block(p4, n_filters=n_filters*16, kernel_size=3, batchnorm=batchnorm)#Expanding pathu6 = Conv2DTranspose(n_filters*8, (3, 3), strides=(2, 2), padding='same') (c5)#skip_connectionsu6 = concatenate([u6, c4])u6 = Dropout(dropout)(u6)c6 = conv2d_block(u6, n_filters=n_filters*8, kernel_size=3, batchnorm=batchnorm)u7 = Conv2DTranspose(n_filters*4, (3, 3), strides=(2, 2), padding='same') (c6)u7 = concatenate([u7, c3])u7 = Dropout(dropout)(u7)c7 = conv2d_block(u7, n_filters=n_filters*4, kernel_size=3, batchnorm=batchnorm)u8 = Conv2DTranspose(n_filters*2, (3, 3), strides=(2, 2), padding='same') (c7)u8 = concatenate([u8, c2])u8 = Dropout(dropout)(u8)c8 = conv2d_block(u8, n_filters=n_filters*2, kernel_size=3, batchnorm=batchnorm)u9 = Conv2DTranspose(n_filters*1, (3, 3), strides=(2, 2), padding='same') (c8)u9 = concatenate([u9, c1], axis=3)u9 = Dropout(dropout)(u9)c9 = conv2d_block(u9, n_filters=n_filters*1, kernel_size=3, batchnorm=batchnorm)output_filter = Conv2D(3, (1, 1), activation='sigmoid') (c9)combined = concatenate([output, output_filter])outputs = Conv2D(1, (1, 1), activation='sigmoid') (combined)model = Model(inputs=[input_img,input_img_filter], outputs=[outputs])def func(directory):pred = np.squeeze(model.predict[directory])plt.imsave('pred_mask.png',pred)im_gray = cv2.imread('pred_mask.png', cv2.IMREAD_GRAYSCALE)(thresh, im_bw) = cv2.threshold(im_gray, 220, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)fig = plt.figure(figsize=(20,10))ax1 = fig.add_subplot(331)ax2 = fig.add_subplot(332)ax3 = fig.add_subplot(333)ax1.set_title("actual_image")ax2.set_title("actual_mask")ax3.set_title("predicted_mask")ax1.imshow(directory)ax2.imshow(np.squeeze(directory))ax3.imshow(np.squeeze(im_bw))plt.plot(history.history['metric'])plt.plot(history.history['val_metric'])plt.title('model f-score')plt.ylabel('metric')plt.xlabel('epoch')plt.legend(['train', 'validation'], loc='bottom right')plt.show()func(‘/content/drive/My Drive/traning/fake/5872331dd0b1c71e4541564654ghv58556.png’)