Deepfake Detection

Source: Deep Learning on Medium

LossIsNan, Detector group

Focus: Image / Video

1. FaceForensics++: Learning to Detect Manipulated Facial Images5

The paper proposes an automated benchmark based on several current facial manipulation methods at random compression level and size.Though the analysis, they believe they can detect facial manipulation and exceeds human observers to a large extent. They used a modified XceptionNet(CNN) reaching the best performance, and the accuracy on GAN-generated LQ images need to be improved.

2. MesoNet: A Compact Facial Video Forgery Detection Network6

This paper states that current image detection cannot be used in video due to the data degradation and compression. So focusing on deep fakes and Face2Face detection, they used a CNN to detect the forgery video using inception block. And they found that detailed eyes, nose and mouth areas are critical in detection.

3. DeepFake Video Detection Using Recurrent Neural Networks3

The paper first analyzed the generation process of a DeepFake video for discovering the anomalies which can be exploited. Then they proposed a temporal-aware system to automatically detect the DeepFake video. The system uses a CNN to extract frame-level features which are then used to train a recurrent neural network for temporal sequence analysis to detect manipulated videos. And the proposed model achieved over 96% accuracy on HOHA dataset with additional customized data.

4. Recurrent Convolutional Strategies for Face Manipulation Detection in Videos4

The paper explores the strategies of combining recurrent-convolutional neural network and face alignment approach for detecting manipulated videos evaluated on FaceForennsiscs++. They adopted two face preprocessing methods including explicit alignment using facial landmarks and implicit alignment using spatial transformer network. For the detection part, they have tried to use a single recurrent network on top of the final features from the backbone net and also learn multiple recurrent networks at different level of the hierarchy of the backbone net. They found landmark-based face alignment with bidirectional-recurrent-densenet have good performance.

5. Deep Fake Image Detection based on Pairwise Learning 5

Given that traditional image forgery detection methods approach performs not so well on popular GAN models today, the authors propose common fake feature network (CFFN). The model can be divided into three stages:

1) Various GANs generate <real_img, fake_img> pairs;

2) Representation Learning using adapted CNN network with dense block and Contrastive loss; 3) Classification Learning through cross entropy loss.

6. Capsule-forensics: using capsule networks to detect forged images and videos 6

This paper divide image forgery detection process into five stages, despite preprocessing, training example generation and postprocessing, the two core steps are:

1) Representation learning using existed method VGG-19;

2) Classification Learning based on Capsule Network and cross entropy loss.

7. Protecting World Leaders Against Deep Fakes

Bappy J H, Simons C, Nataraj L, et al.7 describe a forensic technique that models facial expressions and movements that typify an individual’s speaking pattern. They use the open-source facial behavior analysis toolkit OpenFace2 to extract facial and head movements in a video. As a result, their model shows state-of-the-art performance (an average accuracy of 0.95).

8. Hybrid LSTM and Encoder–Decoder Architecture for Detection of Image Forgeries

Agarwal S, Farid H, Gu Y, et al.8 propose a high-confidence manipulation localization architecture that utilizes resampling features, long short-term memory (LSTM) cells, and an encoder–decoder network to segment out manipulated regions from non-manipulated ones. Also, a large image splicing dataset is introduced to guide the training process.


In general, current detection method is based on CNN and combined some feature extraction methods, such as face alignment and steganalysis. In terms of video detection task, introducing RNN is common to take time context into concern. But the intuition of these methods divided into several directions, such as detecting the flaw of Deepfake generators like unnatural parts in fake images. Or instead of focusing on DeepFake Generator, they want to find the feature in all the modified image, and make a detector that could generalize on all instances.


1.Sabir, Ekraam, et al. “Recurrent Convolutional Strategies for Face Manipulation Detection in Videos.” Interfaces (GUI) 3 (2019): 1.

2. Güera, David, and Edward J. Delp. “Deepfake video detection using recurrent neural networks.” 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). IEEE, 2018.

3.Rössler, Andreas, et al. “Faceforensics: A large-scale video dataset for forgery detection in human faces.” arXiv preprint arXiv:1803.09179 (2018).

4.Afchar, Darius, et al. “Mesonet: a compact facial video forgery detection network.” 2018 IEEE International Workshop on Information Forensics and Security (WIFS). IEEE, 2018.

5. Hsu, C. C., Zhuang, Y. X., and Lee, C. Y. (2019). Deep fake image detection based on pairwise learning. Preprints, 2019050013.

6. Nguyen, H. H., Yamagishi, J., and Echizen, I. (2019, May). Capsule-forensics: Using capsule networks to detect forged images and videos. In 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 2307–2311). IEEE.

7.Bappy J H, Simons C, Nataraj L, et al. Hybrid LSTM and Encoder–Decoder Architecture for Detection of Image Forgeries[J]. IEEE Transactions on Image Processing, 2019, 28(7): 3286–3300.

8.Agarwal S, Farid H, Gu Y, et al. Protecting World Leaders Against Deep Fakes[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2019: 38–45.