Deepfake detection challenge from R

Introduction#

Working with video datasets, particularly with respect to detection of AI-based fake objects, is very challenging due to proper frame selection and face detection. To approach this challenge from R, one can make use of capabilities offered by OpenCV, magick, and keras.

Our approach consists of the following consequent steps:

read all the videos
capture and extract images from the videos
detect faces from the extracted images
crop the faces
build an image classification model with Keras

Let’s quickly introduce the non-deep-learning libraries we’re using. OpenCV is a computer vision library that includes:

Facial recognition technology
Motion tracking
Augmented reality
and more.

On the other hand, magick is the open-source image-processing library that will help to read and extract useful features from video datasets:

Read video files
Extract images per second from the video
Crop the faces from the images

Before we go into a detailed explanation, readers should know that there is no need to copy-paste code chunks. Because at the end of the post one can find a link to Google Colab with GPU acceleration. This kernel allows everyone to run and reproduce the same results.

Data exploration#

The dataset that we are going to analyze is provided by AWS, Facebook, Microsoft, the Partnership on AI’s Media Integrity Steering Committee, and various academics.

It contains both real and AI-generated fake videos. The total size is over 470 GB. However, the sample 4 GB dataset is separately available.

Frame extraction#

The videos in the folders are in the format of mp4 and have various lengths. Our task is to determine the number of images to capture per second of a video. We usually took 1-3 fps for every video.

Note: Set fps to NULL if you want to extract all frames.

1
2
3


video = magick::image_read_video("aagfhgtpmv.mp4",fps = 2)
vid_1 = video[[1]]
vid_1 = magick::image_read(vid_1) %>% image_resize('1000x1000')

We saw just the first frame. What about the rest of them?

Looking at the gif one can observe that some fakes are very easy to differentiate, but a small fraction looks pretty realistic. This is another challenge during data preparation.

Face detection#

At first, face locations need to be determined via bounding boxes, using OpenCV. Then, magick is used to automatically extract them from all images.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# get face location and calculate bounding box
library(opencv)
unconf <- ocv_read('frame_1.jpg')
faces <- ocv_face(unconf)
facemask <- ocv_facemask(unconf)
df = attr(facemask, 'faces')
rectX = (df$x - df$radius) 
rectY = (df$y - df$radius)
x = (df$x + df$radius) 
y = (df$y + df$radius)

# draw with red dashed line the box
imh  = image_draw(image_read('frame_1.jpg'))
rect(rectX, rectY, x, y, border = "red", 
     lty = "dashed", lwd = 2)
dev.off()

Face extraction#

If face locations are found, then it is very easy to extract them all.

1
2
3


edited = image_crop(imh, "49x49+66+34")
edited = image_crop(imh, paste(x-rectX+1,'x',x-rectX+1,'+',rectX, '+',rectY,sep = ''))
edited

Deep learning model#

After dataset preparation, it is time to build a deep learning model with Keras. We can quickly place all the images into folders and, using image generators, feed faces to a pre-trained Keras model.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51


train_dir = 'fakes_reals'
width = 150L
height = 150L
epochs = 10

train_datagen = image_data_generator(
  rescale = 1/255,
  rotation_range = 40,
  width_shift_range = 0.2,
  height_shift_range = 0.2,
  shear_range = 0.2,
  zoom_range = 0.2,
  horizontal_flip = TRUE,
  fill_mode = "nearest",
  validation_split=0.2
)


train_generator <- flow_images_from_directory(
  train_dir,                  
  train_datagen,             
  target_size = c(width,height), 
  batch_size = 10,
  class_mode = "binary"
)

# Build the model ---------------------------------------------------------

conv_base <- application_vgg16(
  weights = "imagenet",
  include_top = FALSE,
  input_shape = c(width, height, 3)
)

model <- keras_model_sequential() %>% 
  conv_base %>% 
  layer_flatten() %>% 
  layer_dense(units = 256, activation = "relu") %>% 
  layer_dense(units = 1, activation = "sigmoid")

model %>% compile(
  loss = "binary_crossentropy",
  optimizer = optimizer_rmsprop(lr = 2e-5),
  metrics = c("accuracy")
)

history <- model %>% fit_generator(
  train_generator,
  steps_per_epoch = ceiling(train_generator$samples/train_generator$batch_size),
  epochs = 10
)