The Doctoral School in Sciences and Engineering is happy to invite you to MEJRI Nesryne’s defence entitled
Unsupervised Anomaly Detection for Type-Agnostic and Cross-Domain Deepfake Detection
Supervisor: Assist. Prof Djamila AOUADA
Deepfakes are visual media created using deep learning models to partially manipulate or fully synthesize human faces. They cover a variety of forgery methods, broadly categorized into four types: face swaps, facial reenactments, facial attribute manipulations, and synthetic face generation.
Their increasing realism in recent years has raised concerns regarding their misuse, thereby creating an urgent need for reliable deepfake detection techniques. Hence, various deepfake detection methods have been proposed. They are predominantly based on Deep Neural Networks trained in a supervised manner. As a result, these methods are often prone to generalization issues, hindering their applicability in real-world settings. Although promising, their performance degrades considerably when encountering unseen images/videos that differ significantly from the training data. This drop can be attributed to two different sources of variation in the visual data: forgery diversity and environmental variability.
Herein, this thesis aims to enhance the robustness of deep-fake detectors to these two complementary sources of variation by reformulating deepfake detection as an unsupervised anomaly detection (UAD) task. This formulation eliminates the need for annotated fake data and reduces the dependence on specific types of forgeries.
In the first part of the manuscript, we focus on improving the generalization of deepfake detectors to unseen forgeries. While some research works have attempted to improve this aspect, they have mainly targeted blending-based artifacts typically induced by several face-swaps generation techniques. As a consequence, they usually show degraded performance when dealing with non-blending-based deepfakes, including diffusion-based face-swaps and other deepfake types characterized by inherently different artifacts.
To address this issue, we propose a self-supervised framework that allows extracting features from different artifact-prone regions. This self-supervision mechanism is then coupled with a one-class classifier that models the feature distribution of real data only, thereby avoiding overfitting specific types of deepfake artifacts. This idea has then been extended to temporal deepfake localization, where the goal is to spot specific frames in an unsegmented stream that have undergone deepfake manipulations. Experiments performed on several benchmarks have demonstrated the improved generalization capabilities of the proposed methods.
In the second part of this work, we also address the lack of robustness to environmental changes. Deep learning models, including deepfake detectors, are often challenged by the domain shift issue, a phenomenon frequently caused by uncontrolled variations in lighting, resolution, or background. These variations are typically unrelated to forgeries and can further compromise the detection performance of deepfake detectors. In the literature, this problem has often been mitigated by adopting an Unsupervised Domain Adaptation (UDA) approach. Nevertheless, existing UDA techniques are primarily designed for binary and multi-class classification, while being incompatible with the core proposed paradigm for deepfake detection, namely, unsupervised anomaly detection. Indeed, UDA for UDA is an ill-posed problem due to what we call the two-fold unsupervised curse. To overcome this issue, we propose to take advantage of the scarcity of anomalies and rely on a clustering technique to isolate a predominant cluster to be used for the alignment step. This pioneering work has been validated on anomaly detection benchmarks, showing great potential for enhancing the generalization of deepfake detectors.