Abstract

Face recognition and analysis systems are widely used for identification, social understanding, behavior analysis, item recommendation, and many other applications. Face embeddings are extracted from face images using deep neural networks and are stored by these systems. These embeddings are susceptible to data leaks, which malicious entities can exploit to extract identifiable information and soft biometrics, gaining unauthorized access and utilizing them for purposes not originally intended. It has been shown that the embeddings can be used to reconstruct the face image. Template protection methods have been proposed to prevent loss of identity. We observe that such template protection methods may not protect soft biometrics. To alleviate such leakage, we propose a novel method to include additional layers of security for the face embeddings through Fully Homomorphic Encryption (FHE) beyond an existing template protection scheme named PolyProtect. We also show that the embeddings can be compressed and encrypted using FHE and transformed into a secure PolyProtect template using polynomial transformation for additional protection. FHE-based methods allow computations on encrypted data, enabling the processing of templates in encrypted form for face recognition and analysis. However, the results can be decrypted only with access to the private key, thus preventing any privacy leakage. We demonstrate the efficacy of the proposed approach through extensive experiments on multiple datasets. Our proposed approach ensures irreversibility and unlinkability, effectively preventing the leakage of soft biometric attributes from face embeddings without compromising recognition accuracy.