Homomorphic Encryption for Biometric Security: Challenges, Progress and Opportunities

Vishnu Naresh Boddeti

Michigan State University

7th March, 2023


Vulnerabilities in Biometrics

    • Biometric systems suffer from vulnerabilities.

Mitigating Security Vulnerabilities

Biometrics + Encryption

Encrypted Biometrics

  • Traditional solutions need data decryption for computation.
  • Security only during data transmission.

Homomorphic Encryption: The Holy Grail?

  • Cryptographic scheme needs to allow computations directly on the encrypted data.
    • Solution: Homomorphic Encryption
    • Attractive Property: Conjectured to be post-quantum secure for appropriate choice of encryption parameters.

Existing Applications of FHE for Biometric Security

  • Template protection using Homomorphic Encryption:
    • Encrypt database of features.
    • Encrypt query feature.
    • Match score computed directly in encrypted domain.

Prior Work: Template Protection with Homomorphic Encryption

  • Boddeti, "Secure Face Matching Using Fully Homomorphic Encryption," BTAS 2018

  • Bassit et.al, "Multiplication-Free Biometric Recognition for Faster Processing under Encryption," IJCB 2022

CITeR Project

Biometric Template Fusion: Aug 2021-Aug 2022
Biometric Score and Decision Fusion: 2023-Present

Fusion of Biometric Information

"A comprehensive overview of biometric fusion."Information Fusion, 2019"

CITeR Project Focus: Template Fusion
"Deep learning approach for multimodal biometric recognition system based on fusion of iris, face, and finger vein traits." Sensors, 2020

Privacy Attacks from Features

Attacks on face features
"Assessing Privacy Risks from Feature Vector Reconstruction Attacks," arXiv:2202.05760

Face reconstruction from template
"On the reconstruction of face images from deep face templates," TPAMI, 2018


Homomorphically Encrypted Fusion of Biometric Templates

HEFT: Overview

HEFT: Concatenation

Homomorphic Concatenation

HEFT: Linear Projection

Linear Projection


Linear Projection Comparison

Computational Complexity

  • Hybrid
    • Pros: Low memory and runtime overhead
    • Cons: Scales linearly with number of samples

HEFT: Feature Normalization

$\ell_2$-Normalization of Vector

$\hat{\mathbf{u}} = \frac{\mathbf{u}}{\|\mathbf{u}\|_2} \quad \rightarrow \quad$ division$\dagger$


$\|\mathbf{u}\|_2 = \sqrt{\sum_{i=1}^d u_i^2} \quad \rightarrow \quad$ square-root$\dagger$

  • $\dagger$: problematic operations for FHE

Inverse Square Root: Polynomial Approximation

$$\frac{1}{\sqrt{x}} = \sum_{i=1}^6 a_i x^i$$

FHE-Aware Learning

Account for the limitations of FHE to improve performance

    • FHE is limited to specific operations on encrypted data.

    • Normalization is not directly computable - need to approximate.

    • Approximation is a source of error and hence a loss of matching performance

    • We incorporate approximate normalization into our training of the projection matrix to recover performance

Loss Function

Main Idea: FHE-Aware Learning
  • $$Loss = \lambda \underbrace{\frac{\sum_M d(\mathbf{c}_i, \mathbf{c}_j)}{|M|}}_{ \color{orange}{Pull} } + (1-\lambda)\underbrace{\frac{\sum_{V}[m + d(\mathbf{c}_i, \mathbf{c}_j) - d(\mathbf{c}_i, \mathbf{c}_k)]_{+}}{|V|}}_{ \color{orange}{Push} }$$
  • where $$d(\mathbf{c}_i, \mathbf{c}_j) = 1-P\underbrace{f(\mathbf{c}_i)}_{ \color{cyan}{approximation} } \cdot P\underbrace{f(\mathbf{c}_j)}_{ \color{cyan}{approximation} }$$ $f(\cdot)$ approximates the inverse norm of a vector.

Numerical Evaluation

Experimental Setup

Cross-Posed Labelled Faces in the Wild

    • Synthetic fusion dataset by randomly pairing classes.
    • 10,760 samples over 188 classes.

Fusion Improves Performance, Reduces Dimensionality

  • Fusion improves performance:
    • Face by 11.07%
    • Voice by 9.58%
  • Dimensionality Reduction: $512D \rightarrow 32D$ (16$\times$ compression)

Comparison of Normalization Methods

Computational Complexity

    • Projection is costliest operation

What Next for Biometric Encryption?

Opportunities for Biometric Encryption

Going beyond template fusion
Ongoing CITeR Project (Jan 2023-Present)

Opportunities for Biometric Encryption

Secure federated learning
Federated learning for face recognition

    • Aggarwal et al. "FedFace: Collaborative Learning of Face Recognition Model," IJCB 2021
    • Meng et al. "Improving Federated Learning Face Recognition via Privacy-Agnostic Clusters," ICLR 2022

    • Preliminary Work
      • Yonetani, Boddeti, Kitani, Sato "Privacy-Preserving Visual Learning Using Doubly Permuted Homomorphic Encryption," ICCV 2017

A Note on Security vs Privacy

  • Security and privacy are very often conflated with each other.
    • Different but related concepts.
    • Homomorphic encryption: controls access to private information.
    • Differential Privacy: allows analysis + controls information.

  • Postulates:
    • There is no privacy without security.
    • Homomorphic encryption is an ideal tool for enhancing privacy but it is not a privacy technique in and of itself.
Ideal solution: Differential privacy + Homomorphic Encryption

Summary: CITeR Projects on Biometric Encryption

Biometric System Threat Model

Many avenues for leveraging homomorphic encryption to enhance biometric security and privacy.