Homomorphically Encrypted Biometric Template Fusion

Vishnu Boddeti

Michigan State University

8th March, 2023

Norwegian Biometrics Laboratory Workshop

Vulnerabilities in Biometrics

    • Biometric systems suffer from vulnerabilities.

Mitigating Security Vulnerabilities

Biometrics + Encryption

Encrypted Biometrics

  • Traditional solutions need data decryption for computation.
  • Security only during data transmission.

Homomorphic Encryption: The Holy Grail?

  • Cryptographic scheme needs to allow computations directly on the encrypted data.
    • Solution: Homomorphic Encryption
    • Attractive Property: Conjectured to be post-quantum secure for appropriate choice of encryption parameters.

Template Security with Homomorphic Encryption

Biometric Features

  • Learned Features:

Information Leakage from Representations

Attacks on Face templates
"Assessing Privacy Risks from Feature Vector Reconstruction Attacks," arXiv:2202.05760

Face reconstruction from template
"On the reconstruction of face images from deep face templates," TPAMI, 2018

Existing Applications of FHE for Biometric Security

  • Template protection using Homomorphic Encryption:
    • Encrypt database of features.
    • Encrypt query feature.
    • Match score computed directly in encrypted domain.

Prior Work: Template Protection with Homomorphic Encryption

  • Boddeti, "Secure Face Matching Using Fully Homomorphic Encryption," BTAS 2018

  • Bassit et.al, "Multiplication-Free Biometric Recognition for Faster Processing under Encryption," IJCB 2022

Secure Biometric Template Fusion

Fusion of Biometric Information

"A comprehensive overview of biometric fusion."Information Fusion, 2019"
"Deep learning approach for multimodal biometric recognition system based on fusion of iris, face, and finger vein traits." Sensors, 2020

HEFT: Overview

  • Sperling et al., "HEFT: Homomorphically Encrypted Fusion of Biometric Templates," IJCB 2022 (Best Student Paper Award)

HEFT: Concatenation

Homomorphic Concatenation

HEFT: Linear Projection

Linear Projection

$z = P \times \begin{bmatrix}\mathbf{x}_1 \\ \mathbf{x}_2\end{bmatrix}$

Linear Projection Comparison

Computational Complexity

  • Hybrid
    • Pros: Low memory and runtime overhead
    • Cons: Scales linearly with number of samples

HEFT: Feature Normalization

$\ell_2$-Normalization of Vector

$\hat{\mathbf{u}} = \frac{\mathbf{u}}{\|\mathbf{u}\|_2} \quad \rightarrow \quad$ division$\dagger$


$\|\mathbf{u}\|_2 = \sqrt{\sum_{i=1}^d u_i^2} \quad \rightarrow \quad$ square-root$\dagger$

  • $\dagger$: problematic operations for FHE

Inverse Square Root: Polynomial Approximation

$$\frac{1}{\sqrt{x}} = \sum_{i=1}^6 a_i x^i$$

FHE-Aware Learning

Account for the limitations of FHE to improve performance

    • FHE is limited to specific operations on encrypted data.

    • Normalization is not directly computable - need to approximate.

    • Approximation is a source of error and hence a loss of matching performance

    • We incorporate approximate normalization into our training of the projection matrix to recover performance

Loss Function

Main Idea: FHE-Aware Learning
  • $$Loss = \lambda \underbrace{\frac{\sum_M d(\mathbf{c}_i, \mathbf{c}_j)}{|M|}}_{ \color{orange}{Pull} } + (1-\lambda)\underbrace{\frac{\sum_{V}[m + d(\mathbf{c}_i, \mathbf{c}_j) - d(\mathbf{c}_i, \mathbf{c}_k)]_{+}}{|V|}}_{ \color{orange}{Push} }$$
  • where $$d(\mathbf{c}_i, \mathbf{c}_j) = 1-P\underbrace{f(\mathbf{c}_i)}_{ \color{cyan}{approximation} } \cdot P\underbrace{f(\mathbf{c}_j)}_{ \color{cyan}{approximation} }$$ $f(\cdot)$ approximates the inverse norm of a vector.

Numerical Evaluation

Experimental Setup

Cross-Posed Labelled Faces in the Wild

    • Synthetic fusion dataset by randomly pairing classes.
    • 10,760 samples over 188 classes.

Encryption and Training Parameters

  • Encryption Parameters: $(n,q)$
    • Library: Microsoft SEAL
    • $n$: $2^{14}$ or $2^{15}$ depending on multiplicative depth.
    • $q$: chain of large prime numbers totaling 420, 580 or 860 bits.

  • Training Parameters:
    • Learning Rate: $5 \times 10^{-3}$
    • Weight Decay: $1 \times 10^{-4}$
    • Training Epochs: 1000

Fusion Improves Performance, Reduces Dimensionality

  • Fusion improves performance:
    • Face by 11.07%
    • Voice by 9.58%
  • Dimensionality Reduction: $512D \rightarrow 32D$ (16$\times$ compression)

Comparison of Normalization Methods

Computational Complexity

    • Projection is costliest operation

Is FHE the Panacea for Privacy?

  • Security and privacy are very often conflated with each other.
    • Different but related concepts.
    • Homomorphic encryption: controls access to private information.
    • Differential Privacy: analysis + manipulation of private information.
  • Postulates:
    • There is no privacy without security.
    • Homomorphic encryption is an ideal tool for enhancing privacy but it is not a privacy technique in and of itself.
Ideal solution: Differential privacy + Homomorphic Encryption


    • Introduces the first multimodal feature-level fusion system in the encrypted domain.
    • Improves performance of original templates while reducing their dimensionality.
    • Incorporates polynomial approximation for approximate normalization.
    • Incorporates FHE-Aware Learning to improve performance.
HEFT: Encrypted Biometric Fusion     VishnuBoddeti