Evaluation

Evaluation of protection measures

Software-based and hardware-based protection measures can be evaluated by means of different criteria. Some important criteria are

  • Accuracy (proportion of correct predictions on test data)
  • Robustness (accuracy on manipulated data)
  • Privacy (effectiveness of membership inference attacks)
  • Training Time (duration of the model’s training process)

To interpret an evaluation adequately, it should be compared to the evaluation of the same AI application without any protection measure.

In the following, the evaluation results of some protection measures on exemplary AI applications are depicted.

Software Measures

MNIST Dataset

Protection MeasureAccuracyRobustnessPrivacyTraining Time
no98,3 %11,9 %0,4572 s
DP-SGD94,2 %4,8 %< 0,198 s
Anomaly Detection98,3 %4,4 %< 0,165 s
Adversarial Training98,9 %76,8 %0,12286 s

CIFAR10 Dataset

Protection MeasureAccuracyRobustnessPrivacyTraining Time
no81,6 %18,4 %0,31373 s
DP-SGD63,9 %54,5 %< 0,11061 s
Anomaly Detection79,4 %17,6 %0,4313 s
Adversarial Training71,9 %23,1 %22,02930 s

Hardware Measures

Protection MeasureDelayPower ConsumptionDevice / Setup
Modell-Signierung282 ms< 0,01 WJetson Nano / Raspberry Pi 3, Zymkey 4i
Modell-Signierung12 msmittelHuawei P20 Pro (Android)
Sensordaten-Attestierung77 ms0,15 WRaspberry Pi 3, NXP SE050 Edge Lock, 3-Axis Accelerometer, Burst-Read (6 Byte, I2C API)
Sensordaten-Attestierung0,221 msgeringHuawei P20 Pro (Android)
Verschlüsselung (AES128)2,68 kB/s0,07 - 0,15 WRaspberry Pi 3, NXP SE050 Edge Lock (CBC Mode)
Verschlüsselung (AES128)2,617 kB/s< 0,01 WRaspberry Pi 3, Zymkey 4i (ECDSA Signature, Mode unknown)
Verschlüsselung (AES128)4566 kB/s0,07 - 0,14 WOP-TEE, STM32MP1 (CTR Mode)
Verschlüsselung (AES128)0,095 msgeringHuawei P20 Pro (Android)
Last modified April 2, 2024: minor change of Best Practices' description (1df7aed)