1 - AI Classes

Taxonomy of AI applications

Classification of AI applications

The stated goal of the SENSIBLE-KI project is to secure embedded and mobile AI applications. In order to ensure a standardized security protection, it is necessary to systematize AI applications. The specific needs for protection can then be determined by means of discrete AI classes.

Based on the evaluation of a wide range of AI applications, the following classes were identified in this project. It is a vertical classification which is based on different properties of the AI applications.

The individual protection needs can be determined by categorizing the application with these different levels.

Source of Input Data

Where does the input data come from?
Class 1:explicit user input
Class 2:implicit user input (Tracking)
Class 3:sensory data

Type of Input Data

What is the format of the input data?
Class 1:image
Class 2:audio
Class 3:text
Class 4:other

Personal Reference

Does the input data contain sensitive information?
Class 1:non-critical
Class 2:indirect personal reference
Class 3:direct personal reference

Processing of Input Data

Is the Input Data processed and if yes, how?
Class 1:no
Class 2:yes, automatically
Class 3:yes, manually

Preparation of Input Data

How is the input data prepared?
Class1:data cleansing
Class 2:anonymization
Class 3:feature engineering

Training Time

When and how often is the model trained?
Class 1:model is trained once (offline learning)
Class 2:model is trained continuously (online learning)

Training Location

Where is the model trained?
Class 1:decentralized und decoupled between different devices
Class 2:decentralized, peer-to-peer
Class 3:centralized on a server
Class 4:federated

Deployment

Are there vulnerable communication paths?
Class 1:Applications which are deployed on a device and don't have to communicate with a server
Class 2:Applications which use a model on a server
Class 3:Applications which get their model from a server

Type of Model

What is the structure of the model?
Class 1:classical (transparent) machine learning algorithm
Class 2:neural networks

Protection Measures

Which measures have been taken?
Class 1:software measures
Class 2:hardware measures
Class 3:both
Class 4:neither

Type of Output

What is the model's task?
Class 1:classification
Class 2:regression
Class 3:data creation

2 - AI Examples

Examples of AI applications

In order to categorize AI mobile and edge applications into several classes regarding their security goals, a functional evaluation was performed for numerous existing AI applications in this project. The evaluated applications originate from a wide scope (biometry, smart home, smart living, smart devices, etc.), to identify a general classification that suits a variety of applications. From this functional evaluation several classes were derived.

Below, four different applications will be evaluated along with its determined classes, showing how a classification can be achieved and to validate that the determined classes may be applied to numerous AI applications.




Gang-Authentifizierung (Seamless.me)
CategoryPartial aspect
InputWhere does the data come from?
  • Sensors
Number of data classes?
  • One Class Classification / Novelty Detection
Type of data?
  • Sensor data
TrainingWhere does training take place?
  • decentral, decoupled
When is new training data added?
  • continuous (online learning)
ModellWhere is the model deployed?
  • on-device
What is known?
  • Supervised Learning
  • own histogramm-based novelty detector
  • Scikit-learn framework
OutputWhat does the output look like?
  • Probability Scores
ArcFace: Additive Angular Margin Loss for Deep Face Recognition
CategoryPartial aspect
InputWhere does the data come from?
  • Sensors
Number of data classes?
  • Multiclass
Type of data?
  • Image
TrainingWhere does training take place?
  • decentral, decoupled
When is new training data added?
  • once when model is deployed
ModellWhere is the model deployed?
  • on-device
What is known?
  • Supervised Learning
  • GAN
  • Tensorflow
OutputWhat does the output look like?
  • Image, Generation Simularity Score
Realtime Object Recognition (YOLObile)
CategoryPartial aspect
InputWhere does the data come from?
  • explicit user input
Number of data classes?
  • Multiclass
Type of data?
  • Image
TrainingWhere does training take place?
  • zentral
When is new training data added?
  • once when model is deployed
ModellWhere is the model deployed?
  • on-device
What is known?
  • Supervised Learning
  • DNN, group Lasso method
  • YOLOv4
OutputWhat does the output look like?
  • Klassifizierung und Bounding Boxes
Object Detection (R-CNN)
CategoryPartial aspect
InputWhere does the data come from?
  • explicit user input
Number of data classes?
  • Multiclass
Type of data?
  • Image
TrainingWhere does training take place?
  • -
When is new training data added?
  • once when model is deployed
ModellWhere is the model deployed?
  • on-device
What is known?
  • Supervised Learning
  • CNN
  • Tensorflow
OutputWhat does the output look like?
  • classification

3 - Security Goals

Security goals to be achieved for AI applications

An important step for evaluating the security of systems using Artificial Intelligence (AI)- applications is to define specific security goals for the AI. With these security goals, applications may be rated and their protection needs can be determined.

Similarly to the type of classical security goals in IT security, the following security goals were derived and transferred to the field of AI.

Details about attack vector against AI applications

The attack vector against AI applications is best described using the knowledge and the skills of the attacker

Attacker knowledge

Depending on the attacker’s access to the internal information of the AI application (full, partially, none) it is referred to as “blackbox, greybox or whitebox” scenarios. Attackers could have knowledge and access to any of the following aspects:

  1. the AI application’s training data
  2. the training algorithm
  3. the AI application’s trained model and its parameters

An attacker with access to all three aspects is called an attacker with “perfect knowledge”. Observing such attackers allows making worst case assessments for the security of AI systems.

However, in practice such attackers are not realistic. Attackers with “limited knowledge” are more realistic.

Attacker capabilities

The capabilities of an attacker can be the following, depending on the influence he has on the trained model und it’s training data

Training dataAI model
add or delete arbitrary data points and labelsunlimited number of interactions after training
add or delete arbitrary data points without influencing labelslimited number of interactions after training
modification of arbitrary or single data pointsmanipulation of the training process
modification of labels of specific data pointsinfluencing of training steps and future adjustments

 

Integrity

Integrity of AI models means the models make the right predictions for a variety of data.

An example would be a spam filter trained on reasonable data. It should still work correctly detecting spam, even if an attacker were to write spam mails containing many elements of a non-spam mail.

Details

Integrity in AI applications can be impaired on three different levels.

  1. Data level: targeted corruption of data the model should predict
  2. Model level: targeted manipulation of the model’s behaviour or the model’s parameter.
  3. Output/Object level: targeted manipulation of reactions from the AI system towards specific model predictions.

Data level

On the data level, the model’s inputs may be altered specifically to produce incorrect predictions.

Model level

On the model level, the model’s parameters may be altered during or after training, producing incorrect predictions.

Object level

On the object level, the model’s outputs may be manipulated in order to make the surrounding system react incorrectly to the input.

Integrity is also linked to confidentiality and controllability. In case of confidentiality, the attacker may achieve invalid system states due to nonsensical inputs. On the other hand the controllability is compromised if the attacker teaches the system unintended behaviour on the model level.

 

Availability

Availability in AI models ensures that the model fulfills the functionality it is used for.

We take a look at the spam filter again. Availability aims less at declining spam (see integrity) but to ensure that all non-spam messages pass.

Details

If an attackers is targeting the availability of a model he makes the system decline harmless instances in which case the system can't work properly anymore. If the output of an AI model is integrated into the function of the system this can be seen as a denial-of-service attack.

 

Confidentiality

Confidentiality describes a state in which the internal properties and confidential information on the trained model remain hidden from a potential attacker.

The attacker should not be able to work out which words, characters or formatting the spam filter uses to detect spam.

Details

An attack on the confidentiality may allow an attacker to gather sensitive and confidential information regarding the trained AI model, its properties, structure and parameters. With this information, the attacker could be able to steal the intellectual properties, manipulate or attack the privacy of the training data.

 

Privacy

Privacy in AI applications means the protecting the privacy of training data used to train the AI model. The models should withhold all private information.

A spam filter which was trained with private mails of Alice and Bob should not allow conclusions on what Alice or Bob wrote. Neither due to properties nor due to behaviour.

Details

In case of an attack on a model’s privacy the attacker may gain information regarding the - maybe sensitive - training data. This can lead to serious consequences for the privacy of the involved parties. An important aspect of privacy is anonymity and the disctinction to pseudonymity. Anonymity can be defined as protections against general identification and pseudonymity as protection against identification of the name.

 

Assignment of security goals and concrete attacks against AI models

There are known attacks in existence against AI applications models targeting the following security goals.

Security goalAttacks
IntegrityAdversarial Attacken
ConfidentialityData Poisoning
AvailabilityModel Stealing und Model Extraction
PrivacyModel Inversion, Membership Inference, Attribute Inference, Property Inference
Details about the attacks

Adversarial attacks

For an adversial attack, the attacker specifically manipulates a data point making the trained AI model predict said data point incorrectly. This can be a targeted change of a pixel in an image so that the object in the picture can’t be recognised anymore.

Data Poisoning

For data poisoning, the attacker may manipulate training data while training. Therefore the quality of the model’s predictions can be manipulated. For example, an attacker can use this to bypass a spam filter. When a model was trained to associate certain words with non-spam messages an attacker can use these words to trick the model into not recognising spam.

Model Stealing, Model Extraction

Model stealing is used to copy or steal a trained model from its original owner.

Model extraction is a special form of model stealing in which a model is stolen via blackbox access. This means an attacker only has access to an interface of the model but the received predictions are sufficient to train a substitute which performs just like the original.

Model Inversion

Model inversion is all about gaining information about the training data of a trained AI model. More specifically, this means an average representation of the training data (of a certrain class) may be recovered.

Membership Inference

Membership Inference attacks focus on the question if a given data point was used in training of the model. This may result in a privacy problem. For example if we have a classificator that proposes treatment methods to cancer patient, we can assume it was trained on data of cancer patients. The information of a specific data point being used in training translates to the information that the associated person has cancer.

Attribute Inference

For attribute inference we use public available characteristics of a data point which was used to train a model and access to this model to gather information about all sensitive and private characteristics of this data point.

Property Inference

For property inference we receive properties of the whole training set of an AI model, for example the distribution of the data. This kind of information can lead zu privacy problems especially for minorities in the distribution.

 

Literature recommendations

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety. 2016. url: https://arxiv.org/pdf/1606.06565.

Marco Barreno, Blaine Nelson, Anthony D. Joseph, and J. D. Tygar. “The security of machine learning”. In: Machine Learning 81.2 (2010), pp. 121–148. issn: 1573-0565. doi: 10.1007/s10994-010-5188-5. url: https://link.springer.com/content/pdf/10.1007/s10994-010-5188-5.pdf.

Mark Bedner and Tobias Ackermann, “Schutzziele der IT-Sicherheit,” Datenschutz und Datensicherheit - DuD 34, no. 5 (May 2010): 323–28, doi: 10.1007/s11623-010-0096-1.

Luis Muñoz-González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C. Lupu, and Fabio Roli. “Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization”. In: Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security - AISec ’17. pp. 27–38. isbn: 9781450352024. doi: 10.1145/3128572.3140451.

Nicolas Papernot. A Marauder’s Map of Security and Privacy in Machine Learning.url: http://arxiv.org/pdf/1811.01134v1. 2016

Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. “Practical Black-Box Attacks against Machine Learning”. In: Proceedings of the 2017 ACM on Asia Conference on Computerand Communications Security. Ed. by Ramesh Karri. ACM, 2017. isbn: 9781450349444. doi: 10.1145/3052973.3053009.

Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, and Michael P.Wellman. “SoK: Security and Privacy in Machine Learning”. In: 3rd IEEE European Symposium on Security and Privacy. Los Alamitos, California: Conference Publishing Services, IEEE Computer Society, 2018. isbn: 9781538642283. doi: 10.1109/eurosp.2018.00035.

4 - Software Measures

Software-based protective measures against attacks on AI applications

In the following, a selection of protection measures which increase the privacy or integrity of machine learning (ML) models (e.g. neural networks) is presented.

First, the general training of neural networks is described since the protective measures are applied within the training. A neural network is a complex mathematical function that is represented by interlinked layers which consist of many parameters. A neural net learns to classify data points of a target data distribution by means of training data. Here, learning refers to the adjustment of the neural net’s internal parameters in an iterative process, the training. Therein, the neural net classifies training data, s.t. the predicted class is propagated back through the network to calculate the direction in which the parameters have to be adjusted in order to get close to the correct class. The adjustment is called gradient descent. The training is terminated when the model’s prediction quality stops improving.


Adversarial Retraining




Adversarial Retraining is a protection measure against so-called adversarial examples (AE). AEs are manipulated inputs for neural networks which aim at causing a wrong prediction. The manipulation is as subtle such that human observers can hardly recognize it as such. In adversarial retraining, AEs are integrated into the training data set of the model to protect so that its adversarial robustness and hence its integrity is improved.

One impressive example of adversarial attacks is the one pixel attack which only changes a single pixel of an image. This minor perturbation is sufficient in many cases to trick the targeted neural net s.t. it predicts a wrong class.


Functioning

An AE can be produced within an adversarial attack by altering a real data point that was classified by the target model. More precisely, the prediction is propagated back through the network up to the input data point, s.t. its direction towards a wrong prediction can be calculated. Accordingly, the input data point is perturbed just enough to be misclassified. Depending on the concrete kind of adversarial attack, this gradient descent within the data point is applied differently or repeatedly. AEs are produced within adversarial retraining and together with the training data are given to the model to train and protect.


Usage

Adversarial retraining cannot be applied to linear models like e.g. support vector machines or linear regression. In contrast, neural networks acquire a significantly improved robustness against AEs when retrained with adversarial retraining.


Differential Privacy




Differential Privacy (DP) is a popular metric that allows to measure the influence of particular data points on the result of a data processing. This is equivalent to the privacy loss of the persons to which the data belongs. The privacy loss can be limited by inducing noise into the data processing. Although DP was originally invented in the context of databases, it is applied to privacy-preserving ML in recent years.

In 2006, the streaming service Netflix announced a public competition called Netflix Prize. The goal was to find the best algorithm to predict the movie ratings of certain users. For this purpose, the streaming service provided user data from more than 500,000 users for the training of competing algorithms. Two researchers at the University of Texas developed a linkage attack that maps the published user data with movie ratings from IMDb, a movie rating platform. As a result, many Netflix users have been de-anonymized.


Functioning

In the context of ML, privacy can be protected by inserting noise into the information about training data during training of a model. More precisely, the gradient corresponding to a single data point is clipped s.t. its norm does not exceed a certain value, and the gradient is perturbed by random noise. The combination of both techniques leads to a reduced privacy loss and even enables the quantification of it.


Homomorphic Encryption




Homomorphic Encryption (HE) enables the execution of mathematical operations like addition or multiplication on encrypted data. Results are still present in an encrypted form and are visible only with knowledge of the corresponding decryption key. Note that the same results are obtained as if neither encryption nor decryption would have been applied.

Researchers work on practical applications of HE like, e.g., the improvement of security and transparency of elections. Encrypted votes could be counted automatically—and therefore correctly—by means of the Paillier crypto system for example. This would enable authenticity of elections while also protecting the privacy of voters.


Usage

HE could be used in ML to protect the privacy of persons whose data are used either for training of a model or for inference by allowing models to operate on encrypted data. Unfortunately, no such technique has been developed for neural networks so far.


Anomaly Detection




Anomaly detection comprises methods to detect unusual patterns in data. Critical inputs like AEs can be filtered out of training data by means of anomaly detection. This leads to an improved robustness and thus to the integrity of ML models.

Anomaly detection is applied in many areas like for example text and image processing, fraud detection, or medicine.


Usage

Usually, anomaly detection is applied to improve the robustness of neural networks by using a second neural network that detects and removes critical inputs. This method requires additional computing resources. As an alternative to protecting the robustness and integrity, anomaly detection can be used to improve the privacy protection of training data in ML. Thus, membership inference attacks can be used to remove highly vulnerable data from the training dataset.


Evaluation

In order to compare the different protection measures and to evaluate their suitability in the context of resource constrained embedded and mobile AI, several ML experiments were conducted. Therein, neural networks were trained each with one or without any protection mechanism on the MNIST or CIFAR-10 dataset. The MNIST dataset contains 70.000 gray-scaled images of hand-written digits. In contrast, the CIFAR-10 dataset comprises 60,000 colored images of animals or vehicles. For the MNIST dataset, a multi-layer perceptron (MLP) consisting of about 670,000 adjustable parameters was used, while a convolutional neural network (CNN) was selected for the CIFAR-10 dataset that consists of approximately 5.8 mio. parameters. The models were trained over 50 epochs and afterwards deployed on different embedded devices to infer 10,000 test data points. For the inference, the most accurate version of a model over all training epochs was selected to classify the test data.


Comparison of Prediction Quality

The chart shows the course of the test accuracy (prediction quality) for each protection mechanism on each dataset. On each dataset, the accuracy of models trained with differential privacy is the worst. ML models without protection measure similarly achieve a high accuracy on each dataset. Adversarial Training and retraining influences the accuracy positively on the MNIST dataset while they lead to a decreased accuracy on the CIFAR-10 dataset.




Comparison of Training Time

Embedded and mobile devices have lower computational power compared to desktop PCs, rack servers, or computing clusters. Therefore, the training of neural networks is executed on one of the latter machines. Nevertheless, some embedded devices like the Jetsons of Nvidia enable training. The following chart shows the training time of the different ML experiments on two different Jetson devices and on a desktop PC. The training time is expressed in seconds on a logarithmic scale. Trainings without protection measure and with anomaly detection require approximately the same time, while differential privacy leads to significantly higher times. Adversarial training in turn requires a multiple of that training time. Furthermore, there are large differences between different devices. The Jetson Nano is way slower than the Jetson Xavier NX, which in turn is substantially slower than the desktop PC. Note that the installation of many code libraries is very difficult on the Jetson devices, wherefore the protection mechanisms anomaly detection and adversarial (re)training were not applied on them.

Comparison of Inference Time

The trained ML models were deployed on different devices to infer each of the 10,000 test samples of each dataset. The average inference time in milliseconds per data point is illustrated in the following chart. A clear order between the devices PC, Jetson Xavier NX, Jetson Nano, Coral Dev Board Mini, and Neural Compute Stick 2 (NCS2) can be recognized for the simpler MNIST dataset, where the protection measures do not have any influence on the inference time. On the more complex CIFAR-10 dataset, the NCS2 is faster than the Jetson Nano. Further, the inference of models protected by differential privacy is slightly slower than others. This is due to a replaced model layer in the CNN so that differential privacy can be tracked. Unfortunately, no CIFAR-10 experiments could be executed on the Coral Dev Board Mini since the CNN could not have been compiled for its inference chip named Edge TPU.

Literaturempfehlungen

Goodfellow, I. J., Shlens, J., and Szegedy, C. “Explaining and Harnessing Adversarial Examples” in ICLR (2015).

Abadi, M., Chu, A., Goodfellow, I. J., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. “Deep Learning with Differential Privacy” in ACM CCS (2016).

Acar, A., Aksu, H., Uluagac, A. S., and Conti, M. “A Survey on Homomorphic Encryption Schemes: Theory and Implementation” in ACM CSUR (2018).

Chandola, V., Banerjee, A., and Kumar, V. “Anomaly Detection: A Survey” in ACM CSUR (2009).

5 - Hardware Measures

Trusted computing techniques to protect AI applications against attackers

Evaluation Results - Trusted Computing Solutions

General Overview of Identified Mechanisms

The solutions and mechanisms of Trusted Computing were sorted into two groups, which differ in the way they realize the goals of Trusted Computing.

The first group, Trusted Execution Environments and related Technologies, comprises implementations that use preexisting features of the application processor. Opposed to that, the second group consists of Tamper Resistant Hardware and includes hardware modules specifically designed to provide functionality needed to reach the goals of Trusted Computing.

Trusted Execution Environments (TEEs) and Frameworks

Underlying Technologies

The idea of Trusted Execution Environments (TEEs) is to achieve the separation of critical processes and their data from any sort of non-trustworthy code. In the case of a TEE the domain of “non-trustworthy code” even includes the normal operating system (Rich Execution Environment (REE)) and all processes running in it. The TEE has got its own secure operating system and own processes (Trusted Applications (TAs)).

To ensure the separation of the secure domain of the TEE from the non-secure REE, an underlying hardware-based technology is required. Application processors from different manufacturers provide differing technologies to enforce separation which will be described in the following:

TrustZone (TZ) is supported by (some) ARM Cortex-A processors. It supports the separation of resources into a single “Secure World” and a single “Non-Secure World”. The separation is enforced and realized by processor bits signaling the current context, which can be changed by specific instruction set operations that trigger a secure context switch from one world into the other. The resources governed by TrustZone include regions of the main memory, access to certain peripherals and CPU usage. Due to the high prevalence of ARM Cortex-A processors in mobile and embedded systems, TrustZone is supported by many devices in these sectors.

TrustZone-M (TZM) is the equivalent technology for ARM Cortex-M processors. TZ and TZM work mostly in the same way, except that TZM facilitates the context switches via branch instructions which allows a lower latency. TZM is optimized for low power profiles.

Intel SGX is an Instruction Set Extension provided by Intel on (some) of its x86 CPUs. Intel SGX allows to separate the main memory into “Enclaves”. The separation is enforced by encryption of the respective memory region. This technology allows an arbitrary number of secure “Enclaves” but does not secure any other resources than the memory.

AMD SEV/PSP: AMD provides a virtualization technology using the AMD Platform Security Processor (PSP), which is itself an ARM chip supporting TrustZone. The PSP generates and manages the cryptographic key material used to encrypt the memory regions and CPU registers belonging to the secured “Guests”. An arbitrary amount of “Guest” can be secured this way.

RISC-V Physical Memory Protection (PMP) is an additional module, part of RISC-V architecture, that enforces memory access policies via the application processor. These policies can be used to implement a secure domain for a TEE on a RISC-V processor.

Implementations

In the following table, the necessary requirements of the CPU (Platform) have been listed for each solution. In addition, it has been indicated whether the implementation is compliant with the GlobalPlatform TEE API Specification (GP Compliance).

Table 1: Commercial / Proprietary Solutions

NamePlatformGP ComplianceComments
Qualcomm TEE (QTEE / QSEE)TZYes (SFS API)QTEE uses the Secure Processing Unit (SPU) on some Qualcomm SoCs next to TrustZone. Certifications: FIPS 140-2 Level 1 (Crypto Lib), FIPS Level 2 (crypto engine core), FIPS Level 1 (PRNG), FIPS Level 1 (Inline Crypto engine)
KinibiTZYesImplementation from Trustonic especially for Exynos Chips.. Certifications: Common Criteria (TEE)
TEEGRISTZYesFor Samsung Exynos, by Samsung. Starting with the Samsung Galaxy S10 TEEGRIS replaced Kinibi as the TEE implementation on many Samsung smartphones.
iTrusteeTZYesHuawei’s own implementation. Replaced Huawei’s older solution, Trusted Core, on recent Huawei Smartphones. Certifications: CC EAL2+
Kinibi-MTZMNo infoImplementation by Trustonic for ARM Cortex-M CPUs. Currently only supports Microchips SAML11.

 
Table 2: Research- and. Open-Source Projects

NamePlatformGP ComplianceComments
TrustyTZ, Intel SGXNoOpen-source implementation by Android
GoTEETZNoOpen-source implementation by f-secure. Currently only supports following SoCs: NXP i.MX6ULL, NXP i.MX6ULZ.
OP-TEETZYesOpen-source implementation by Linaro/TrustedFirmware Project. Target platform is a Linux kernel without external TRH, with a CPU with TZ support. Design goals: Isolation, low memory profile and portability.
uTangoTZMNoOpen-source project in progress. Targeting IoT devices. Goal is a Multi-World TEE following a Microkernel-Approach and Zero Trust Model: The Trusted Worlds run in TZ Non-Secure Mode. While suspended, their data is stored in the Secure World. The only process running in Secure Mode is the uTango kernel itself. The project’s Trusted Computing Base is, with 6kB, quite small and thus easier to verify.
Trusted FirmwareTZ, TZMNoImplementation by Linaro. Serves as a base layer for the creation of TEEs targeting ARM Cortex-A and –M platforms. Supports Secure Boot, various crypto primitives, secure storage and attestation. Provides a reference implementation with Secure Monitor for TrustZone.
Pelagi EnclaveRISC-V (PMP)NoA work in progress Open-source project aiming to develop a TEE for RISC-V platforms. Scalable and based on RISC-Vs PMP feature. Currently barely any information available, how exactly the project can be used when finished and what features it will support.
MultiZone SecurityRISC-V (PMP)NoOpen-source implementation by Hex Five. The implementation allows the creation of an arbitrary number of secure environments. The security of each environment can be configured via policies.
KeystoneRISC-V (PMP)NoOpen-source framework for developing TEEs for RISC-V. Supports monitor-based primitives to create TEEs tailored to needs of differing host platforms (e.g., IoT vs. Server).

 

Tamper Resistant Hardware (TRH)

Types

It is hard to cleanly differentiate between the different types of TRH, since there appears to be no universally accepted definition about when which term is applicable. The only type that is officially specified by the Trusted Computing Group (TCG) is the term Trusted Platform Module (TPM).

To avoid unnecessary confusion, the types used in this document to describe and characterize the identified TRH, will be defined. If a specific module did not fit any of these descriptions or not enough information was available, the term used by the manufacturer was adopted.

Trusted Plattform Module (TPM): Provides secure, tamper resistant on-chip storage, means for key generation and management, a random number generator (RNG) - which does not need to be a True Random Number Generator (TRNG) -, and attestation. Specified by the TCG and in ISO/IEC 11889. The currently “active” Versions of the TPM specification are TPM 1.2 and TPM 2.0 which differ mainly regarding their supported cryptographic primitives.

Secure Element (SE): A module or chip resistant to physical manipulations. Provides key generation and management as well as an unspecified set of cryptographic primitives like hash- or encryption algorithms. On chip-storage is optional.

Implementations

For every TRH the respective Type is given (as explained in Section “Types”), as well as – in case of TRH from the mobile sector - the corresponding Platform. Some products can be used with arbitrary platforms and are marked accordingly.

Androids StrongBox Keymaster API was introduced with Android version 9 and provides functions to check whether key material is stored and managed by a StrongBox compliant Secure Element, providing a separate CPU, secure storage, a TRNG1 and tamper resistance. In case of TRH belonging to mobile devices, compliance with StrongBox is given as well (StrongBox).

Table 3: Tamper Resistant Hardware

NameTypePlatformStrongBoxComment
Titan MSE>= Google Pixel 3YesSeparate chip by Google. Verifies the device intergity at boot using a signature stored in the chip. Further, verifies unlock requests of LockScreen, stores keys and supports secure firmware updates. Based on ARM Cortex-A3. Protection against side-channel attacks and provides tamper detection. 64 kB flash available.
Apple Secure EnclaveSecure EnclaveiOS, >= iPhone 5sNo infoIntegrated security subsystem on Apples system on chips (SoCs). Contains the Apple Secure Enclave Processor (SEP), 4MB internal storage and additional external RAM, that is solely connected to the Secure Enclave and encrypted by the SEP.
Samsung eSE S3K250AFSE>= Samsung Galaxy S20, standaloneYesMight be part of the Knox Vault. 252 kB Flash. Certifications: CC EAL 5+
Samsung eSE S3FV9RRSEstandaloneNo info1.5/2.0 MB Flash. Might be part of the Knox Vault since Samsung Galaxy S21. Certifications: CC EAL 6+
Knox VaultTamper Resistant Environment>= Samsung Galaxy S21YesLike Apple Secure Enclave, Knox Vault is a security subsystem based on a SE security subsystem based on a SE, which provides, amongst other functions, secure storage for key material and data.
Huawei inSESE>= Kirin 960No infoDedicated security chip embedded into the application processor in HiSilicon Kirin 960, 970, 980, 710 (P 30 line, Mate 20 line) Certifications: CFNR Technology Certification of Mobile Financial Service Chip Security, China UnionPay Card Chip Security Specifications, Certificate for Commercial Cipher Product Models, EMVCo (für Kirin 980)
Qualcomm SPUSE>= Qualcomm Snapdragon 855YesSubsystem integrated in some Qualcomm SoCs providing among other functions secure key management, crypto-primitives and –accelerators. Certifications: EAL 4+
TPM 1.2TPMstandaloneNo (no TRNG)See section “Types”. Interface for data transmission and other details not covered by the specification are implementation and profile dependent.
TPM 2.0TPMstandaloneNo (no TRNG)See section “Types”. Interface for data transmission and other details not covered by the specification are implementation dependent.
Microchip ATECC608SEstandaloneNo infoSecure key storage for up to 16 elements. Provides signature- and hash-alogrithms and AES encryption .
NXP EdgeLock SE050SEstandaloneNo infoProvides signature- and hash-alogrithms, AES encryption and 50kB secure flash. Supported host interface is I2C. Certifications: EAL 6+

 

Criteria of Evaluation

To evaluate some of the implementations, which seemed promising regarding the goals of the project SENSIBLE-KI, the following criteria of evaluation were chosen:

  • Accessibility and Usability:
    The probability that a proposed security mechanism is actually used, is lowered by the direct (financial) and indirect (additional knowledge and time required) cost of this mechanism.

  • Supported Features:
    The actual functionality of a specific solution or module is relevant, especially when matching Trusted Computing mechanisms to AI Use-Cases, later on. The evaluated functionality was focused on the supported crypto-primitives and if or how much data can be securely stored by the solution.

  • Performance:
    Using any Trusted Computing mechanism will cause additional overhead e.g., from context switches or data transmission to an external module. To gain an overview of the respective performance effects of the solutions, performance measurements were conducted on selected cryptographic functions. The results are relevant when matching Trusted Computing mechanisms to AI Use-Cases, later on.

  • Market share:
    The prevalence of a specific implementation is important to rate its relevance in the current economy and was hence – if possible – evaluated.

Evaluated Solutions - Mobile devices

Choice of Devices

Choosing devices by Apple and Samsung, we are covering two of the largest smartphone manufacturers on the market. Because of the inherent differences in both soft- and hardware between iOS and Android devices, comparing phones with each operating system is essential.

Android devices currently cover about 70% of the market in smartphones. Because of this, different Pixel devices by Google were evaluated, since they use an optimized and unedited Android implementation, which does not contain any “bloatware” - meaning third-party apps by the Original Equipment Manufacturer (OEM).

In each case current high-end devices were picked, because these are sure to contain all of the important functionalities and features provided by the OEM at that time.

Note on the Performance Measurements

8 MB Blocks of random data were used as input for the symmetric crypto algorithms.

The API of the iOS Secure Enclave solely supports the usage of asymmetric cryptography. Hence, the performance measurements of the AES algorithm were conducted via ECIES decryption with a secp256r1 key (for AES-128-GCM) and a secp521r1 key (for AES-256-GCM) respectively. It must be noted that the performance of AES-GCM via ECIES under iOS is not comparable to the AES-GCM under Android, since key stored in hardware is only used for initial key-derivation when using the ECIES method.

Evaluations

Google Pixel (Android with Tensor SoC and Titan M-/M2-Chip)

  • Accessibility and Usability:
    Google provides the freely available Android Security Library, which is high in quality. In addition, developers can rely on the IDE Android Studio, various SDKs, thorough documentation and example-code. Furthermore, a large community of developers exists.

  • Supported Features: The Android Security Library provides various mechanisms and algorithms, including many algorithms covering encryption (symmetric and asymmetric), signatures, message digest, key generation and attestation. In addition, the Android KeyStore System API supports functionality to securely store cryptographic keys. On some devices with StrongBox compatible TRH, this hardware can be used with the KeyStore API. The number of keys that can be stored by the keystore is unlimited.

    Within the Trusted Execution Engine (Tensor Security Core) only the algorithms preloaded by Google can be executed. We assume that the evaluated devices use Androids own TEE implementation Trusty TEE. The TEE is used to secure the KeyStore in hardware, if StrongBox cannot be used.

  • Performance:
    Table 4: Performance – Pixel 5, Android 12

    Algorithmus / ProviderBouncyCastleAndroidKeyStore (Trusty TEE)AndroidKeyStore + StrongBox
    AES-128-GCM645.28 MB/s6.77 MB/s0.01 MB/s
    AES-256-GCM599,19 MB/s6,68 MB/s0,01 MB/s
    HMAC-SHA-2561.398,10 MB/s15,36 MB/s0,02 MB/s
    secp256r1 keygen1.666,67 keys/s39,35 keys/s4,25 keys/s
    secp256r1 ECDSA sign5.882,35 ops/s77,16 ops/s9,54 ops/s
    secp256r1 ECDSA verify2.500,00 ops/s2.500,00 ops/s833,33 ops/s
    Table 5: Performance – Pixel 6 Pro, Android 12
    Algorithmus / ProviderBouncyCastleAndroidKeyStore (Trusty TEE)AndroidKeyStore + StrongBox
    AES-128-GCM838.86 MB/s3.4 MB/s0.04 MB/s
    AES-256-GCM762,60 MB/s4,86 MB/s0,05 MB/s
    HMAC-SHA-2561.677,72 MB/s9,07 MB/s0,05 MB/s
    secp256r1 keygen2.777,78 keys/s112,23 keys/s11,50 keys/s
    secp256r1 ECDSA sign7.692,31 ops/s190,48 ops/s22,64 ops/s
    secp256r1 ECDSA verify4.761,90 ops/s1.694,92 ops/s714,29 ops/s

  • Market share: Google Pixel devices hold a relatively low share of the smartphone market. In 2021 they held about 1,2% of the German market and about 3,8% in the US.

 
Samsung Knox (Android)

  • Accessibility and Usability: In addition to the Andorid Security Library (see section for Google Pixel), Samsung provides its own freely available security library Samsung Knox SDK. The SDK is less thoroughly documented, and the usage is less prevalent than the Android pendant, because they are Samsung exclusive and solely used to access Samsungs own TRH Samsung Knox Vault.

  • Supported Features: Next to the Android libraries, Samsung supports the following security functions:

    • Weaver (Android password authentication),
    • Credential Storage (stores keys, biometric data and more) and
    • Samsung Attestation Key (supports firmware and other devices).
    • The keystore Knox Vault Storage can store an arbitrary number of keys.
    • The Knox Vault Processor solely executes algorithms preloaded by Samsung. We assume that the evaluated devices are equipped with Samsung’s own TEE implementation TEEGRIS, which was introduced with the Samsung Galaxy S10. TEEGRIS can be used to secure the KeyStore in hardware, if StrongBox cannot be used.
  • Performance:
    Table 6: Performance – Samsung Galaxy A50, Android 11

    Algorithmus / ProviderBouncyCastleAndroidKeyStore (TEEGRIS)AndroidKeyStore + StrongBox
    AES-128-GCM399.46 MB/s5.89 MB/sn.V.
    AES-256-GCM399.46 MB/s5.78 MB/sn.V.
    HMAC-SHA-2561198.37 MB/s14.39 MB/sn.V.
    secp256r1 keygen694.44 keys/s15.89 keys/sn.V.
    secp256r1 ECDSA sign4000 ops/s63.29 ops/sn.V.
    secp256r1 ECDSA verify1724.14 ops/s1298.7 ops/sn.V.
    Table 7: Performance – Samsung Galaxy S20FE, Android 11
    Algorithmus / ProviderBouncyCastleAndroidKeyStore (TEEGRIS)AndroidKeyStore + StrongBox
    AES-128-GCM838,86 MB/s7,9 MB/s0,18 MB/s
    AES-256-GCM838,86 MB/s8,62 MB/s0,18 MB/s
    HMAC-SHA-2561.677,72 MB/s21,51 MB/s0,37 MB/s
    secp256r1 keygen2.000 keys/s11,06 keys/s0,96 keys/s
    secp256r1 ECDSA sign11.111,11 ops/s74,85 ops/s12,17 ops/s
    secp256r1 ECDSA verify4.545,45 ops/s1.960,78 ops/s1.204,82 ops/s

  • Market share: In 2021 Samsung came in second in market dominance, holding about 20% of smartphone market shares.

 
Apple iPhone (Secure Enclave)

  • Accessibility and Usability: Apple provides the free and high-quality Apple CryptoKit library for security features. Additionally, developers can use XCode and a large amount of documentation and code examples. The community of iOS developers is large, due to the prevalence of Apple mobile devices.

  • Supported Features: CryptoKit offers the SecKey API (for asymmetric keys), the Common Crypto Library (for symmetric encryption, hash-based MACs, message digest and attestation) and CryptoTokenKit (for Smart-Card support). Many ECC-256bit keys can be stored in an exclusive, secure storage unit. Different key formats are not supported. Within the Trusted Execution Engine (Secure Enclave), only the algorithms provided can be executed. Deployment of third-party code is not possible.

  • Performance:
    Table 9: Performance – iPhone X

    Algorithmus / Providerwithout Secure Enclavewith Secure Enclave
    AES-128-GCM Decrypt2 (ECIES via secp256r1-Schlüssel)763,26 MB/s472,95 MB/s
    AES-256-GCM Decrypt (ECIES via secp521r1-Schlüssel)495,82 MB/sn. V.
    secp256r1 ECDH key exchange3.892,00 ops/s135,97 ops/s
    secp256r1 ECDSA sign3.769,10 ops/s95,36 ops/s
    secp256r1 ECDSA verify5.272,65 ops/s1.492,70 ops/s
    Table 10: Performance – iPhone 12 Pro
    Algorithmus / Providerwithout Secure Enclavewith Secure Enclave
    AES-128-GCM Decrypt (ECIES via secp256r1-Schlüssel)1.849,27 MB/s755,34 MB/s
    AES-256-GCM Decrypt (ECIES via secp521r1-Schlüssel)1473,18 MB/sn. V.
    secp256r1 ECDH key exchange3.417,93 ops/s148,43 ops/s
    secp256r1 keygen192,37 keys/s100,93 keys/s
    secp256r1 ECDSA sign3832,87 ops/s124,64 ops/s
    secp256r1 ECDSA verify6416,70 ops/s2181,78 ops/s
    Table 11: Performance – iPhone 13 Pro
    Algorithmus / Providerwithout Secure Enclavewith Secure Enclave
    AES-128-GCM Decrypt (ECIES via secp256r1-Schlüssel)1.172,12 MB/s757,24 MB/s
    AES-256-GCM Decrypt (ECIES via secp521r1-Schlüssel)883,49 MB/sn. V.
    secp256r1 ECDH key exchange7.820,13 ops/s212,38 ops/s
    secp256r1 keygen283,47 keys/s135,35 keys/s
    secp256r1 ECDSA sign6.527,13 ops/s193,38 ops/s
    secp256r1 ECDSA verify7.216,83 ops/s3.392,44 ops/s

  • Market share: In 2021 Apple was the third largest manufacturer of smartphones worldwide and held a market share of 19,2%.

 

Results - Overview

Table 12: Results Mobile Devices - Overview

ImplementationAccessibility and UsabilitySupported FeaturesPerformanceMarket share
Google PixelgoodhighHigh negative performance impact when using Android KeyStore TEE-backed, compared to mere software-based security; Very high negative performance impact when using KeyStore with StrongBox (TRH)very low
Samsung Knoxgood - averagehighHigh negative performance impact when using Android KeyStore TEE-backed, compared to mere software-based security; Very high negative performance impact when using KeyStore with StrongBox (TRH)very high
Apple iPhonegoodhighHigh negative performance impact when using the Secure Enclavevery high

Evaluated Solutions – Embedded Systems

Choice of Devices

OP-TEE was chosen as a representative for TEEs on embedded systems with ARM Cortex-A CPUs, because it is open-source and hence one of the few implementations that allows the cost-free implementation and integration of third-party TAs, which was necessary for our experiments. In addition, OP-TEE implements the GlobalPlatform (GP) TEE API, so implemented TAs for OP-TEE can thus be easily ported to other GP conforming TEEs.

As an alternative TEE, targeting ARM-Cortex M Processors with TrustZone-M, we also chose the commercial implementation Kinibi-M by Trustonic.

The NXP SE050 and the Zymkey 4i by Zymbit both represent Secure Elements and can be used with the Raspberry Pi, which makes them easier to evaluate and ideal for prototyping. The two SEs differ in the available crypto-primitives and the developer target-group. Additionally, a TPM 2.0 implementation by Let’sTrust was picked to cover the group of TPMs. The chosen module too is compatible with the RaspberryPi and implements the TCG TPM 2.0 specification.

Note on Performance Measurements

For the crypto algorithms either the maximum supported buffer size or, in case of no obvious cap, 1MB was used as size for the input buffer. In every case, the time was measured on the lowest possible API level: either right before and after the context switch (TEE) or at the start and end of data transmission to the module (TRH).

Evaluations

OP-TEE (STM32MP157A-DK1)

  • Accessibility and Usability:
    Next to OP-TEEs open-source code TrustedFirmware also provides developers with documentation, describing the usage and the process to setup a working OP-TEE on a platform. The documentation contains a build-guide and a description of the general architecture of the OS. For more detailed information regarding the different API calls, developers have to consult the respective GP TEE API 1.0. The documentation provided by TrustedFirmware is kept reasonably up-to-date and exhaustive. In addition, TrustedFirmware provides various – unfortunately entirely uncommented – example implementations of TAs and a build.git containing an assortment of ready-to-use builds for specific platforms. To use OP-TEE on a platform not covered by the build.git, the setup process is significantly more challenging and requires some experience and knowledge with operating-system development and –architecture, since in that case the image needs to be correctly assembled using the correct elements (op-tee, linux kernel, u-boot, platform specific drivers …) creating the needed Makefiles. OP-TEE supports many different platforms, however only a few of them are represented in the build.git.

  • Supported Features: OP-TEE provides functions according to the GP TEE Specification 1.0 and is thus rich in features. Among others, the TEE supports RSA- and AES-encryption and provides secure storage. Currently OP-TEE does not support remote- or key-attestation. As mentioned above, the implementation of TAs is possible, and OP-TEE even enables developers to secure certain peripherals via “pseudo-TAs” using TrustZones features.
    The STM board used for the experiments includes a TRNG and OP-TEE is, as far as we were able to determine, actually using it, when its random-function is called.

  • Perfomance:
    Table 13: Performance – OP-TEE (STM32MP157A-DK1)

    AlgorithmPerformance
    AES128-CTR Enc4,56 MB/s
    AES128-CTR Dec4,56 MB/s
    RSA2048-RSAES Enc2,44 kB/s
    RSA2048-RSAES Dec0,16 kB/s

  • Market share: No information can be provided on market distribution, as no data is available. OP-TEE is, however, well-known in the community.

 
NXP EdgeLock SE050 Development Kit

  • Accessibility and Usability:
    The documentation of the NXP Plug&Trust Middleware is available for download after a cost-free registration on the NXP website. The middleware itself and its source code can be obtained in the same way (a minimal version is also available on GitHub).
    The documentation is structured in a sometimes-confusing way and explanations are brief. All in all, it is, however, complete in regards of relevant information and developers with a certain knowledge of cryptography and SEs that know what they are looking for, will be able to use it without problems. The module communicates with the host via I2C and allows, in theory (we’ve only experimented with RaspberryPi 3 as host), the usage with all linux-based host devices with the respective interface.

  • Supported Features: The SE provides a PRNG conforming with NIST SP800-90A. Both AES- and RSA-encryption are supported, as well as different ECC-algorithms. In general, the API is quite extensive and supports many crypto primitives.
    The device offers 50 kB secured user flash memory, in which Secure Objects like cryptographic keys can be stored. The implemented key store supports key-attestation. The NXP SE050 also offers a second I2C interface that can be used to connect the SE to a sensor which would allow attested sensor data reads from the host via the TRH.

  • Perfomance:
    Table 14: Performance – NXP EdgeLock SE050 Development Kit

    AlgorithmPerformance
    AES128-CBC Enc2,7 kB/s
    AES128-CBC Dec2,7 kB/s
    HMAC-SHA2562,8 kB/s
    ECDSA-SHA256 Sign0,5 kB/s

  • Market share: No information can be provided on market distribution, as no data is available. The NXP EdgeLock is part of the Google SE Alliance, which could potentially heighten its prevalence and relevance in the future.

 
Zymkey 4i

  • Accessibility and Usability: The Zymbit API documentation is available on the Zymbit website in html and as PDF3. There are three distinct documentations for the C-API, the C++-API and the Python-API. The respective documents are partly inconsistent, especially regarding the descriptions of the equivalent functions in the three different APIs. One would not gain the same information about the device’s abilities and the functionalities of the methods by reading the Python-API than by reading the C-API. Even within the same language-API, different information can be gathered by reading the html version than by reading the PDF. Additionally, each documentation contains function descriptions about all methods supported by the sum of all Zymbit-devices and there is no way to filter them, in order to display only the ones relevant for e.g., the Zymkey 4i. So, for every function one must check the descriptive text, whether it is mentioned that this function is exclusive to a specific device or not. However, the reduced feature set, which will be covered in the next section, the available example code and the support of Python, the required pre-existing knowledge is relatively low and even developers that are not familiar with the C programming language, can use the API. Usage of the Zymkey 4i is theoretically possible with all devices with a I2C interface, the libraries however, are – at the time of the experiments – solely available as Debian packages and not open-source, so porting them could be difficult.

  • Supported Features: The Zymkey 4i offers a lock-Method, which first encrypts the input data with AES-128 and then signs it with the pre-installed ECDSA-key. It is not possible to store your own keys in the Zymkey 4i or generate new ones. A key store could only be implemented by encrypting them with the lock-function and storing them in the host memory. We assume that the Zymkey 4i uses the TRNG of the integrated ATECC608 chip, when the respective random-function is called, it was however not possible to verify this given or time and resources. Apart from the mentioned lock- and random-methods, the Zymkey supports only one more crypto-function which is ECDSA-SHA256 signatures, where only the signing is conducted on the Zymkey while the SHA digest appears to be calculated on the host device. Although the integrated ATECC608 supports more cryptographic algorithms than that, those are not used and supported by the Zymkey 4i. One would have to use the Zymbit HSM6 which is based on the same chip, in order to use these functions. Remote attestation is mentioned as a feature in the Zymkey’s Product Brief , however no information regarding how to use it could be found in the API documentation or the Zymbit forum.

  • Perfomance:
    Table 15: Performance – Zymkey 4i

    AlgorithmPerformance
    “Lock” (AES-128 Enc & ECDSA Sign)2,61 kB/s
    “Unlock” (AES-128 Dec & ECDSA Verify)2,61 kB/s
    ECDSA Sign0,23 kB/s

  • Market share: No information can be provided on market distribution, as no data is available.

 
Let'sTrust TPM 2.0

  • Accessibility and Usability:
    Different implementations of the TPM 2.0 API exist. For our experiments we used this stack implementation. To use the API one must refer to the applicable official specification documents provided by the TCG, which is quite complex and extensive. For easier use, a command line tool is provided in the above-mentioned GitHub repository. The tool itself is well documented and has examples, while the API implementation itself provides no guidance on how to use it correctly. The TPM 2.0 by Let’sTrust connects to the host device via SPI and should be usable with any Linux-based host.

  • Supported Features: The supported features of the Let’sTrust TPM 2.0 conform with the mandatory features specified by the TCG for the “Mobile Profile”. The TPM includes a deterministic RNG according to NIST SP800-90A. RSA and different signature algorithms are supported, as well as ECC and ECDH. However, it does not support AES encryption, or rather not the respective TPM command TPM2_EncryptDecrypt, since it is optional. Keys or other critical data can be stored in the 6kB tamper proof flash. At the time of evaluation, there was no way for remote attestation with the Let’sTrust TPM, however there are there are plans to change that. With the TPM2_Certify command, the presence of a key on the TPM can be attested, but the command TPM2_CertifyX509, which would also attest the attributes of a key/object with a X.509 certificate, is not supported by the module. The TPM does not have any other interfaces that would allow to attach further peripherals.

  • Perfomance:
    Table 16: Performance –Let’sTrust TPM 2.0

    AlgorithmPerformance
    HMAC-SHA2564,79 kB/s
    RSA2048-RSAES Enc3,45 kB/s
    RSA2048-RSAES Dec0,77 kB/s

  • Market share: No information can be provided on market distribution, as no data is available. However, TPM 2.0 implementations are generally widely used.

 
Kinibi-M

  • Accessibility and Usability:
    The Kinibi-M SDK is not open-source and can only be obtained (for free) after registration and request on the Trustonic website. Currently the only supported chip is the Atmel SAML11 (the production SDK is restricted to the SAML11 KPH, that includes the pre-installed unique device key). The SDK includes extensive, well-structured documentation of the API, a step-by-step Getting Started guide and several, well-documented code examples that make the SDK accessible.

  • Supported Features: The Kinibi-M Crypto API offers, among others, the following relevant functionalities: AES-128 encryption, SHA256 hashing, secure data storage in the Secure Storage (1kB on-board data flash) and random number generation. The latter could either be obtaining numbers from the TRNG of the SAML11 or a DRNG implementation. The documentation does not give any information regarding this.
    Key- or remote-attestation, in the sense that those terms are usually used in, is not implemented. Using the features of TrustZone-M, peripherals can be secured and accessed by the TAs. The SDKs intention is to enable developers to create TAs for Kinibi-M, so third party code can be executed.

  • Perfomance: Since time measurement on SAML11 Xplained Pro would have been incomparably more complex - as well as fundamentally different - than with the Linux-capable other boards, no performance measurements could be carried out on this board in the short time available.

  • Market share: No information can be provided on market distribution, as no data is available. Kinibi-M is currently only available on a single platform and hence the market share is supposedly very low. If more devices based on ARM Cortex-M CPUs with TrustZone-M support emerge, the prevalence of Kinibi-M may increase, due to the former popularity of Kinibi in the mobile sector.

 

Results - Overview

Table 12: Results Embedded Devices - Overview

ImplementationAccessibility and UsabilitySupported FeaturesPerformanceMarket share
NXP SE050average – highhighlowNo info
Zymkey 4ilow - averagelowlowNo info
Let’sTrust TPM 2.0highhighlowNo info
OP-TEEaverage – highhighaverage – highNo info
Kinibi-MlowlowNo infoNo info

  1. True Random Number Generator.

  2. Since the encryption via ECIES is the only algorithm requiring access to the asymmetric key in the secure enclave, only this specific algorithm was timed.

  3. https://s3.amazonaws.com/zk-sw-repo/zk_app_utils.py.pdf or https://s3.amazonaws.com/zk-sw-repo/zk_app_utils.c.pdf respectively.

6 - Evaluation

Evaluation of Protection Measures

Software-based and hardware-based protection measures can be evaluated by means of different criteria. Some important criteria are

  • Accuracy (proportion of correct predictions on test data)
  • Robustness (accuracy on manipulated data)
  • Privacy (effectiveness of membership inference attacks)
  • Training Time (duration of the model’s training process)

To interpret an evaluation adequately, it should be compared to the evaluation of the same AI application without any protection measure.

In the following, the evaluation results of some protection measures on exemplary AI applications are depicted.

Software Measures

MNIST Dataset

Protection MeasureAccuracyRobustnessPrivacyTraining Time
no98,3 %11,9 %0,4572 s
DP-SGD94,2 %4,8 %< 0,198 s
Anomaly Detection98,3 %4,4 %< 0,165 s
Adversarial Training98,9 %76,8 %0,12286 s

CIFAR10 Dataset

Protection MeasureAccuracyRobustnessPrivacyTraining Time
no81,6 %18,4 %0,31373 s
DP-SGD63,9 %54,5 %< 0,11061 s
Anomaly Detection79,4 %17,6 %0,4313 s
Adversarial Training71,9 %23,1 %22,02930 s

Hardware Measures

Protection MeasureDelayPower ConsumptionDevice / Setup
Modell-Signierung282 ms< 0,01 WJetson Nano / Raspberry Pi 3, Zymkey 4i
Modell-Signierung12 msmittelHuawei P20 Pro (Android)
Sensordaten-Attestierung77 ms0,15 WRaspberry Pi 3, NXP SE050 Edge Lock, 3-Axis Accelerometer, Burst-Read (6 Byte, I2C API)
Sensordaten-Attestierung0,221 msgeringHuawei P20 Pro (Android)
Verschlüsselung (AES128)2,68 kB/s0,07 - 0,15 WRaspberry Pi 3, NXP SE050 Edge Lock (CBC Mode)
Verschlüsselung (AES128)2,617 kB/s< 0,01 WRaspberry Pi 3, Zymkey 4i (ECDSA Signature, Mode unknown)
Verschlüsselung (AES128)4566 kB/s0,07 - 0,14 WOP-TEE, STM32MP1 (CTR Mode)
Verschlüsselung (AES128)0,095 msgeringHuawei P20 Pro (Android)

7 - Recommendations

Protective measures for prototypes and application scenarios

Protective Measures for Prototypes and Application Scenarios

1. Introduction

In the following, software-based protective measures and procedures from the field of Trusted Computing are proposed to enhance protection against attacks on AI applications on mobile and embedded devices.

Chapter 2. Considered Measures provides an overview of all the measures considered in this document, along with their respective protective effects. Subsequently, in chapter 3. Protective Measures Based on Application Characteristics, these protective measures are initially suggested generically for applications with specific characteristics. Then, based on these generic suggestions, protective concepts for two concrete applications developed as part of the SENSIBLE-KI project, “SeamlessMe” and “Self-ID,” are presented in Chapter 4. Reference Architectures.

The proposed measures and protective concepts should be considered in addition to general system and application security procedures (e.g., TLS encryption for network communication), which are not within the scope of this collaborative project.

2. Considered Measures

The measures considered here fall into either the realm of software-based protective measures or Trusted Computing procedures. In this section, all considered measures, along with their protective effects and other notable effects, are listed for reference.

2.1 Software-Based Measures

MeasureProtective EffectComment
adversarial trainingrobustness
  • reduced accuracy
  • adverse impact on privacy
anomaly detectionrobustness
  • increased latency and additional communication
  • additional hardware/devices or storage space required
data sanitization (input data)robustness
data sanitization (output data)privacy
  • reduced accuracy and fairness
differential privacyprivacy
  • reduced accuracy and fairness
  • adverse impact on robustness

2.2 Trusted Computing

ProcedureProtective EffectComment
remote/device attestationrobustness
  • technical requirements on the device
data attestation (e.g., external sensor data)robustness
  • high latency
  • requires specific setup with SE for input sensor data
model signaturerobustness
  • requires its own trusted application
  • increased latency at application startup
encryption of output dataprivacy
  • additional latency compared to non-TC-based encryption
inference execution in TEE (trusted execution environment)confidentiality, privacy, robustness
  • very high implementation hurdles
  • requires its own trusted application
  • significant performance impact depending on implementation
  • suitable only for embedded applications

3. Protective Measures Based on Application Characteristics

In the following, application scenarios are differentiated into two abstract groups based on the architectural features of AI systems: network-based applications (such as P2P networks or the classical server-client architecture) and end-node-only applications – architectures limited to an end device. Network-based architectures require regular data exchange or, in the case of OTA updates, model updates over a network. In contrast, end-node-only architectures are restricted to an end device, and communication with another device is limited to information not related to the AI component of the application.

As these two architectural types mainly differ in the need for network-based exchange, general measures for AI applications with specific characteristics are recommended. For a specific application, the intersection of all proposed protective measures for that application can then be determined based on its characteristics.

It should be noted that software-based protective measures for privacy and robustness tend to negatively influence each other. This means that software-based privacy protection adversely affects robustness and vice versa. Therefore, it is necessary to assess on a case-by-case basis which protection goal is more important or whether adequate protection can be achieved solely through hardware-based measures. Furthermore, all protective measures that require the execution of custom code within a Trusted Execution Environment (TEE) are currently not feasible for mobile applications. This limitation arises from the fact that any applications to be executed in a TEE must be compiled together with its code. Therefore, as of today, only a few, mostly manufacturer-specific system apps can execute code within a user’s mobile device TEE.

3.1 Measures Based on the Source of Input Data:

When using trusted input data to improve robustness and overall accuracy, Data Sanitization is recommended to clean or remove erroneous or incomplete data and normalize all data, if necessary.

For untrusted input data sources, the additional use of Adversarial Training is recommended to strengthen system robustness. Additionally, if the necessary resources are available, Anomaly Detection should be applied to detect manipulated training and inference data to protect against Data Poisoning or Adversarial Attacks. This is particularly relevant for systems with significant potential for harm but is desirable in all cases.

When using external sensor data on a system with significant potential for harm, which is transmitted via an untrusted medium (e.g., but not limited to, the network), a Secure Element (SE) can be used to perform data attestation directly when reading from the sensor. In addition, attacks on input data, as described earlier, should be intercepted through Adversarial Training and Anomaly Detection.

3.2 Measures Based on Personalization and Confidentiality

When processing personal data or sensitive data in general, the privacy of the model should be enhanced. This can be achieved, for example, by using Differential Privacy during training and Data Sanitization of sensitive output data.

In cases of significant potential for harm and/or direct personal data involvement and/or high confidentiality of the model, execution of inference within a Trusted Execution Environment (TEE) should be considered. However, the specific implementation of this protective measure may involve some adverse effects (such as increased latency, power consumption, and technical hurdles). Furthermore, it must be ensured that the protected data never leaves the TEE unencrypted. If this cannot be guaranteed, the measure is no longer suitable for privacy protection and is only suitable for enhancing model robustness and confidentiality when executed on the end device.

3.3 Specific Measures for Network-Based Architectures

In a server-client architecture or other network-based architectures, data and/or model updates are transmitted over the Internet. In this architecture, depending on the level of protection needed, it is advisable to take additional measures against manipulations during transmission, in addition to general application security. All measures mentioned here can generally be implemented without using Trusted Computing but may be more vulnerable, although they also result in lower latencies. Therefore, after selecting the measures, it is advisable to reconsider when Trusted Computing is beneficial and appropriate. Additionally, in the case of P2P network architectures, distributing keys, such as for signatures, may be more complex than in a client-server architecture. In P2P architectures, trust must be extended to multiple devices, not just to a single device (the server). Therefore, the corresponding concepts must be taken into account when distributing keys for data integrity and authenticity verification.

In the case of a system with significant damage potential, a device attestation by means of a TEE can enable the receiver (usually the server) to verify the integrity of the target device’s software. Analogously, the data can be attested, e.g., to prove that the data were produced by the latest version of the AI model. This could further improve the application’s robustness.

In the case of transmitting model updates over the network, it should be ensured that the recipient obtained the correct model by applying a model signature. The signature is created by the sender and is verified at the end-node before the model’s execution. This measure should be taken in the course of the general application security, but in the case of significant damage potential, the signature and attestation should be done by means of trusted computing.

In the case of transmitting data with personal reference, an encryption is mandatory. Depending on the sensitivity of the data, the encryption can be enhanced by incorporating a secure element (SE).

4. Reference Architectures

4.1 SeamlessMe

Description
The Machine Learning algorithm used in the “SeamlessMe” demonstrator is a One Class Classification algorithm extended by a Novelty Detection methodology. A generic classifier is loaded onto the mobile device. Subsequently, the model learns locally based on the user data collected on the device (User Specific Classifier). The trained model is then used to generate a Trust Level (confidence score). The calculation of the user’s trust level takes place locally on the smartphone without communication with an external server. Thus, both training and inference occur directly on the end device.

“SeamlessMe” is primarily implemented on smartphones. During the research and development phase, various wearables such as smartwatches were also tested. However, these devices currently have limited functionality and are not yet suitable for the application.

The biometric data processed by “SeamlessMe” is highly sensitive since it can be used for direct person identification. In a medical context, gait profiles, for example, can serve as indicators for certain diseases, further increasing the sensitivity of this data. However, the model architecture does not need protection as it is already publicly known.

Proposed Measures
Based on the associations of protective measures with characteristic properties of the application outlined in Chapter 3. Protective Measures According to Application Characteristics, the following protective measures are to be implemented:

The personally identifiable nature of the processed biometric data leads to an increased privacy protection requirement for the application. However, since it is a mobile application, the use of custom Trusted Applications is not possible, and therefore, execution in a Trusted Execution Environment (TEE) is not an option. In general, on mobile devices, access to Trusted Computing solutions (TEE/SE) present in the device is limited to the functions provided by the Android API. Therefore, to enhance the privacy of the application, only the use of the hardware-backed keystore or StrongBox, or the key management of the Secure Enclave in iOS can be implemented to secure the collected data. The use of Differential Privacy is not suitable in this case because it would result in a significant loss of prediction quality. This is because the training data all come from the same person, and therefore, much stronger noise would be needed for practical privacy protection than usual.

Since training is unsupervised on the mobile device and thus performed with untrusted data, Anomaly Detection on the input data is used to increase the model’s robustness. SeamlessMe already employs the approach of Anomaly Detection as part of the authentication algorithm. The application is designed to recognize gait sequences that do not match the previously trained samples. If this is the case, the person is not considered authenticated. Also, the data collected at that time is not used for model training and therefore no longer poses a threat to the model.

In addition, Data Sanitization of input data is performed to remove unusable data. The application already implicitly implements this measure since only data that corresponds to the walking pattern of a human is used for gait authentication - and thus with the model.

Adversarial Training is not suitable for this application because it is assumed that attackers cannot manipulate sensor data within the end device. If they could, it would imply that they could compromise the entire AI application, making the manipulation of sensor data unnecessary from the attackers’ perspective. Instead, it is assumed that attackers can only manipulate analog input data by impersonating a authorized person for that individual, i.e., mimicking their gait profile using the smartphone. To defend against such an analog attack, a high overall prediction quality of the application must be ensured.

4.2 Self-ID

Description
The “Self-ID” demonstrator application employs a binary classifier (Self-image vs. Non-self-image) trained on a population. This classifier learns to distinguish which class of face images the user is currently viewing based on eye-tracking data. The training process takes place offline and is based on previously collected training data. If personal enrollment is necessary, it will occur on the end device, and the trained model will subsequently be transmitted to the server. The inference takes place online on the server, with the eye-tracking data being transmitted from the client to the server. The server evaluates the result and takes appropriate actions if necessary.

In the initial stage, for accessibility reasons, a desktop client is used via a plugin or extension of an existing video conferencing system. This desktop prototype serves to gather initial experience with the technology. In a later stage, a mobile application will also be implemented on smartphones (Android and iOS). However, as of the current state of technology, eye-tracking technology is not yet available in sufficient quality on these platforms.

The eye tracker data contains biometric information and serves to validate the user’s identity. Therefore, this data is sensitive and falls under the General Data Protection Regulation (GDPR) as a form of personally identifiable data. Additionally, the data may have medical implications, such as indications of disease symptoms. The model ensures the security of the video conferencing system and, accordingly, has increased robustness requirements depending on the confidentiality of the conversation content. In the target system, an attack on the availability of video conferences (generating false positives) or successful identity theft (generating false negatives) can be carried out by manipulating the model or inference data. Furthermore, the model can be misused to identify individuals in systems where this is not permitted according to the intended use. The model may potentially reveal insights into the biometric data with which it was trained. In the finished product, the model, which will be trained with internal data, is the intellectual property of the Bundesdruckerei and should therefore be kept confidential.

Proposed Measures
Based on the associations of protective measures with characteristic properties of the application outlined in Chapter 3. Protective Measures According to Application Characteristics, the following protective measures are to be implemented:

The confidentiality to be achieved for the model at the time of enrollment on the end device can only be strengthened through general application security procedures (e.g., obfuscation) because access to a Trusted Execution Environment (TEE) is not possible on either desktop or mobile devices. The confidentiality of the model on the server is ensured through general system security measures or available server-side Trusted Computing methods. The consideration of these measures is beyond the scope of this document.

For the intended implementation on mobile devices to protect the privacy of personal data, the hardware-backed keystore or StrongBox, or the Secure Enclave, should be used to secure any data stored on the end device and, if applicable, for data encryption before transmission to the server. In the case of the desktop prototype, the use of the available encryption functions of the device/operating system is recommended instead.

The more complex the output of an ML model, the more effective privacy attacks on the model’s training data can be (see Shokri et al.: “Membership Inference Attacks Against Machine Learning Models,” IEEE Symposium on Security and Privacy, 2017). Therefore, the personal data of the application should be additionally protected by data sanitization of the output data if the classifier used not only outputs the class but also the confidence. Furthermore, Differential Privacy can be applied during the training of the classifier to obfuscate information about individual training data.

The increased potential for harm necessitates the application’s robustness against manipulated input data, which can be improved through general system security by implementing client authentication. In the mobile client, this can be reinforced by using the hardware-backed keystore or StrongBox. Additionally, the use of anomaly detection and adversarial training is required since it cannot be ensured that no manipulated input data will be processed. Anomaly detection is used to identify and filter out manipulated input data before the model processes it, while adversarial training enhances the model’s robustness against manipulated input data.

Current methods for privacy protection often degrade robustness, and vice versa. Moreover, methods for privacy and robustness protection typically reduce the prediction quality of the model. Therefore, careful consideration is needed regarding the acceptable trade-offs between privacy, robustness, and overall prediction quality. For this application, it appears best to prioritize robustness over privacy protection within the model. Since the model itself is not executed on an end device, only black-box attacks on privacy are possible, which are significantly weaker than white-box attacks. Therefore, no further measures for privacy protection are proposed here in favor of software-based robustness protection. It is assumed that general system security measures make it difficult for attackers to gain access to the model on the server.

5. Conclusion

In this document, selected measures that were evaluated in the collaborative project SENSIBLE-KI are proposed for general use cases and for two specific applications - the prototypes to be developed in the project. The more specific the use case, the more the selection of measures to be sensibly employed was narrowed down, as both the practicality in the specific application and the interaction between different measures with varying protective effects need to be considered. In particular, hardware-based Trusted Computing methods can only be used to a limited extent in the protection concepts of the prototypes, as access to them is highly restricted in mobile applications. Software-based measures do not face such limitations but are more susceptible to negative interactions, so the use of two negatively influencing methods must be carefully weighed against each other on a case-by-case basis.

8 - Best Practices

Best Practice Recommendations

Description of Security Goals

This chapter briefly describes the security goals mentioned in this document.

Robustness

The robustness of an AI-Model consists of two factors: The integrity and the availability of the model. Integrity of AI models means the models make the right predictions for a variety of data. This can be impaired by influencing the model-inputs, -outputs and/or parameters. The availability in AI models ensures that the model fulfills the functionality it is used for. If an attackers is targeting the availability of a model he makes the system decline harmless instances in which case the system can’t work properly anymore. If the output of an AI model is integrated into the function of the system this can be seen as a denial-of-service attack.

Privacy

Privacy in AI applications means the protecting the privacy of training data used to train the AI model. The models should withhold all private information. In case of an attack on a model’s privacy the attacker may gain information regarding the - maybe sensitive - training data. This can lead to serious consequences for the privacy of the involved parties. An important aspect of privacy is anonymity and its disctinction to pseudonymity. Anonymity can be defined as protections against general identification and pseudonymity as protection against identification of the name. This implies, that in order to achieve anonymity two picutures should not be correlated with each other. Especially during Training of AI-based identification with biometrics, anonymity is unachiveable and pseudonymity is the main goal. However pseudonymity could be circumvented by correlating multiple databases in so called “Linkage Attacks”.

Confidentiality

Confidentiality describes a state in which the internal properties and confidential information on the trained model remain hidden from a potential attacker. An attack on the confidentiality may allow an attacker to gather sensitive and confidential information regarding the trained AI model, its properties, structure and parameters. With this information, the attacker could be able to steal the intellectual properties, manipulate or attack the privacy of the training data.

Prototypes

Two prototypes were developed during the projects runtime. The application SeamlessMe authenticates a person based on their gait pattern. Self-ID verifies the identity of a user during a video call by analysing eyetracker-data in response to their displayed image. The security requirements of these prototypes were identified and based on the requirements project related security measures were recommended. Recommendations were implemented and evaluated.

Gait-Authentication (SeamlessMe)

The machine learning algorithm used in the “SeamlessMe” demonstrator is a one-class classification, which is enhanced by an outlier detection method (novelty detection). A generic classifier is loaded onto the mobile device. The model then learns locally based on the user data collected on the device (User Specific Classifier). The trained model is then used to generate a trust level (confidence value). The user’s trust level is calculated locally on the smartphone and without communication with an external server. Both the training and the inference therefore take place directly on the end device.

Self identification (Self-ID)

In the application of the demonstrator “Self-ID”, a binary classifier (self-image vs. external image) is trained on a population. Using eye-tracking data, this classifier learns to distinguish which class of facial images the user is currently seeing. The training process takes place in advance - offline - and on the basis of previously collected training data. If personal enrollment is necessary, it takes place on the end device and the trained model is then transferred to the server. The inference runs online on the server. The eye-tracking data is transferred from the client to the server which evaluates the result and initiates appropriate measures if necessary.

Security Measures

Adversarial Training

Adversarial retraining is a protective measure to strengthen the robustness of a machine learning model and a defense technique against so-called Adversarial Examples (AE). AEs are manipulated inputs for neural networks that are intended to lead to an incorrect prediction. The manipulation is so subtle that it is barely recognizable to the human observer. In adversarial retraining, such AEs are included in the training data of the model in order to increase its integrity.

Scenario

The proposed security measure was tested using the Self-ID demonstrator.

In the selected attack scenario, subtle changes to the input data of the model led to incorrect outputs. This reduced the reliability and confidence of the machine learning system. Neural networks were not used.

In the chosen scenario the targeted manipulation of input data was carried out by an “internal attacker” . More complex evasion attacks, such as the generation of AEs by Generative Adversarial Networks (GANs), were not used. For the example attack, a sufficiently strong bias had to be found first so that a false prediction would result.

The designed attack scenario had various prerequisites: Access to the input data

In the chosen example, the type and exact structure of the input data were known, thus relatively small changes of the data were sufficient to manipulate the outcome.

Insight into the predictions

For the example, the degree of change in the input data at which a different label was predicted was recorded. Within the test environment, it was thus possible to trace when the values exceeded a threshold value.

Arbitrary number of predictions

The number of predictions made in the test environment was unlimited.

Experiences

In the selected example, an “internal” attacker who had access to the entire project and the results of the classification was simulated . Without knowledge of the effects of the AEs, these would not have changed the prediction result in a foreseeable way. Furthermore, knowledge of the used features is required, which is not necessarily available to the attacker.

Experience has shown that a change without sufficient knowledge cannot lead to the desired result. Depending on the system architecture, the attacker may not be able to understand whether his AEs have actually triggered a change in the prediction.

Implementation help

Eine Vielzahl an Beispielen und Implementierungsmöglichkeit finden sich in folgender Bibliothek: Adversarial Robustness Toolbox1

Code-Snippets “Tips & Tricks” A large number of examples and implementation options can be found in the following library: Adversarial Robustness Toolbox1

Code snippets Manipulate input data

    perturbation_sample = sample_array + perturbation_strength

View effects and compare predictions

    original_prediction = classifier.predict(original_sample_df)
    perturbed_prediction = classifier.predict(perturbation_sample)

Anomaly Detection

Anomaly detection is used to strengthen the robustness of a machine learning system. By using anomaly detection, critical inputs (such as AEs) can be filtered to remove unusually deviant data inputs that could affect the integrity of the model.

Scenario

The proposed security measure was tested using the SeamlessMe demonstrator. The aim is to find the smallest possible boundary that surrounds the normal data points, while at the same time outliers lie outside this boundary. SeamlessMe already uses the anomaly detection procedure indirectly as an authentication algorithm:

SeamlessMe recognizes gait sequences that do not match the previously trained samples. If this is the case, the person is not considered authenticated. Likewise, the data recorded at the time (non-authenticated) is not used to train the model. Only if the algorithm is certain that the data matches the trained person the data is used for training.

A One-Class Support Vector Machine was used to detect outliers. A one-class SVM is a machine learning algorithm that is able to recognize outliers in a data set. In the selected example, only one class of data is available. In the demonstrator, this class is the person carrying the cell phone. The deviating data or outliers are data that point to a different person and therefore belong to a different class.

The One-Class SVM constructs a decision boundary around the “normal” data points so that an area is delimited. All data points outside this area are considered outliers. An illustration of this distinction is shown below:

Figure 1: One-Class SVM 2

Anomaly detection was proposed for Self-ID, but was not implemented because the effort required was too high for a research prototype. Data cleansing was sufficient for data acquisition and anomaly detection was not necessary. The measure should be reconsidered for the final product.

Implementation help

“Tips & Tricks” Used SVM parameters:

  • Kernel: Gaussian Radial Basis Function (RBF)

  • Kernel functions: “Polynomial” and “Sigmoid” not used

The gamma describes the kernel coefficient. The gamma parameter refers to $ \frac{1}{2*\sigma^2} $ in the following formula: $ K(x_i,x_j) = exp(-\frac{| x_i - x_j |^2}{2*\sigma^2}) $

Code-Snippet (Python: Initialization One-Class-SVM)

    from sklearn.svm import OneClassSVM

    OneClassSVM.__init__(self, kernel='rbf', degree=3, gamma='scale')

Data-Sanitization

Data sanitization of input data is a measure to improve robustness by e.g, cleansing or removing incomplete data in order to bring all data into the appropriate form.

Data sanitization of output data, on the other hand, mitigates privacy attacks. Sensitive output data is filtered before it is emitted. Data is considered sensitive if it allows conclusions about the model’s training data. An example of this in the case of classification would be the confidence values per class. These values can be used by attackers to carry out stronger privacy attacks. Instead of giving the attacker access to every value, only the class with the highest confidence value is emitted.

Scenario

Unusable or incorrect data is filtered. For the Self-ID demonstrator, incorrect images or images containing a blink are therefore removed. The filtering also includes the removal of unnecessary data from the eye tracker data. The data is normalized and scaled. Part of the output data are the confidence values. These values are only shown in the prototype phase in the video client. In the finished product, the strength of confidence in the video client should be represented in a more generic way for the video call participant in question.

With SeamlessMe, activity recognition of the authentication model is performed upstream. Only data that corresponds to the walking pattern of a person is used for gait authentication. This means that data sanitization of the input data is already carried out.

Experiences

Data Sanitization of Input-Data

One experience that has been made involves approximating the correct threshold value for filtering. It should be noted that no training data is removed that still holds enough significance, eventhough a smaller part is invalid. To enable comparability within the Self-ID demonstrator, a minimum number of data points is calculated before an image is presented and a minimum number of data points is calculated after the image is presented. Data points that fall outside this window are removed so that all trials are the same length. The minimum number of data points describes the shortest period of time that does not distort the image. The differences between the recordings result from the latency of the eye tracker.

Data Sanitization of Output-Data

As described in the section “Scenario”, a log of the model results is required to interpret the results of the model creation. Each run with different parameters was logged and stored. The log files describe the changes in the performance metrics caused by the change in input parameters. This logging included the confidence values of all classes. An overview of these values was necessary for the optimization of the models. These values are no longer supplied for the final application, so that no conclusions can be drawn about the model’s training data. Feedback between the user and the application is useful if a confidence level is to be returned. An example of this is that the recognition of classes should be marked within the demonstrators.

Implementation help

“Tips & Tricks”

The research project worked with time series. One type of data cleansing is the normalization of data. In this subject area, the source Normalization and Bias in Time Series Data3 by Aishwarya Asesh has proven to be helpful

Code-Snippet

    def normalize(x):
        minimum = np.min(x)
        maximum = np.max(x)
        y = (x - minimum) / (maximum - minimum)

        return np.array(y)

Differential Privacy

Differential privacy (DP) is a popular metric used to measure the impact of individual data on the outcome of data processing. This also corresponds to the privacy loss of the individuals to whom the data belongs. The loss of privacy is limited by the addition of noise in the data processing. DP originally comes from the field of databases, but has also been used for privacy-preserving ML for several years.

Scenario

Differential Privacy was tested within the Self-ID demonstrator to determine the impact of noise on prediction quality.

A publicly available library was used to examine the effects of differential privacy (DP): Diffprivlib4 is a general library for experimenting, investigating and developing applications in the field of differential privacy.

Experiences

The influence of differential privacy led to a reduction in the values of the metrics for all classes considered. Precision, Recall and F1-Score decreased for all classes. The use of differential privacy can be compared to the addition of noise, so a trade-off must be made between prediction quality and protection. Too much noise can affect the accuracy of predictions, while too little noise jeopardizes privacy.

Figure 2: Differences in metrics

The noise of input data is viewed critically. Reference is made to the paper “Jekyll and Hyde: On The Double-Faced Nature of Smart-Phone Sensor Noise Injection “. It observes the introduction of noise into smartphone sensor data used for gait authentication.

The influence of “noise injection on the average F-scores obtained for gait-based authentication” can be seen in the following graph from the paper:

Figure 3: Influence of Differential Privacy

It can be shown how the F-score decreases significantly when more noise is added. High accuracy is important for the functionality of SeamlessMe so that a person can be recognized with a high probability.

Implementation help

“Tips & Tricks” The use of matplot is recommended to illustrate the results, as shown in the diagrams above.

Code-Snippet Example use of the Differential Privacy Library

    from diffprivlib.models import GaussianNB

    clf = GaussianNB()
    clf.fit(X_train, y_train)

    clf.predict(X_test)

Device attestation

In device attestation, the integrity of a device (e.g. a smartphone) is confirmed by a cryptographic signature of a trusted component - e.g. Trusted Execution Environment (TEE), Secure Element (SE). The technical mechanisms used for this differ depending on the implementation, but the general aim is to confirm that the device is in a certain state that is not considered compromised. Device attestatioinferencee of a trusted component is also used for data attestation. This time, however, the signature is calculated using a specific date and attests that it has certain properties (e.g. that the data originates from a specific hardware). This form of attestation is also used to strengthen robustness.

This protective measure was not implemented in the demonstrators of the SENSIBLE-KI joint project, as no data is collected from external sensor nodes or mobile devices that could be attested, and the measure can also only be implemented by specific hardware that could not be used sensibly for the development of the demonstrators. However, the evaluations of the measure as part of AP2 showed an increased latency in data collection, which could make the measure impractical depending on the type of data collection of the ML application.

Model-Signatur

When updating models via the network, the model should be signed to ensure that the receiver has received the correct model. The signature is created by the sender and checked on the end device before execution. For critical applications in particular, it makes sense to perform the attestation and signature using a trustworthy component. This measure strengthens the robustness of the ML system.

This measure was not implemented in the demonstrators of the SENSIBLE-KI joint project, as there are no plans to transfer model updates via the network for either of the two applications. The evaluations of the measure carried out earlier in the project revealed increased latency when starting the application, the length of which depended heavily on the hardware used for signature and verification. It should also be noted that implementing this measure in a TEE would require the implementation and execution of a separate trusted application, which is currently not easily possible on standard mobile devices, for example.

Encryption of Output Data

One measure to improve privacy is to encrypt personalized output data before transmission or storage. It is advisable to use a trusted component for encryption, especially if the data is highly personal, as this offers better protection for the key material used.

Scenario

The encryption was tested within the SeamlessMe demonstrator. Save key material

iOs:

The key material is stored in the Secure Enclave. The Secure Enclave is a secure hardware component. This isolated area in the processor chip serves as security and protection for sensitive data.

Android:

The key material is stored in the hardware-backed keystore. The hardware-backed keystore is a physical hardware component that is responsible for the secure storage of cryptographic keys and sensitive data. Not all devices support the cryptographic algorithms required to secure communication. This will result in additional work during implementation.

Experiences

Correct choice of encryption algorithms:

The choice of a robust encryption algorithm such as AES or RSA depends on the desired level of security.

Key length and key management:

It is recommended to choose an appropriate key length, as a longer key length provides a higher level of security. The keys used should also be managed securely and only be accessible to authorized persons. Regular rotation of the keys is also recommended.

No loss of performance

The relatively small amount of data in the project did not lead to any loss of performance.

Implementation help

The documentation on the Android Keystore System5 and the iOS Secure Enclave6 is recommended for further research and for current examples.

Inference in a TEE

One way to achieve confidentiality, robustness and privacy at the same time is to execute an ML application within a Trusted Execution Environment as a so-called Trusted Application. In this way, the trained model would not come into contact with potentially malicious third-party applications and would be protected even if the operating system were compromised. Unfortunately, this measure cannot currently be implemented in practice without high technical hurdles and significant performance losses.

Accordingly, the protective measure was not implemented in the demonstrators of the SENSIBLE-KI joint project.