Awesome Mobile On-Device AI Security

SoK: Attack and Defense Landscape of Mobile On-device AI Systems

Yujin Huang, Xin Zheng, Xingliang Yuan, Kwok-Yan Lam

Mobile on-device AI systems execute AI models locally through ML frameworks such as LiteRT/TFLite, Core ML, ExecuTorch, ONNX, and hardware-backed accelerators. This repo tracks the security research needed to understand and protect such systems, as the local storage of on-device models introduces new security risks.

Overview of a Mobile On-Device AI system

Contents

• Reading roadmap
• Taxonomy at a glance
• Cross-pillar Security Analysis
• Attacks on MoAI systems
  • Adversarial Attacks
    • Model Similarity Exploitation
    • Gradient Reconstruction
    • Preprocessing Manipulation
  • Backdoor Attacks
    • Payload Injection
    • Model Quantization
    • Image Steganography
  • Adversarial Weight Attacks
  • Model Stealing Attacks
    • Static Analysis
    • Dynamic Analysis
    • Side Channel
  • Energy-latency Attacks
• Defenses for MoAI systems
  • Model Obfuscation
    • Software-level Concealment
    • Hardware-level Concealment
  • Model Authorization
  • Trusted Execution Environments
    • Monolithic Execution
    • Partitioned Execution
    • Obfuscated Offloading
  • Model Watermarking
• Open problems
• Emerging directions

Reading roadmap

New to MoAI security? Start here:

1. Understand the ecosystem. Read empirical studies on deep learning apps and on-device models in Android/iOS apps.
2. Learn the core risk. Study model extraction and model protection papers, because local model residency is the central security shift in MoAI systems.
3. Understand the attack surfaces. Study how MoAI attacks arise across input interfaces, model artifacts, runtime execution, and hardware-backed environments.
4. Connect defenses to the surfaces. Examine how MoAI defenses protect these surfaces across pre-deployment, runtime execution, and post-deployment phases.
5. Look forward. Explore new security challenges in on-device training, on-device GenAI, and agentic MoAI systems.

•  A First Look at Deep Learning Apps on Smartphones
•  A First Look at On-device Models in iOS Apps

•  Mind Your Weight(s): A Large-scale Study on Insufficient ML Model Protection in Mobile Apps

•  Robustness of On-device Models: Adversarial Attack to Deep Learning Models on Android Apps
•  DeepPayload: Black-box Backdoor Attack on Deep Learning Models through Neural Payload Injection
•  Typhon Unleashed: Practical Adversarial Weight Attacks Against On-Device Deep Learning Models
•  Energy-Latency Attacks to On-Device Neural Networks via Sponge Poisoning

•  ModelObfuscator: Obfuscating Model Information to Protect Deployed ML-based Systems
•  ShadowNet: A Secure and Efficient On-device Model Inference System
•  THEMIS: Towards Practical IP Protection for Post-Deployment On-Device DL Models

Taxonomy at a glance

Cross-pillar Security Analysis

Attacks on MoAI systems

Model Similarity Exploitation

• Robustness of On-device Models: Adversarial Attack to Deep Learning Models on Android Apps [Code]
  _{IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP 2021)}

• Smart App Attack: Hacking Deep Learning Models in Android Apps [Code]
  _{IEEE Transactions on Information Forensics and Security (TIFS 2022)}

• Understanding Real-world Threats to Deep Learning Models in Android Apps [Code]
  _{ACM SIGSAC Conference on Computer and Communications Security (CCS 2022)}

• Cheating Your Apps: Black-box Adversarial Attacks on Deep Learning Apps
  _{Journal of Software: Evolution and Process (JSEP 2024)}

• A First Look at On-device Models in iOS Apps [Code]
  _{ACM Transactions on Software Engineering and Methodology (TOSEM 2024)}

Gradient Reconstruction

• Investigating White-Box Attacks for On-Device Models [Code]
  _{IEEE/ACM International Conference on Software Engineering (ICSE 2024)}

• TIM: Enabling Large-Scale White-Box Testing on In-App Deep Learning Models [Code]
  _{IEEE Transactions on Information Forensics and Security (TIFS 2024)}

Preprocessing Manipulation

• Beyond the Model: Data Pre-processing Attack to Deep Learning Models in Android Apps [Code]
  _{Secure and Trustworthy Deep Learning Systems Workshop (SecTL 2023)}

Payload Injection

• DeepPayload: Black-box Backdoor Attack on Deep Learning Models through Neural Payload Injection [Code]
  _{IEEE/ACM International Conference on Software Engineering (ICSE 2021)}

• MalModel: Hiding Malicious Payload in Mobile Deep Learning Models with Black-box Backdoor Attack [Code]
  _{Automated Software Engineering (ASEJ 2026)}

Model Quantization

• Quantization Backdoors to Deep Learning Commercial Frameworks [Code]
  _{IEEE Transactions on Dependable and Secure Computing (TDSC 2024)}

Image Steganography

• Stealthy Backdoor Attack to Real-world Models in Android Apps
  _{arXiv preprint (arXiv 2025)}

• Typhon Unleashed: Practical Adversarial Weight Attacks Against On-Device Deep Learning Models
  _{IEEE Transactions on Dependable and Secure Computing (TDSC 2026)}

Static Analysis

• A First Look at Deep Learning Apps on Smartphones [Code]
  _{The World Wide Web Conference (WWW 2019)}

• A First Look at On-device Models in iOS Apps [Code]
  _{ACM Transactions on Software Engineering and Methodology (TOSEM 2023)}

• Mind Your Weight(s): A Large-scale Study on Insufficient Machine Learning Model Protection in Mobile Apps [Code]
  _{USENIX Security Symposium (USENIX Security 2021)}

• REDLC: Learning-Driven Reverse Engineering for Deep Learning Compilers
  _{IEEE International Symposium on Software Reliability Engineering (ISSRE 2024)}

Dynamic Analysis

• Mind Your Weight(s): A Large-scale Study on Insufficient Machine Learning Model Protection in Mobile Apps [Code]
  _{USENIX Security Symposium (USENIX Security 2021)}

• Understanding Real-world Threats to Deep Learning Models in Android Apps [Code]
  _{ACM SIGSAC Conference on Computer and Communications Security (CCS 2022)}

• DeMistify: Identifying On-device Machine Learning Models Stealing and Reuse Vulnerabilities in Mobile Apps [Code]
  _{IEEE/ACM International Conference on Software Engineering (ICSE 2024)}

• Game of Arrows: On the (In-)Security of Weight Obfuscation for On-Device TEE-Shielded LLM Partition Algorithms [Code]
  _{USENIX Security Symposium (USENIX Security 2025)}

Side Channel

• Model Extraction Attack against On-Device Deep Learning with Power Side Channel
  _{IEEE International Symposium on Quality Electronic Design (ISQED 2024)}

• DeepCache: Revisiting Cache Side-Channel Attacks in Deep Neural Networks Executables
  _{ACM SIGSAC Conference on Computer and Communications Security (CCS 2024)}

• Energy-Latency Attacks to On-Device Neural Networks via Sponge Poisoning
  _{Secure and Trustworthy Deep Learning Systems Workshop (SecTL 2023)}

Defenses for MoAI systems

Software-level Concealment

• ModelObfuscator: Obfuscating Model Information to Protect Deployed ML-Based Systems [Code]
  _{ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2023)}

• DynaMO: Protecting Mobile DL Models through Coupling Obfuscated DL Operators [Code]
  _{IEEE/ACM International Conference on Automated Software Engineering (ASE 2024)}

• Model-less Is the Best Model: Generating Pure Code Implementations to Replace On-Device DL Models [Code]
  _{ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2024)}

Hardware-level Concealment

• NNSplitter: An Active Defense Solution for DNN Model via Automated Weight Obfuscation [Code]
  _{International Conference on Machine Learning (ICML 2023)}

• A Novel Obfuscation Method Based on Majority Logic for Preventing Unauthorized Access to Binary Deep Neural Networks
  _{Scientific Reports (Sci. Rep. 2025)}

• Securing On-device Transformer with Hardware Binding and Reversible Obfuscation [Code]
  _{Annual Computer Security Applications Conference (ACSAC 2025)}

• MMGuard: Automatically Protecting On-Device Deep Learning Models in Android Apps [Code]
  _{IEEE Security and Privacy Workshops (SPW 2021)}

• Securing On-device Transformer with Hardware Binding and Reversible Obfuscation [Code]
  _{Annual Computer Security Applications Conference (ACSAC 2025)}

Monolithic Execution

• Offline Model Guard: Secure and Private ML on Mobile Devices
  _{Design, Automation and Test in Europe Conference (DATE 2020)}

• GuardiaNN: Fast and Secure On-Device Inference in TrustZone Using Embedded SRAM and Cryptographic Hardware
  _{ACM/IFIP International Middleware Conference (Middleware 2022)}

• Secure and Efficient Mobile DNN Using Trusted Execution Environments
  _{ACM Asia Conference on Computer and Communications Security (AsiaCCS 2023)}

• T-Slices: Confidential Execution of Deep Learning Inference at the Untrusted Edge with Arm TrustZone
  _{ACM Conference on Data and Application Security and Privacy (CODASPY 2023)}

• LEAP: TrustZone Based Developer-Friendly TEE for Intelligent Mobile Apps
  _{IEEE Transactions on Mobile Computing (TMC 2022)}

• ASGARD: Protecting On-Device Deep Neural Networks with Virtualization-Based Trusted Execution Environments [Code]
  _{Network and Distributed System Security Symposium (NDSS 2025)}

• TZ-LLM: Protecting On-Device Large Language Models with Arm TrustZone [Code]
  _{European Conference on Computer Systems (EuroSys 2026)}

• FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation
  _{arXiv preprint (arXiv 2026)}

Partitioned Execution

• DarkneTZ: Towards Model Privacy at the Edge Using Trusted Execution Environments [Code]
  _{Annual International Conference on Mobile Systems, Applications, and Services (MobiSys 2020)}

• HybridTEE: Secure Mobile DNN Execution Using Hybrid Trusted Execution Environment [Code]
  _{Asian Hardware Oriented Security and Trust Symposium (AsianHOST 2020)}

• SecDeep: Secure and Performant On-Device Deep Learning Inference Framework for Mobile and IoT Devices
  _{International Conference on Internet-of-Things Design and Implementation (IoTDI 2021)}

• ShadowNet: A Secure and Efficient On-Device Model Inference System for Convolutional Neural Networks [Code]
  _{IEEE Symposium on Security and Privacy (S&P 2023)}

• MirrorNet: A TEE-Friendly Framework for Secure On-Device DNN Inference
  _{IEEE/ACM International Conference on Computer-Aided Design (ICCAD 2023)}

• TSQP: Safeguarding Real-Time Inference for Quantization Neural Networks on Edge Devices [Code]
  _{IEEE Symposium on Security and Privacy (S&P 2025)}

• TEESlice: Protecting Sensitive Neural Network Models in Trusted Execution Environments When Attackers Have Pre-Trained Models
  _{ACM Transactions on Software Engineering and Methodology (TOSEM 2025)}

• TensorShield: Safeguarding On-Device Inference by Shielding Critical DNN Tensors with TEE [Code]

Obfuscated Offloading

• GroupCover: A Secure, Efficient and Scalable Inference Framework for On-Device Model Protection Based on TEEs [Code]
  _{International Conference on Machine Learning (ICML 2024)}

• Game of Arrows: On the (In-)Security of Weight Obfuscation for On-Device TEE-Shielded LLM Partition Algorithms [Code]
  _{USENIX Security Symposium (USENIX Security 2025)}

• MirageNet: A Secure, Efficient, and Scalable On-Device Model Protection in Heterogeneous TEE and GPU System
  _{arXiv preprint (arXiv 2026)}

• THEMIS: Towards Practical Intellectual Property Protection for Post-Deployment On-Device Deep Learning Models [Code]
  _{USENIX Security Symposium (USENIX Security 2025)}

Open problems

The following open problems summarize the main research gaps identified in our SoK. We keep the descriptions here high-level for readers using this repository. More technical discussions can be found in the paper.

1. Attack Deployment Practicality.
   Adversarial attacks against on-device models remain hard to realize after deployment because they often require control over model inputs, insertion of adversarial perturbations, or app repackaging to modify preprocessing code. These steps can be impractical or detectable in real end-user deployments.

2. Stealthy Model Modification.
   Backdoor attacks need to find post-deployment entry points beyond standard training-time poisoning because on-device models are typically read-only and inference-only. The key challenge is to introduce hidden malicious behavior without producing observable changes in model artifacts.

3. Precise Weight Localization.
   Adversarial weight attacks expose a parameter-level integrity risk, but practical deployment depends on locating behavior-critical weights in the large parameter search space. This is difficult because attackers often lack gradient guidance and need to preserve benign utility while modifying only selected parameters.

4. Reliable Model Extraction.
   Local model storage does not make model stealing straightforward. Practical extraction still depends on reliable model identification, decryption, and reconstruction in the presence of customized encryption algorithms, nonstandard AI frameworks, and runtime-specific loading behavior.

5. Hardware Heterogeneity.
   Energy-latency attacks depend on how poisoned activation patterns interact with device-specific execution behavior. They may amplify latency and energy consumption on sparsity-sensitive accelerators, but fail to transfer to hardware without sparsity-dependent execution.

6. Executable Equivalence.
   Model obfuscation still needs to preserve the original prediction function during authorized inference. This executable equivalence can expose recoverable runtime states, transformed weights, operator semantics, or structural traces that enable semantic, structural, or parameter recovery.

7. Client-side Enforcement.
   Model authorization binds correct inference to credentials, integrity checks, and packed-weight recovery. However, these checks need to execute inside the mobile stack, making enforcement dependent on client-side code that can be reverse engineered, repackaged, hooked, or instrumented after deployment.

8. TEE Deployment Feasibility.
   TEE defenses require coordinated support across model formats, AI frameworks, operator libraries, delegates, accelerators, and CPU/GPU/NPU isolation interfaces. Current mobile ecosystems still lack widely adopted, developer-transparent TEE-backed inference stacks.

9. Watermark Robustness.
   Model watermarking enables post-deployment ownership verification, but stolen models may be redeployed through framework conversion, encryption, or app-level input-output mediation. These transformations can preserve benign inference while disrupting trigger responses, confidence patterns, or output semantics used for verification.

Emerging directions

Beyond the nine open problems above, our SoK highlights three emerging directions where MOAI security is likely to expand next. These directions move beyond static, inference-only on-device models toward adaptive, generative, and action-oriented MoAI systems. We summarize them here at a high level. The companion paper provides more detailed motivation, threat surfaces, and research challenges.

Current MoAI security research mainly focuses on deployed models that are read-only and inference-only. On-device training changes this assumption by allowing models to be updated locally, which exposes gradients, parameter updates, and user data during the training process. This opens new questions around local fine-tuning, update integrity, training-data exposure, personalization poisoning, and defenses for training-time states on end-user devices.

Existing MoAI security studies are still largely centered on vision-based tasks such as image classification. As LLMs and generative models move onto smartphones, MoAI security must expand to prompt-driven and content-generating systems. Important challenges include prompt injection, jailbreaks, unintended information disclosure, and local context leakage for on-device LLM.

MoAI systems are evolving from passive local inference toward agentic workflows that connect models with sensors, private user data, app contexts, OS services, and cross-app interfaces. This shifts the security focus from protecting model artifacts alone to governing context-to-action chains. Future work should study provenance for mobile context, separation of trusted user intent from untrusted environmental content, task-scoped permissions for tool and API use, confirmation and rollback for sensitive actions, and auditing of agent plans, memory, and actions.