阅读笔记 | Mistify: Automating DNN Model Porting for On-Device Inference at the Edge

Summary

Deep neural networks (DNNs) are increasingly used in edge computing applications, where running inference on the edge provides benefits like lower latency, better privacy, and reliability compared to the cloud. However, this requires porting cloud-designed DNN models to diverse edge devices with varying hardware capabilities. Manually porting models does not scale due to the large search space and effort required per deployment target. Existing algorithms like MorphNet need tedious model code annotations. Meanwhile, retraining on private edge data raises concerns over data privacy versus model quality. The tight coupling of model design and deployment also requires expertise in both deep learning and systems.

To address these challenges, the paper designs an intermediate middleware Mistify that decouples traditional DNN model design and deployment, and automates porting DNNs from cloud to edge. The main design is as follow.

• The paper designs an intermediate middleware Mistify in computer system architecture, which decouples the traditional DNN model design and deployment, and highly automates the porting process of DNNs from cloud to edge devices.

• Replacing repeat manual modifications with concise API interfaces and configuration files.

• Providing edge side with dynamic monitoring and multi-branch model switching and re-adaptation mechanisms to cope with the fluctuating characteristics at the edge runtime.

• Utilizing the inherent correlations among models under different configurations to achieve collective adaptation, reducing the adaptation time for multiple DNN variants.

• Realizing privacy-aware collaborative fine-tuning techniques using knowledge distillation at the edge side, which makes full use of private data at the edge to prevent overfitting while protecting privacy.

The paper implements Mistify and conducts multiple evaluations on object recognition and text QA datasets. The results show it reduces porting time by 10 times and manual work by several orders of magnitude.

Strengths

• Uses interfaces and configuration files for adaptation, which is non-invasive to source code and highly scalable

• Achieves full automation and loose coupling of the end-to-end DNN porting process

• Considers large-scale adaptation scenarios, greatly reducing adaptation time

• Makes full use of private data for fine-tuning on the edge side while protecting data privacy

Weaknesses

• Evaluation metrics are relatively singular across multiple evaluation scenarios, narrow evaluation coverage

• Does not discuss pre-trained model paradigms to address the need for training from scratch on the edge

• Communication security and efficiency for collaborative fine-tuning on the edge side still lack sufficient consideration

Comments

This paper accurately locates and resolves various problems and needs in the DNN model porting process, and cleverly utilizes object-oriented design thinking to enhance the system’s extensibility. Although there are some inadequate considerations in the design and insufficient evaluation, it still provides a good end-to-end DNN porting framework that is worth learning from.