Authors: Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Jiancheng Liu, Konstantinos Parasyris, Yihua Zhang, Zheng Zhang, Bhavya Kailkhura, Sijia Liu

Published on: October 03, 2023

Impact Score: 8.15

Arxiv code: Arxiv:2310.02025

Summary

• What is new: Introduces DeepZero, scaling ZO optimization for DNN training without performance loss, a first in the field.
• Why this is important: Scalability of ZO optimization for deep learning has been limited, preventing its application in training DNNs effectively.
• What the research proposes: DeepZero, a framework with innovations like coordinatewise gradient estimation, sparsity-induced training, and methods for feature reuse and forward parallelization.
• Results: Achieves SOTA accuracy on training ResNet-20 on CIFAR-10, comparable to FO training, and demonstrates utility in certified adversarial defense and error correction with 10-20% improvement.

Technical Details

Technological frameworks used: DeepZero

Models used: ResNet-20

Data used: CIFAR-10

Potential Impact

Could impact companies in digital security, AI development platforms, and industries involved in complex simulations requiring DNNs.

Want to implement this idea in a business?

We have generated a startup concept here: DeepSparseOpt.