2022-06-28-amid22a.md

title	booktitle	abstract	layout	series	publisher	issn	id	month	tex_title	firstpage	lastpage	page	order	cycles	bibtex_author	author	date	address	container-title	volume	genre	issued	pdf	extras
Public Data-Assisted Mirror Descent for Private Model Training	Proceedings of the 39th International Conference on Machine Learning	In this paper, we revisit the problem of using in-distribution public data to improve the privacy/utility trade-offs for differentially private (DP) model training. (Here, public data refers to auxiliary data sets that have no privacy concerns.) We design a natural variant of DP mirror descent, where the DP gradients of the private/sensitive data act as the linear term, and the loss generated by the public data as the mirror map. We show that, for linear regression with feature vectors drawn from a non-isotropic sub-Gaussian distribution, our algorithm, PDA-DPMD (a variant of mirror descent), provides population risk guarantees that are asymptotically better than the best known guarantees under DP (without having access to public data), when the number of public data samples is sufficiently large. We further show that our algorithm has natural “noise stability” properties that control the variance due to noise added to ensure DP. We demonstrate the efficacy of our algorithm by showing privacy/utility trade-offs on four benchmark datasets (StackOverflow, WikiText-2, CIFAR-10, and EMNIST). We show that our algorithm not only significantly improves over traditional DP-SGD, which does not have access to public data, but to our knowledge is the first to improve over DP-SGD on models that have been pre-trained with public data.	inproceedings	Proceedings of Machine Learning Research	PMLR	2640-3498	amid22a	0	Public Data-Assisted Mirror Descent for Private Model Training	517	535	517-535	517	false	Amid, Ehsan and Ganesh, Arun and Mathews, Rajiv and Ramaswamy, Swaroop and Song, Shuang and Steinke, Thomas and Steinke, Thomas and Suriyakumar, Vinith M and Thakkar, Om and Thakurta, Abhradeep	given family Ehsan Amid given family Arun Ganesh given family Rajiv Mathews given family Swaroop Ramaswamy given family Shuang Song given family Thomas Steinke given family Thomas Steinke given family Vinith M Suriyakumar given family Om Thakkar given family Abhradeep Thakurta	2022-06-28		Proceedings of the 39th International Conference on Machine Learning	162	inproceedings	date-parts 2022 6 28	https://proceedings.mlr.press/v162/amid22a/amid22a.pdf