【论文笔记09】Differentially Private Hypothesis Transfer Learning 差分隐私迁移学习模型, ECML&PKDD 2018

系列传送

我的论文笔记频道

【Active Learning】
【论文笔记01】Learning Loss for Active Learning, CVPR 2019
【论文笔记02】Active Learning For Convolutional Neural Networks: A Core-Set Approch, ICLR 2018
【论文笔记03】Variational Adversarial Active Learning, ICCV 2019
【论文笔记04】Ranked Batch-Mode Active Learning，ICCV 2016

【Transfer Learning】
【论文笔记05】Active Transfer Learning, IEEE T CIRC SYST VID 2020
【论文笔记06】Domain-Adversarial Training of Neural Networks, JMLR 2016
【论文笔记10】A unified framework of active transfer learning for cross-system recommendation, AI 2017
【论文笔记14】Transfer Learning via Minimizing the Performance Gap Between Domains, NIPS 2019

【Differential Privacy】
【论文笔记07】A Survey on Differentially Private Machine Learning, IEEE CIM 2020
【论文笔记09】Differentially Private Hypothesis Transfer Learning, ECML&PKDD 2018
【论文笔记11】Deep Domain Adaptation With Differential Privay, IEEE TIFS 2020
【论文笔记12】Differential privacy based on importance weighting, Mach Learn 2013
【论文笔记13】Differentially Private Optimal Transport: Application to Domain Adaptation, IJCAI 2019

【Model inversion attack】
【论文笔记08】Model inversion attacks that exploit confidence information and basic countermeasures, SIGSAC 2015

Differentially Private Hypothesis Transfer Learning

原文传送

1 Abstract

2 Bg & Rw

更多关于差分隐私的背景可以看
差分隐私机器学习综述
模型反转攻击

Differential privacy essentially implies that the existence of any particular data point $s$ in a private data set $S$ cannot be determined by analyzing the output of a differentially private algorithm $\mathcal{M}$ applied on $S$.

Popular approaches to achieving differential privacy

• output perturbation, adding carefully calibrated noises to the parameters of the elarned hypothesis before releasing it.
• distributed privacy-preserving ML, where private data sets are collected by multiple parties.
• • One line of research involves exchanging differentially private information (e.g. gradients) among multiple parties during the iterative hypothesis training process.
• • An alternative line of work focuses on privacy-preserving model aggregation techniques.

The most related works to this paper are focusing on multi-task learning.

One of the key drawbacks of iterative differentially private methods is that privacy risks accumulate with each iteraion. So, according to the composition theorem of differential privacy, there is a limit on how many iterations can be performed on a specific private dataset under a certain level of total privacy budget and this severely affects the utility-privacy trade-off of iterative solutions. 换句话说就是utility会因为privacy的牵制下降。

3 Setting & The proposed methods

I think hypothesis means the same as model/mapping/function.

My understanding of the setting and algorithm:

1. This is a multi-source transfer learning.
2. In each source domain ${\mathcal{D}}^k$, there are i.i.d. labeled samples $S_l^k=\{(x_i^k,y_i^k):1⩽i⩽n_l^k\}$ and unlabeled samples $S_{ul}^k=\{x_j^k:1⩽j⩽n_{ul}^k\}$. $x_i^k,x_j^k\in {\mathbb{R}}^d$ are the feature vectors, $y_i^k$ is the corresponding label.
3. The target domsin ${\mathcal{D}}^T$ is assumed to be a mixture of source distributions and there are abundant unlabeled samples and very few labeled samples drawn i.i.d. from it.
4. The public data $P$ of size $n^P$ is from the target domain ${\mathcal{D}}^T$ and can be accessed by each source domain ${\mathcal{D}}^k,k=1,2,...,K$.
5. In each source domain ${\mathcal{D}}^k$, $S_l^k$ is used to learn a source hypothesis which will be perturbed into ${\theta }_{priv}^k$. Also, each source will compute the differentially private “importance weight”(IW, $w^k$) for each sample in $P$ using ${\mathcal{S}}_{ul}^k$. [Question]
6. ${\theta }_{priv}^k$ and IW are transferred to the target domain.
7. The target will compute its own non-private “importance weight” vector $w^T$. Given $w^k$s and $w^T$, the target further compute the “hypothesis weight”(HW, $w^H$) to assign to each source hypothesis (by solving an optimization problem that minimizes the (KL) divergence between target IW vector and a linear combination of the source IW vectors).
8. Implied assumptions: 1) A data point in $P$ will have large IW if it is similar to the samples in $S_{ul}^k$ and small otherwise. 2) By comparing $w^k$s, we can discover which source domain is similar to the target domain. 3) Hypotheses trained on sources similar to the target should be assigned higher weights in the model aggregation process.
9. Finally, the target will construct an informative Bayesian prior for its logistic regression model using ${\theta }_{priv}^k$ and $w_H$.
10. Remark The purpose of performing logistic regression here is similar to 《Differential privacy based on importance weighting》, that is to estimate the weights $w(x)=N_P/Z\times exp({\beta }_{priv}^{T}x)$. However, a difference between these two is that this paper set the target with label 0 while Zhanglong Ji & Charles Elkan set the source with label 0. Thus, the directions of weighting are opposite.

The algorithm procedures:

Remark

• Step 3 uses DPIW(T, P, $\infty$, ${\lambda }_{IW}$), which means there is no privacy concern, to compute the non-private “importance weights” vector $w^T$.
• Step 5: the feature vector is d-dimensional
• Bayesian logistic regression model

Def DPIW(Private data set, Public data set, $ϵ$, $\lambda$)
Remark Regularized logistic regression + Laplace noise + Weight estimation

Reference

[1] Wang, Yang, Quanquan Gu, and Donald Brown. “Differentially private hypothesis transfer learning.” Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, Cham, 2018.