Some useful transfer learning and domain adaptation codes
It is a waste of time looking for the codes from others. So I collect or reimplement them here in a way that you can easily use. The following are some of the popular transfer learning (domain adaptation) methods in recent years, and I know most of them will be chosen to compare with your own method.
You are welcome to contribute and suggest other methods.
This document contains codes from several aspects: tutorial, theory, traditional methods, and deep methods.
Testing dataset can be found here.
-
Fine-tune using AlexNet and ResNet
-
Fast learn transfer learning:
-
Google's Tensorflow Hub (Tensorflow library released by Google for transfer learning)
- MMD and MK-MMD：Python | Pytorch | Matlab
-
$A$ -distance: Python - CORAL loss: Pytorch
- Several metric learning algorithms: Python
- Wasserstein distance (earch mover's distance):
- Scipy built-in function: scipy.stats.wasserstein_distance
- OpenCV built-in function:
cv.CalcEMD2 - Google's implementation: Tensorflow
- Feature visualization using t-SNE (用t-SNE进行特征可视化)：Python
-
SVM (baseline)
-
TCA (Transfer Component Anaysis, TNN-11) [1]
-
KMM (Kernel Mean Matching, NIPS-06) [67]
-
GFK (Geodesic Flow Kernel, CVPR-12) [2]
-
DA-NBNN (Frustratingly Easy NBNN Domain Adaptation, ICCV-13) [39]
-
JDA (Joint Distribution Adaptation, ICCV-13) [3]
-
TJM (Transfer Joint Matching, CVPR-14) [4]
-
CORAL (CORrelation ALignment, AAAI-15) [5]
-
JGSA (Joint Geometrical and Statistical Alignment, CVPR-17) [6]
-
TrAdaBoost (ICML-07)[8]
-
SA (Subspace Alignment, ICCV-13) [11]
-
BDA (Balanced Distribution Adaptation for Transfer Learning, ICDM-17) [15]
-
MTLF (Metric Transfer Learning, TKDE-17) [16]
-
Open Set Domain Adaptation (ICCV-17) [19]
-
TAISL (When Unsupervised Domain Adaptation Meets Tensor Representations, ICCV-17) [21]
-
STL (Stratified Transfer Learning for Cross-domain Activity Recognition, PerCom-18) [22]
-
LSA (Landmarks-based kernelized subspace alignment for unsupervised domain adaptation, CVPR-15) [29]
-
OTL (Online Transfer Learning, ICML-10) [31]
-
RWA (Random Walking, arXiv, simple but powerful) [46]
-
MEDA (Manifold Embedded Distribution Alignment, ACM MM-18) [47]
-
EasyTL (Practically Easy Transfer Learning, ICME-19) [63]
-
SCA (Scatter Component Analysis, TPAMI-17) [79]
-
DaNN (Domain Adaptive Neural Network, PRICAI-14) [41]
-
DDC (Deep Domain Confusion, arXiv-14)
-
DeepCORAL (Deep CORAL: Correlation Alignment for Deep Domain Adaptation) [33]
-
DAN/JAN (Deep Adaptation Network/Joint Adaptation Network, ICML-15,17) [9,10]
-
RTN (Unsupervised Domain Adaptation with Residual Transfer Networks, NIPS-16) [12]
-
ADDA (Adversarial Discriminative Domain Adaptation, arXiv-17) [13]
-
DANN/RevGrad (Unsupervised Domain Adaptation by Backpropagation, ICML-15) [14]
-
DANN Domain-Adversarial Training of Neural Networks (JMLR-16)[17]
-
Associative Domain Adaptation (ICCV-17) [18]
-
Deep Hashing Network for Unsupervised Domain (CVPR-17) [20]
-
CCSA (Unified Deep Supervised Domain Adaptation and Generalization, ICCV-17) [23]
-
MRN (Learning Multiple Tasks with Multilinear Relationship Networks, NIPS-17) [24]
-
AutoDIAL (Automatic DomaIn Alignment Layers, ICCV-17) [25]
-
DSN (Domain Separation Networks, NIPS-16) [26]
-
DRCN (Deep Reconstruction-Classification Networks for Unsupervised Domain Adaptation, ECCV-16) [27]
-
Multi-task Autoencoders for Domain Generalization (ICCV-15) [28]
-
Encoder based lifelong learning (ICCV-17) [30]
-
MECA (Minimal-Entropy Correlation Alignment, ICLR-18) [32]
-
WAE (Wasserstein Auto-Encoders, ICLR-18) [34]
-
ATDA (Asymmetric Tri-training for Unsupervised Domain Adaptation, ICML-15) [35]
-
PixelDA_GAN (Unsupervised pixel-level domain adaptation with GAN, CVPR-17) [36]
-
ARDA (Adversarial Representation Learning for Domain Adaptation) [37]
-
DiscoGAN (Learning to Discover Cross-Domain Relations with Generative Adversarial Networks) [38]
-
MADA (Multi-Adversarial Domain Adaptation, AAAI-18) [40]
-
MCD (Maximum Classifier Discrepancy, CVPR-18) [42]
-
Adversarial Feature Augmentation for Unsupervised Domain Adaptation (CVPR-18) [43]
-
DML (Deep Mutual Learning, CVPR-18) [44]
-
Self-ensembling for visual domain adaptation (ICLR 2018) [45]
-
PADA (Partial Adversarial Domain Adaptation, ECCV-18) [48]
-
iCAN (Incremental Collaborative and Adversarial Network for Unsupervised Domain Adaptation, CVPR-18) [49]
-
WeightedGAN (Importance Weighted Adversarial Nets for Partial Domain Adaptation, CVPR-18) [50]
-
OpenSet (Open Set Domain Adaptation by Backpropagation) [51]
-
WDGRL (Wasserstein Distance Guided Representation Learning, AAAI-18) [52]
-
JDDA (Joint Domain Alignment and Discriminative Feature Learning) [53]
-
Multi-modal Cycle-consistent Generalized Zero-Shot Learning (ECCV-18) [54]
-
MSTN (Moving Semantic Transfer Network, ICML-18) [55]
-
SAN (Partial Transfer Learning With Selective Adversarial Networks, CVPR-18) [56]
-
M-ADDA (Metric-based Adversarial Discriminative Domain Adaptation, ICML-18 workshop) [57]
-
Openset_DA (Open Set Domain Adaptation by Backpropagation) [58]
-
DIRT-T (A DIRT-T Approach to Unsupervised Domain Adaptation, ICLR-18) [59]
-
CDAN (Conditional Adversarial Domain Adaptation, NeurIPS-18) [60]
-
CMD (Central Moment Discrepancy, ICLR-17 and InfSc-19) [61], [62]
-
OPDA_BP (Open Set Domain Adaptation by Back-propagation, ECCV-18) [64]
-
TCP (Transfer Channel Prunning, IJCNN-19) [65]
-
MTAN (Multi-Task Attention Network, CVPR-19) [66]
-
L2T_ww (Learning What and Where to Transfer, ICML-19) [68]
-
SSDA_MME (Semi-supervised Domain Adaptation via Minimax Entropy, ICCV-19) [71]
-
MRAN (Multi-representation adaptation network for cross-domain image classification, Neural Networks 2019) [72]
-
TA3N (Temporal Attentive Alignment for Large-Scale Video Domain Adaptation, ICCV-19) [73]
-
MDAN (Multiple Source Domain Adaptation with Adversarial Learning, NeurIPS-18) [74]
-
Deep model transferribility from attribution maps (NeurIPS-19) [75]
-
DIVA (Domain Invariant Variational Autoencoders, arXiv-19) [76]
-
CDCL (Cross-Domain Complementary Learning with Synthetic Data for Multi-Person Part Segmentation, arXiv, ICCV-19 Demo) [77]
-
DTA (Drop to Adapt: Learning Discriminative Features for Unsupervised Domain Adaptation, arXiv, ICCV-19) [78]
-
DAAN (Dynamic Adversarial Adaptation Network, ICDM 2019) [80]
-
DAEL (Domain Adaptive Ensemble Learning, ArXiv 2020) [81]
-
DSAN (Deep Subdomain Adaptation Network for Image Classification, DSAN 2020) [82]
-
Learning to select data for transfer learning with Bayesian Optimization (EMNLP-17) [69]
-
SDG4DA (Reinforced Training Data Selection for Domain Adaptation, ACL-19) [70]
Code from HKUST [a bit old]
[1] Pan S J, Tsang I W, Kwok J T, et al. Domain adaptation via transfer component analysis[J]TNN, 2011, 22(2): 199-210.
[2] Gong B, Shi Y, Sha F, et al. Geodesic flow kernel for unsupervised domain adaptation[C]//CVPR, 2012: 2066-2073.
[3] Long M, Wang J, Ding G, et al. Transfer feature learning with joint distribution adaptation[C]//ICCV. 2013: 2200-2207.
[4] Long M, Wang J, Ding G, et al. Transfer joint matching for unsupervised domain adaptation[C]//CVPR. 2014: 1410-1417.
[5] Sun B, Feng J, Saenko K. Return of Frustratingly Easy Domain Adaptation[C]//AAAI. 2016, 6(7): 8.
[6] Zhang J, Li W, Ogunbona P. Joint Geometrical and Statistical Alignment for Visual Domain Adaptation[C]//CVPR 2017.
[8] Dai W, Yang Q, Xue G R, et al. Boosting for transfer learning[C]//ICML, 2007: 193-200.
[9] Long M, Cao Y, Wang J, et al. Learning transferable features with deep adaptation networks[C]//ICML. 2015: 97-105.
[10] Long M, Wang J, Jordan M I. Deep transfer learning with joint adaptation networks[J]//ICML 2017.
[11] Fernando B, Habrard A, Sebban M, et al. Unsupervised visual domain adaptation using subspace alignment[C]//ICCV. 2013: 2960-2967.
[12] Long M, Zhu H, Wang J, et al. Unsupervised domain adaptation with residual transfer networks[C]//NIPS. 2016.
[13] Tzeng E, Hoffman J, Saenko K, et al. Adversarial discriminative domain adaptation[J]. arXiv preprint arXiv:1702.05464, 2017.
[14] Ganin Y, Lempitsky V. Unsupervised domain adaptation by backpropagation[C]//International Conference on Machine Learning. 2015: 1180-1189.
[15] Jindong Wang, Yiqiang Chen, Shuji Hao, Wenjie Feng, and Zhiqi Shen. Balanced Distribution Adaptation for Transfer Learning. ICDM 2017.
[16] Y. Xu et al., "A Unified Framework for Metric Transfer Learning," in IEEE Transactions on Knowledge and Data Engineering, vol. 29, no. 6, pp. 1158-1171, June 1 2017. doi: 10.1109/TKDE.2017.2669193
[17] Ganin Y, Ustinova E, Ajakan H, et al. Domain-adversarial training of neural networks[J]. Journal of Machine Learning Research, 2016, 17(59): 1-35.
[18] Haeusser P, Frerix T, Mordvintsev A, et al. Associative Domain Adaptation[C]. ICCV, 2017.
[19] Pau Panareda Busto, Juergen Gall. Open set domain adaptation. ICCV 2017.
[20] Venkateswara H, Eusebio J, Chakraborty S, et al. Deep hashing network for unsupervised domain adaptation[C]. CVPR 2017.
[21] H. Lu, L. Zhang, et al. When Unsupervised Domain Adaptation Meets Tensor Representations. ICCV 2017.
[22] J. Wang, Y. Chen, L. Hu, X. Peng, and P. Yu. Stratified Transfer Learning for Cross-domain Activity Recognition. 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom).
[23] Motiian S, Piccirilli M, Adjeroh D A, et al. Unified deep supervised domain adaptation and generalization[C]//The IEEE International Conference on Computer Vision (ICCV). 2017, 2.
[24] Long M, Cao Z, Wang J, et al. Learning Multiple Tasks with Multilinear Relationship Networks[C]//Advances in Neural Information Processing Systems. 2017: 1593-1602.
[25] Maria Carlucci F, Porzi L, Caputo B, et al. AutoDIAL: Automatic DomaIn Alignment Layers[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 5067-5075.
[26] Bousmalis K, Trigeorgis G, Silberman N, et al. Domain separation networks[C]//Advances in Neural Information Processing Systems. 2016: 343-351.
[27] M. Ghifary, W. B. Kleijn, M. Zhang, D. Balduzzi, and W. Li. "Deep Reconstruction-Classification Networks for Unsupervised Domain Adaptation (DRCN)", European Conference on Computer Vision (ECCV), 2016
[28] M. Ghifary, W. B. Kleijn, M. Zhang, D. Balduzzi. Domain Generalization for Object Recognition with Multi-task Autoencoders, accepted in International Conference on Computer Vision (ICCV 2015), Santiago, Chile.
[29] Aljundi R, Emonet R, Muselet D, et al. Landmarks-based kernelized subspace alignment for unsupervised domain adaptation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015: 56-63.
[30] Rannen A, Aljundi R, Blaschko M B, et al. Encoder based lifelong learning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 1320-1328.
[31] Peilin Zhao and Steven C.H. Hoi. OTL: A Framework of Online Transfer Learning. ICML 2010.
[32] Pietro Morerio, Jacopo Cavazza, Vittorio Murino. Minimal-Entropy Correlation Alignment for Unsupervised Deep Domain Adaptation. ICLR 2018.
[33] Sun B, Saenko K. Deep coral: Correlation alignment for deep domain adaptation[C]//European Conference on Computer Vision. Springer, Cham, 2016: 443-450.
[34] Tolstikhin I, Bousquet O, Gelly S, et al. Wasserstein Auto-Encoders[J]. arXiv preprint arXiv:1711.01558, 2017.
[35] Saito K, Ushiku Y, Harada T. Asymmetric tri-training for unsupervised domain adaptation[J]. arXiv preprint arXiv:1702.08400, 2017.
[36] Bousmalis K, Silberman N, Dohan D, et al. Unsupervised pixel-level domain adaptation with generative adversarial networks[C]//The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017, 1(2): 7.
[37] Shen J, Qu Y, Zhang W, et al. Adversarial representation learning for domain adaptation[J]. arXiv preprint arXiv:1707.01217, 2017.
[38] Kim T, Cha M, Kim H, et al. Learning to discover cross-domain relations with generative adversarial networks[J]. arXiv preprint arXiv:1703.05192, 2017.
[39] Tommasi T, Caputo B. Frustratingly Easy NBNN Domain Adaptation[C]. international conference on computer vision, 2013: 897-904.
[40] Pei Z, Cao Z, Long M, et al. Multi-Adversarial Domain Adaptation[C] // AAAI 2018.
[41] Ghifary M, Kleijn W B, Zhang M. Domain adaptive neural networks for object recognition[C]//Pacific Rim International Conference on Artificial Intelligence. Springer, Cham, 2014: 898-904.
[42] Saito K, Watanabe K, Ushiku Y, et al. Maximum Classifier Discrepancy for Unsupervised Domain Adaptation[J]. arXiv preprint arXiv:1712.02560, 2017.
[43] Volpi R, Morerio P, Savarese S, et al. Adversarial Feature Augmentation for Unsupervised Domain Adaptation[J]. arXiv preprint arXiv:1711.08561, 2017.
[44] Zhang Y, Xiang T, Hospedales T M, et al. Deep Mutual Learning[C]. CVPR 2018.
[45] French G, Mackiewicz M, Fisher M. Self-ensembling for visual domain adaptation[C]//International Conference on Learning Representations. 2018.
[46] van Laarhoven T, Marchiori E. Unsupervised Domain Adaptation with Random Walks on Target Labelings[J]. arXiv preprint arXiv:1706.05335, 2017.
[47] Jindong Wang, Wenjie Feng, Yiqiang Chen, Han Yu, Meiyu Huang, Philip S. Yu. Visual Domain Adaptation with Manifold Embedded Distribution Alignment. ACM Multimedia conference 2018.
[48] Zhangjie Cao, Mingsheng Long, et al. Partial Adversarial Domain Adaptation. ECCV 2018.
[49] Zhang W, Ouyang W, Li W, et al. Collaborative and Adversarial Network for Unsupervised domain adaptation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 3801-3809.
[50] Zhang J, Ding Z, Li W, et al. Importance Weighted Adversarial Nets for Partial Domain Adaptation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 8156-8164.
[51] Saito K, Yamamoto S, Ushiku Y, et al. Open Set Domain Adaptation by Backpropagation[J]. arXiv preprint arXiv:1804.10427, 2018.
[52] Shen J, Qu Y, Zhang W, et al. Wasserstein Distance Guided Representation Learning for Domain Adaptation[C]//AAAI. 2018.
[53] Chen C, Chen Z, Jiang B, et al. Joint Domain Alignment and Discriminative Feature Learning for Unsupervised Deep Domain Adaptation[J]. arXiv preprint arXiv:1808.09347, 2018.
[54] Felix R, Vijay Kumar B G, Reid I, et al. Multi-modal Cycle-consistent Generalized Zero-Shot Learning. ECCV 2018.
[55] Xie S, Zheng Z, Chen L, et al. Learning Semantic Representations for Unsupervised Domain Adaptation[C]//International Conference on Machine Learning. 2018: 5419-5428.
[56] Cao Z, Long M, Wang J, et al. Partial transfer learning with selective adversarial networks. CVPR 2018.
[57] Issam Laradji, Reza Babanezhad. M-ADDA: Unsupervised Domain Adaptation with Deep Metric Learning. ICML 2018 workshop.
[58] Saito K, Yamamoto S, Ushiku Y, et al. Open Set Domain Adaptation by Backpropagation[J]. arXiv preprint arXiv:1804.10427, 2018.
[59] Shu R, Bui H H, Narui H, et al. A DIRT-T Approach to Unsupervised Domain Adaptation[J]. arXiv preprint arXiv:1802.08735, 2018.
[60] Mingsheng Long, et al. Conditional Adversarial Domain Adaptation. NeurIPS 2018.
[61] W.Zellinger, T. Grubinger, E. Lughofer, T. Natschlaeger, and Susanne Saminger-Platz, "Central moment discrepancy (cmd) for domain-invariant representation learning," ICLR 2017.
[62] W. Zellinger, B.A. Moser, T. Grubinger, E. Lughofer, T. Natschlaeger, and S. Saminger-Platz, "Robust unsupervised domain adaptation for neural networks via moment alignment," Information Sciences (in press), 2019, https://doi.org/10.1016/j.ins.2019.01.025, arXiv preprint arxiv:1711.06114
[63] Jindong Wang, Yiqiang Chen, Han Yu, Meiyu Huang, Qiang Yang. Easy Transfer Learning By Exploiting Intra-domain Structures. IEEE International Conference on Multimedia & Expo (ICME) 2019.
[64] Saito K, Yamamoto S, Ushiku Y, et al. Open set domain adaptation by backpropagation[C]//Proceedings of the European Conference on Computer Vision (ECCV). 2018: 153-168.
[65] Chaohui Yu, Jindong Wang, Yiqiang Chen, Zijing Wu. Accelerating Deep Unsupervised Domain Adaptation with Transfer Channel Pruning. IJCNN 2019.
[66] Shikun Liu, Edward Johns, and Andrew Davison. End-to-End Multi-Task Learning with Attention. CVPR 2019.
[67] Huang J, Gretton A, Borgwardt K, et al. Correcting sample selection bias by unlabeled data[C]//Advances in neural information processing systems. 2007: 601-608.
[68] Yunhun Jang, Hankook Lee, Sung Ju Hwang, Jinwoo Shin. Learning what and where to transfer. ICML 2019.
[69] Sebastian Ruder, Barbara Plank (2017). Learning to select data for transfer learning with Bayesian Optimization. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.
[70] Liu M, Song Y, Zou H, et al. Reinforced Training Data Selection for Domain Adaptation[C]//Proceedings of the 57th Conference of the Association for Computational Linguistics. 2019: 1957-1968.
[71] Saito K, Kim D, Sclaroff S, et al. Semi-supervised Domain Adaptation via Minimax Entropy. ICCV 2019.
[72] Zhu Y, Zhuang F, Wang J, et al. Multi-representation adaptation network for cross-domain image classification[J]. Neural Networks, 2019.
[73] Min-Hung Chen, Zsolt Kira, Ghassan AlRegib, et al. Temporal Attentive Alignment for Large-Scale Video Domain Adaptation. ICCV 2019.
[74] Zhao H, Zhang S, Wu G, et al. Multiple source domain adaptation with adversarial learning. NeurIPS 2018.
[75] Jie Song, et al. Deep model transferrability from attirbution maps. NeurIPS 2019.
[76] Ilse, M., Tomczak, J. M., C. Louizos & Welling, M. (2018). DIVA: Domain Invariant Variational Autoencoders. arXiv preprint arXiv:1905.10427
[77] Lin K., et al. Cross-Domain Complementary Learning with Synthetic Data for Multi-Person Part Segmentation[J]. arXiv preprint arXiv:1907.05193, ICCV demo, 2019.
[78] Lee S., Kim D., et al. Drop to Adapt: Learning Discriminative Features for Unsupervised Domain Adaptation. ICCV 2019.
[79] Ghifary M, Balduzzi D, Kleijn W B, et al. Scatter component analysis: A unified framework for domain adaptation and domain generalization[J]. IEEE transactions on pattern analysis and machine intelligence, 2016, 39(7): 1414-1430.
[80] Chaohui Yu, Jindong Wang, Yiqiang Chen, Meihu Huang. Transfer learnign with dynamic adversarial adaptation network. ICDM 2019.
[81] Kaiyang Zhou, Yongxin Yang, Yu Qiao, Tao Xiang. Domain Adaptive Ensemble Learning. ArXiv preprint, 2020.