Konstantin Schürholt

Konstantin Schürholt

Machine Learning researcher investigating populations of Neural Networks, self-supervised and meta learning, sparsity and ML for science.

An Investigation of the Weight Space to Monitor the Training Progress of Neural Networks

Published in Arxiv, 2021

Recommended citation: Schürholt et al., 2021. "An Investigation of the Weight Space to Monitor the Training Progress of Neural Networks." Arxiv 2021. https://arxiv.org/abs/2006.10424

Abstract

Safe use of Deep Neural Networks (DNNs) requires careful testing. However, deployed models are often trained further to improve in performance. As rigorous testing and evaluation is expensive, triggers are in need to determine the degree of change of a model. In this paper we investigate the weight space of DNN models for structure that can be exploited to that end. Our results show that DNN models evolve on unique, smooth trajectories in weight space which can be used to track DNN training progress. We hypothesize that curvature and smoothness of the trajectories as well as step length along it may contain information on the state of training as well as potential domain shifts. We show that the model trajectories can be separated and the order of checkpoints on the trajectories recovered, which may serve as a first step towards DNN model versioning.

Bibtex:

@article{schurholtInvestigationWeightSpace2021,
  title = {An  of the  to  the  of },
  author = {Sch{\"u}rholt, Konstantin and Borth, Damian},
  year = {2021},
  month = mar,
  doi = {10.48550/arXiv.2006.10424},
  url={https://arxiv.org/abs/2006.10424}

}