Workshop | Deep Learning - Recent Advances in Kernel Methods for Neural Networks

Oct 5 - 6, 2023 | Triangel OPEN SPACE, Kaiserstr. 93, Karlsruhe

Lecturer: Adit Radhakrishnan (MIT and Broad Institute)

The Institute of Economics at KIT, MathSEE, KCDS and Helmholtz Information and Data Science Academy (HIDA) proudly present the Deep Learning workshop. The workshop is made possible with financial support by MathSEE workshop funding and HIDA course funding.

How to register

The workshop is open to MathSEE and KCDS members as well as other interested (doctoral) researchers at KIT, other Helmholtz centers or other research institutions and universities. As places are limited, available spots will be given to those with a corresponding research interest and strong motivation to join the workshop on a first-come-first-serve basis. Please fill out the application form below.

Motivation

Sparked by the success of deep neural networks in many practical applications, there has been an increasing interest in analyzing these methods from a statistical perspective. In recent years, advances in the theory of neural networks and kernels have brought both fields closer together. This not only sparked new interest and with that fresh ideas in the field of kernels, it also enabled research to explain phenomena occurring in neural networks. Double descent [Belkin et al., 2018], the usefulness of overparametrization (for example in Autoencoder architectures) [Radhakrishnan et al., 2018] or the Neural Tangent Kernel [Jacot et al., 2018] are among those. Notably, Radhakrishnan and co-authors [Radhakrishnan et al., 2022] showed that neural networks merely learn something called the ‘Expected Gradient Outer Product’ (EGOP). They showed that using the EGOP within a simple kernel framework outperforms not only neural networks but also methods such as Gradient Boosting and Random Forests in most cases.

Workshop Content

Adit will teach the mathematical reasoning behind the above-mentioned methods, but also how they are applied in practical applications (mainly from biology). In interactive Python coding sessions, participants will have the chance to develop these methods themselves. The overarching goal of the workshop therefore is to learn about a mathematically founded way of applying deep learning in practice and to disentangle myth and actual capabilities of deep learning and kernels. The power of these methods is that they make use of mechanisms in neural networks without the overhead of training them. These methods are hence applicable to many practical problems, with or without a plethora of available data.

References

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning practice and the bias-variance trade-off. Proceedings of the National Academy of Sciences of the United States of America, 116: 15849–15854, 2018.
Adityanarayanan Radhakrishnan, Karren Yang, Mikhail Belkin, and Carolin Uhler. Memorization in overparameterized autoencoders. 2018.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems, pages 8571–8580, 2018.
Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. Feature learning in neural networks and kernel machines that recursively learn features, 2022.