Among all those mysterious findings and observations of deep learning, which are the real clues that can potentially open the black box of neural networks?
This repository is a sandbox for conducting concept validating experiments for any interesting and insightful observation in deep learning. The primary goal for this project is to gain comprehensive understanding about details, conditions, and potentially unexplored aspects of some well-known phenomena and theoretical claims.
I personally believe current status of deep learning theory is like what physists were before General Relativity or Maxwell's equation being proposed. We need substantial novel and intricate observations as "clues" to intersect and combine to reveal the true story. Only through extensive empirical explorations can we finally propose an elegant and principled theory for deep learning.
Therefore, this repository will go through interesting findings and observations researchers have discovered in the past years. It will delve deep into every detail and main claim for those papers, reproducing the results, in order to validate the claim and comprehensively understand the condition and implications.
Maybe we can find some novel insights, who knows!
| Name | Introduction | Paper | Author |
|---|---|---|---|
| 1. Information Bottleneck | Deep Learning and the Information Bottleneck Principle | Tishby et al. | |
| 2. Edge of Stability | Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability | Cohen et al. | |
| 3. Lottery Ticket Hypothesis | The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks | Frankle et al. |
To be continued...
You can add pull Request to propose the mechanism and phenomena that you feel interesting and want to be explored! We will add detailed experimental log and report in this repo!