gone

A simple neural network library in Go from scratch. 0 dependencies*

there are 0 neural network related dependencies, the only dependencies are for testing (stretchr/testify) and for persistence (golang/protobuf)

Example

Getting started

 g := gone.New(
  0.1,
  gone.MSE(),
  gone.Layer{
   Nodes: 2,
  },
  gone.Layer{
   Nodes:     4,
   Activator: gone.Sigmoid(),
  },
  gone.Layer{
   Nodes: 1,
  },
 )

 g.Train(gone.SGD(), gone.DataSet{
  {
   Inputs:  []float64{1, 0},
   Targets: []float64{1},
  },
  {
   Inputs:  []float64{0, 1},
   Targets: []float64{1},
  },
  {
   Inputs:  []float64{1, 1},
   Targets: []float64{0},
  },
  {
   Inputs:  []float64{0, 0},
   Targets: []float64{0},
  },
 }, 5000)

 g.Predict([]float64{1, 1})

Saving model to disk

 g.Save("test.gone")

Loading model back into memory

 g, err := gone.Load("test.gone")

TODO

gone/

• Types of task:
  • Classification
  • Regression
• Bias
  • Matrix, rather than a single number
• Feedforward (Predict)
• Train
  • Support shuffling the data
  • Epochs
  • Backpropagation
  • Batching
  • Different loss functions
    • Mean Squared Error
    • Cross Entropy Error
• Saving data - Done thanks to protobuf
• Loading data
• Adam optimizer
• Nestrov + Momentum for GD
• Fix MSE computation in debug mode (not used in actual backpropagation)
• Somehow persist configurations for Activation, Loss and Optimizer functions in the protobuf messages (???, if we want to do it like it tensorflow, we'd have to do interface{} and do type assertions)
• Convolutional Layers
  • Flatten layer

matrix/

• Randomize
• Transpose
• Scale
• AddMatrix
• Add
• SubtractMatrix
• Subtract
• Multiply
• Multiply
• Flatten
• Unflatten
• NewFromArray - makes a single row
• Map
• Fold
• Methods to support chaining

     n.Weights[i].
  Multiply(output).                         // weighted sum of the previous layer)
  Add(n.Layers[i+1].Bias).                  // bias
  Map(func(val float64, x, y int) float64 { // activation
   return n.Layers[i+1].Activator.F(val)
  })

Research

• Derivatives ~
• Partial Derivatives ~
• Linear vs non-linear problems (activation function)
• Gradient Descent
  • (Batch) Gradient Descent (GD)
  • Stochastic Gradient Descent (SGD)
  • Mini-Batch Gradient Descent (MBGD?)
• Softmax (needed for multi class classification!)
• Mean Squared Error
• Cross Entropy Error (needed for multi class classification!)
• How to determine how many layers and nodes to use
• One Hot Encoding
• Convolutional Layers
• Reinforcment learning
• Genetic Algorithms

Examples

• XOR Problem
• Digit Classifier

Shoutouts

• David Josephs - was of HUGE help with algebra and other ML-related questions; also helped me spot some nasty bugs!

Resources used

• https://www.analyticsvidhya.com/blog/2020/01/fundamentals-deep-learning-activation-functions-when-to-use-them/
• https://www.youtube.com/watch?v=XJ7HLz9VYz0&list=PLRqwX-V7Uu6Y7MdSCaIfsxc561QI0U0Tb
• https://www.youtube.com/watch?v=aircAruvnKk&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi
• http://matrixmultiplication.xyz/
• https://www.khanacademy.org/math/precalculus/x9e81a4f98389efdf:matrices/x9e81a4f98389efdf:properties-of-matrix-addition-and-scalar-multiplication/a/properties-of-matrix-addition
• https://www.wikiwand.com/en/Matrix_(mathematics)
• https://www.wikiwand.com/en/Activation_function
• https://www.wikiwand.com/en/Delta_rule
• https://www.jeremyjordan.me/intro-to-neural-networks/
• https://www.arxiv-vanity.com/papers/2003.02139/
• https://machinelearningmastery.com/gentle-introduction-mini-batch-gradient-descent-configure-batch-size/
• http://neuralnetworksanddeeplearning.com/chap2.html
• https://arxiv.org/pdf/1802.01528.pdf
• https://github.com/stevenmiller888/mind/blob/master/index.js
• https://github.com/stevenmiller888/go-mind
• https://medium.com/yottabytes/everything-you-need-to-know-about-gradient-descent-applied-to-neural-networks-d70f85e0cc14
• https://towardsdatascience.com/deep-learning-which-loss-and-activation-functions-should-i-use-ac02f1c56aa8
• https://github.com/milosgajdos/go-neural
• https://ml-cheatsheet.readthedocs.io/en/latest/forwardpropagation.html
• https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/
• https://medium.com/coinmonks/representing-neural-network-with-vectors-and-matrices-c6b0e64db9fb
• https://towardsdatascience.com/classifying-cat-pics-with-a-logistic-regression-model-e35dfb9159bb
• https://towardsdatascience.com/math-neural-network-from-scratch-in-python-d6da9f29ce65
• https://towardsdatascience.com/gradient-descent-from-scratch-e8b75fa986cc
• https://rstudio-pubs-static.s3.amazonaws.com/337306_79a7966fad184532ab3ad66b322fe96e.html
• https://gombru.github.io/2018/05/23/cross_entropy_loss
• https://medium.com/@tomvykruta/memory-aid-for-softmax-and-cross-entropy-loss-5704c66d795d