Advice or review for minimal neural network port from Python to Rust


#1

Hi,
for self-educational purposes I read a book (details below) which explains how neural networks work (basically) and how to implement a minimal network. In the book, this is done in python, and the source code is on github (see below), so there is a reference implementation. I tried to do the same in rust, and while I could not find any mistake, the precision in digit recognition (MNIST) is actually not better random, while at this stage in the book it’s claimed to have some significant, visible effect: 60% on the same data with only one epoch. My implementation does not even perform well on the training data. So there is definitely a problem. I already checked the math, but if the implementation does what I think it does (and I investigated), it should be correct…

My implementation is a crate, the example is the first application of the crate, which is not performing better than random. Do you have any advice for me, how I could debug this or where the usual errors are? Any good advices for “porting” things from Python to Rust? Ideas for better, helpful unit tests? Do you have some experience with rulinalg crate and can detect my mistakes? Help is really appreciated :slight_smile:

The code is pretty short, and I think the readability is okay. Thanks :slight_smile:

My lib: https://github.com/Pfeil/BabyNN/blob/master/src/lib.rs
My example (csv data needs to be downloaded from the book repo): https://github.com/Pfeil/BabyNN/blob/master/examples/mnist1.rs
Reference implementation: https://github.com/makeyourownneuralnetwork/makeyourownneuralnetwork/blob/master/part2_neural_network_mnist_data.ipynb

Book Blog: https://makeyourownneuralnetwork.blogspot.de/


#2

It don’t have any Rust specific advice. It’s very common to have a small bug in your first NN implementation, regardless of the language. I’d start by checking your gradients. Often a sign is flipped or a factor is missing. Just go through the calculations, print out intermediate values and compare to expected values computed by hand. While frustrating, the process should help develop your intuition for how NN training works.