Malaikannan The Deep Learning way of life

Deep Learning using Numpy

DeepLearning is a verstaile tool to solve problems that cannot be solved using traditional programming approach. I am a CTO at Datalog.ai where we solve lot of cool problems using Deep Learning. ML Researchers and Engineers use lot of Deep Learning packages like Theano, Tensorflow, Torch, Keras etc. Packages are really good but when you want to get an understanding on how Deep Learning works, it is better to go back to basics and understand how it is done. This blog is at an attempt at that, it is going to be a 3 part of series with topics being

  1. DeepLearning using Numpy
  2. Why TensorFlow/Theano not Numpy?
  3. Why Keras not TensorFlow/Theano?

Neural Network with 1 hidden layer

Deep learning refers to artificial neural networks that are composed of many layers like the one shown above. Deep Learning has many flavor’s like Convolution Neural Networks, Recurrent Neural Networks, Reinforcement Learning, Feed Forward Neural Network etc. This blog is going to take the simplest of them, Feed Forward Neural network as an example to explain.

Machine Learning deals with lot of Linear Algebra operations like dot product, transpose, reshape etc. If you are not familiar with it, I would suggest refer to my previous blog post in All about Math section.

Deep Learning needs an activation function to squish real numbers to probability values between 0 and 1 , there are different activation functions like sigmoid, Tanh, RELU etc. For this toy example i have used sigmoid activation function.

Sigmoid

We are going to use Gradient Descent to find optimal parameters to solve for Y. Gradient descent uses the derivative of the sum of errors to update the systems parameters a little bit in such a way that the error decreases as much as possible.After every update the system learns to predict with a lower error. Let it run many iterations and it will converge at some optima(local). Sigmoid function takes a parameter to calculate Derivative. Don’t worry if you don’t understand this explanation, it is very intuitive if you can follow the code along. If you are looking for more explanation refer to this video by Prof Andrew Ng.

For this example on Numpy Deep Learning Code, I am going to use a synthetic dataset. Output is the target we are going to predict.

Input and Output

Randomly initialize weights for 2 synapses. Synapses 0 will be of shape 3x4, Synapses 1 will be of shape 4x1

With Gradient descent you have to run the process for n number of iterations, in ML lingo it is called epoch (since it will take ages to complete). In our case we are going to run it for 50 iterations. Since this is a 1 hidden Layer network, we do a dot product between input l0 and synapses_0 and then squish it using sigmoid function. Pass output of l1 as input to hidden layer and do dot product between l1 and synapses_1 weights and then squish it using sigmoid function.

Now we are off to calculate what is the error for our prediction for l2 layer. Then use derivative to find out how much we should update our Synapses 1.

Same step should be done for l1 layer, but error should be calculated based on how much we are off on l2.

Update weights for synapses_0 and synapses_1 based on calculated l1_delta and l2_delta respectively.

See below on how loss is decreasing for each iteration.

With just 50 iterations we are very close to actual value

Output

Siraj Raval has a really good youtube video on Intro to Deep Learning check it out too.

How to Learn Deep Learning?

I oscillated between different blogs and videos to become a deep learning practitioner. This blog is to document my learning and to follow an optimal path to become Deep Learning practioner faster

It is all about Math

Don’t be shy if you haven’t brushed your Math skills for a while. When you are programming for while bad habits creep in , it takes time to unlearn and learn new things. I had a tough time initially then started refreshing my Math again. I used Khan academy , i like how most of the sessions are only 10 minutes long. I followed below order

  1. Algebra – Yes you have to refresh Algebra. Remember the equation for straight line y = mx + b. That is the best equation you learned in your life. Most of machine learning is about finding the value of “m” called weights and “b” called biases.
  2. Trigonometry
  3. Differential Calculus – Machine Learning/Deep Learning is all about finding slope aka derivatives, hence do it thoroughly
  4. Partial Differential Equations
  5. Integral Calculus
  6. Probability and Statistics – this is important for anything in Machine Learning.
  7. Linear Algebra – Most of calculations are done using Matrix multiplication, dot products, transpose so learn this well.
  8. Linear Algebra Advanced – Yes it is that important. I referred to Prof Gilbert Strang lectures from MIT.

Intro into Machine Learning

I took Prof Andrew Ng’s Coursera Machine Learning course in 2012. It is the bible if you are starting with Machine Learning. Take your time and learn the basics.

Mining Massive Datasets

This was one of the best courses i took, it helped me to understand Mathematical intuition behind lot of Machine Learning algorithms.

Deep Learning

This course is Math heavy, but Prof Ali Ghodsi lectures explains it well. It is one of the hidden gems there are quite a series of lectures in youtube, watch it all. Watch it in loop, till you get hang of every concept.

Convolution Neural Networks

Convolution Neural networks is a class of Deep Learning that is predominantly used for computer vision. AndreJ Karpathy and Justin Johnson taught a great course cs231n in Stanford on CNN. It gives lot of practical tips on building Deep Learning models. I wrote an intro level CNN tutorial for Keras.

Natural Language processing

Richard Socher’s class on Natural Language processing is must if you want to work on Unstructured text. CS224d is heavy on Recurrent Neural Networks. Recently Convolution Neural Networks are being used more for NLP.

Learn Python

Python is becoming a de-facto language for scientific and numerical computation. Most Deep Learning libraries have a python front end. If you are new to python then use Byte of Python book to learn. There are lot of good youtube tutorials too.

GPU’s

If you have managed to take all these classes mentioned in the list above, then you are a serious about being a Deep Learning practitioner. Invest in a good NVIDIA GPU for trying out different models. You can use AWS for training but you will end up spending lot of money to train different models, in a long run it will make sense to buy your own hardware. Hey you can use it for gaming too, if you feel bored about Deep Learning.

Must Buy/Read Books

Ian GoodFellow, Aaron CourVille, and Yoshua Bengio wrote an awesome Deep Learning Book. I bought it, since it is a text-book theory book. Another book i often referred to is Neural Networks and DeepLearning. This book explains [Backpropagation]((http://neuralnetworksanddeeplearning.com/chap2.html), one of the most important concepts in deep learning very well.

Blogs to Read

I read Machine Learning Mastery, it has practical tips and good blogs.

Deep Learning Frameworks

There are quite a few options when it comes to Deep Learning frameworks

  1. Tensorflow
  2. Theano
  3. Keras
  4. Caffe
  5. CNTK
  6. Lasagne
  7. Other

I am personally big fan of Keras (wrapper over Tensorflow and Theano), since it abstracts lot of complexity of building a Deep Learning model, i can build a model and test whether it works or not very fast. There are tons of online tutorials on Tensorflow and Theano.

Gokul Krishnan wrote a really good blog on Anatomy of Deep Learning Frameworks

Kaggle

Kaggle is a data science competition forum, lot of researchers compete there and share their approach they used for solving that problem. Compete actively to learn and improve.

Follow Researchers on Twitter

I used twitter recommendation engine (learning machine learning using machine learning) to keep myself updated with latest research papers. Check whom i am following , on my Twitter

Indian Railway Status Check Chatbot

Indian Railways Status check Chatbot is integrated with Facebook Messenger. It can be used to find details about

  • PNR Number
  • Find Station By name
  • Find Station By stationcode
  • Find Stations on a Train route
  • Find Trains leaving from a Station in next 4 hours
  • Find Trains updating
  • Find Rescheduled Trains for a date
  • Find Cancelled Trains for a date
  • Find Train between stations
  • Find Train fare
  • Find Train Seat availability

Bot is will available for public use in another week.