## Neural networks

A neural network is based somewhat on a real brain - the idea being that data comes into a bunch of neurons at the front, gets propagated through multiple layers, and an answer comes out the end. We've covered this a little in another blog post, so here we'll just focus on what goes on under the covers.

## Math time

A neural network is a type of non-linear function that can approximate any other function. How well it can approximate is based on the number of links it has between its layers of neurons. In other words, more neurons and more layers gives you a better (but slower) neural network. If you want to have a play with this idea, have a look at the Tensorflow Playground - if you can't get it to match, try a mix of more layers and more neurons per layer.

## More math time

We have a non-linear function, and we need to approximate something based on stochastic gradient descent. In other words, we get a bunch of samples as input, and we need to adjust the network based on what the output *should* be, and then the network will change to give us answers that are slightly closer to what we expect. If our training data is a diverse sample of our real data, then we can train a neural network on our training data, and it'll also work as well with other data that it hasn't seen before.

This does seem kind of like magic though, right? What's going on under the covers?

This does seem kind of like magic though, right? What's going on under the covers?

## The calculus bit

We generally use a method called Stochastic Gradient Descent, or a method related to it. The "stochastic" part means we randomly choose samples from our training data so that the model doesn't get biased, and all the little nudges we give it start to shape it in the way we want it to behave. The "gradient descent" part means for each sample, we look at how wrong the network currently is (the "error"), and calculate a gradient (or slope, if you like), and nudge the network in that direction.

This approach is robust and it works for a lot of things, but the hard part with a neural network is trying to find the gradient for every single weight in the network - if your network is 10 layers deep (a small network by modern standards), how does an error at the output adjust the weights 10 layers back at the start?

## Backpropagation

We use partial derivatives to find the gradient at each layer, and then use that gradient to adjust the network. At each neuron, we take the sum of all of the downstream gradients (they get "backpropagated" to us), multiplied by the derivative of the activation function applied to the forward-propagated data. We then take this number, multiplied by the weight of the previous link, and add a fraction of this (multiplied by the training rate) to the current weight of that link. We also take this number multiplied by the forward-propagated data and backpropagate that to the previous neuron.

Simple... right?

This is some fiddly stuff if you're not currently a calculus student, and it took me a couple weeks to get my head around. You can find it all on the wikipedia page for backpropagation, and there are a couple of papers floating around the internet that explain it in different ways.

## It shouldn't be this hard

The idea of multiplying partial derivatives is a simple one, and I spent a long time thinking about how to make this accessible. The architecture that I've settled on for Lernmi has made a couple of trade-offs, but I feel it breaks out the backpropagation into bite-sized pieces, and I hope that it makes it easier to understand.

## Lernmi

I've published a neural network application in ruby, hosted here. The logic is broken up between Neurons and Links, and instead of the word "gradient", I've used the word "sensitivity" as it better translates to what we're using it for - high sensitivity means we adjust the weight more, low sensitivity means we adjust it less.

When we propagate forwards, it's fairly simple. Each Link in a layer takes the output from it's input Neuron, multiplies it by its weight, and inputs it to its output Neuron. Each Neuron sums all of its inputs, applies the activation function (the sigmoid function), and waits for the next Link to take the resulting value.

When we backpropagate, the logic is spread across the Neurons and Links. For the output layer, the sensitivity is just the error (the actual output minus the expected output). Each Link in the last layer takes the sensitivity from its output Neuron, and multiplies this by the sensitivity of the activation function - we call this the "output sensitivity". The reason we do this is because our neurons use their activation function when they propagate, so we need to find the sensitivity of that, and we multiply it by the sensitivity of the output Neuron to get the sensitivity of the Link. Perhaps a better name could be the "link sensitivity"? We'll call it the "output sensitivity" for now though.

We use the output sensitivity in two ways: when we update the weight, we multiply it by its input value (the larger our input value, the more we probably contributed to the error), and adjust the weight of this Link by that amount. We also need to send a sensitivity back to the input Neuron - we multiply the output sensitivity by our weight, so that the earlier layer knows how much this Link contributed to the error.

## Whose fault is this?

Don't be surprised if this is confusing the first time your read through. I would love to find a way to simplify this further - the principle is as simple as asking "what part of the network is to blame for the error". The math is a bit frustrating, as we're ultimately multiplying partial derivatives along the length of the network, but the goal of it is to do lots of tiny adjustments until the network is a good-enough approximation of the underlying data.

## What are we approximating though?

This is where it gets a little freaky. It turns out that everything - handwriting, faces, pictures of various different objects, they all have a mathematical approximation. With a big enough neural network, you can recognise people, animals, objects, and various different styles of handwriting and signwriting. Even in a board game like chess or go, you can make a mathematical approximation of which moves are better, to the point where you can have a computer that can play at a world class level.

This isn't the same as intelligence; the computer isn't thinking, but at the same time, what is thinking anyway? Is what we call "thinking" and "judging" just a mathematical approximation in our heads?

Useful models are made accessible by utilizing OLAP for anticipating, pattern investigation and so forth. They bolster clients in information investigation. data science course in pune

ReplyDeleteWell, The information which you posted here is very helpful & it is very useful for the needy like me.., Wonderful information you posted here. Thank you so much for helping me out to find the Data Analytics Course in MumbaiData analytics course in Mumbai

ReplyDeleteNice Post...I have learn some new information.thanks for sharing.

ReplyDeleteExcelR data analytics course in Pune | business analytics course | data scientist course in Pune

Such a very useful article. I have learn some new information.thanks for sharing.

ReplyDeletedata scientist course in mumbai

I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.

ReplyDeleteData Analytics Course in Mumbai

Such a very useful article. Very interesting to read this article. I have learn some new information.thanks for sharing. ExcelR

ReplyDeleteVery nice blog here and thanks for post it.. Keep blogging...

ReplyDeleteExcelR data science training

Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.

ReplyDeletedata science course in mumbai

Just saying thanks will not just be sufficient, for the fantasti c lucidity in your writing. I will instantly grab your rss feed to stay informed of any updates.

ReplyDeletedata analytics course mumbai

This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck... data science course

ReplyDeleteI finally found great post here.I will get back here. I just added your blog to my bookmark sites. thanks.Quality posts is the crucial to invite the visitors to visit the web page, that's what this web page is providing.

ReplyDeletemachine learning course

artificial intelligence course in mumbai

This is a wonderful article. I really enjoyed reading this article. Thanks for sharing such detailed information.

ReplyDeleteData Science Course in Marathahalli

Data Science Course Training in Bangalore

Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article.

ReplyDeletebusiness analytics course

data analytics courses in mumbai

data science interview questions

data science course in mumbai

I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.

ReplyDeletedata analytics courses

business analytics course

data science interview questions

data science course in mumbai

Thanks for sharing a such a information on Artificial Intelligence. I am interested learning more about this topic.

ReplyDeleteArtificial Intelligence Training In Hyderabad

Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites!

ReplyDeletebusiness analytics course

AI course in mumbai

Nice Article.very impressed for this informative

ReplyDeleteExcelR data analytics courses

This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.

ReplyDeletebest digital marketing course mumbai

wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.

ReplyDeleteData science Interview Questions

You have clarified the idea truly well. Was searching for this data from some time and fortunately I discovered your post. Searching forward for a greater amount of such useful updates from you

ReplyDeleteData Science Training In Hyderabad

Data Science Course In Hyderabad

Hi, Thanks for sharing nice information...

ReplyDeleteData Science Training In Hyderabad

wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.

ReplyDeleteData science Interview Questions

ReplyDeleteExcellent! I love to post a comment that "The content of your post is awesome" Great work!

business analytics courses

wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.

ReplyDeleteData Science Course

Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!

ReplyDeletedata science course

I am looking for and I love to post a comment that "The content of your post is awesome" Great work!

ReplyDeletedata science course

wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.

ReplyDeleteData Science Course

I am looking for and I love to post a comment that "The content of your post is awesome" Great work!

ReplyDeletedata science course

I have to search sites with relevant information ,This is a

ReplyDeletewonderful blog,These type of blog keeps the users interest in

the website, i am impressed. thank you.

machine learning course in hyderabad

Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article.

ReplyDeletedata science course

wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries. keep it up.

ReplyDeletedata analytics course in Bangalore

Informative post, i love reading such posts. Read my posts here

ReplyDeleteLaravel web development

Instapaper

Boostesports

wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries. keep it up.

ReplyDeletedata analytics course in Bangalore

Attend The Artificial Intelligence course From ExcelR. Practical Artificial Intelligence course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Artificial Intelligence course.

ReplyDeleteArtificial Intelligence CourseHi Thanks For Sharing The Knowledge With us

ReplyDeleteData Science Training in Hyderabad

Data Science Course in Hyderabad

Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspried me to read more. keep it up.

ReplyDeleteCorrelation vs Covariance

Hey, i liked reading your article. You may go through few of my creative works here

ReplyDeleteGlitch

Exercism

Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.

ReplyDeleteCorrelation vs Covariance

I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.

ReplyDeleteCorrelation vs Covariance

The context has been explained really well. Looking forward to see more of such informative updates

ReplyDeleteMachine Learning Training in Hyderabad

Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article.

ReplyDeleteSimple Linear Regression

Attend The Course in Data Analytics From ExcelR. Practical Course in Data Analytics Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Course in Data Analytics.

ReplyDeleteCourse in Data AnalyticsVery interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.

ReplyDeleteCorrelation vs Covariance

Simple linear regression

Wow what a Great Information about World Day its exceptionally pleasant educational post. a debt of gratitude is in order for the post.

ReplyDeleteData Science Course in Bangalore

Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.

ReplyDeleteCorrelation vs Covariance

Simple linear regression

Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.

ReplyDeleteSimple Linear Regression

Correlation vs Covariance

Very interesting blog. Many blogs I see these days do not really provide anything that attracts others, but believe me the way you interact is literally awesome.You can also check my articles as well.

ReplyDeleteData Science In Banglore With Placements

Data Science Course In Bangalore

Data Science Training In Bangalore

Best Data Science Courses In Bangalore

Data Science Institute In Bangalore

Thank you..

I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you!

ReplyDeleteSimple Linear Regression

Correlation vs Covariance

This post is very simple to read and appreciate without leaving any details out. Great work!

ReplyDeletelearn360digitmg artificial intelligence online course

Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.

ReplyDeleteCorrelation vs Covariance

Simple linear regression

data science interview questions

Positive site, where did u come up with the information on this posting?I have read a few of the articles on your website now, and I really like your style. Thanks a million and please keep up the effective work. best free instagram likes and followers app

ReplyDeleteI am looking for and I love to post a comment that "The content of your post is awesome" Great work!

ReplyDeletedata science interview questions

Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.

ReplyDeleteCorrelation vs Covariance

Simple linear regression

data science interview questions

I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.

ReplyDeletedata science interview questions

this blog was really great, never seen a great blog like this before. i think im gonna share this to my friends..

ReplyDeletedata science training in guwahati

cool stuff you have and you keep overhaul every one of us

ReplyDeleteSimple Linear Regression

Correlation vs Covariance

Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.

ReplyDeleteCorrelation vs Covariance

Simple linear regression

data science interview questions

There's no doubt i would fully rate it after i read what is the idea about this article. You did a nice job..

ReplyDelete360digitmg ai and machine learning courses

The Way Of Explaination Is Very Good And Nice Informative Topic You Have Choosen..

ReplyDeleteAWS Course in Hyderabad

This Was An Amazing ! I Haven't Seen This Type of Blog Ever ! Thankyou For Sharing, data science training

ReplyDelete

ReplyDeleteI have recently visited your blog profile. I am totally impressed by your blogging skills and knowledge.

Data Science Course in Hyderabad

Correlation vs Covariance

Simple linear regression

data science interview questions

Very educating story, saved your site for hopes to read more!

ReplyDelete360digitmg ai courses

I am looking for and I love to post a comment that "The content of your post is awesome" Great work!

ReplyDeleteSimple Linear Regression

Correlation vs Covariance

Correlation vs Covariance

Simple linear regression

data science interview questions