Newton–Raphson method - Introduction

This page only deals with the algorithm theory for example of this theory 

check out  

newtons-raphson-method-for-arriving-at.html

 

Alright  the theory says for a function 

y = x - a , the optimization of parameter ' x ' where y = 0 /close to zero

a technique  used for optimization of parameters in machine learning problem,

 consider a = 5 , x = 5 , y will be zero but this case is not straight forward 

the value for x is obtained using Newton–Raphson method, which is via iteration

says  xn+1 = xn - f(x)/f '(x)

 f(x) = y = x- a

f '(x) = partial derivative of x  = dy/dx = 1

consider 2 iterations here n =0 ,1

it starts with assuming a initial value for x which is n =0

x0  , y0 plotting the values in graph

 from the graph u can see i have plotted the x0, y0 

tangent line to the point cuts the x - axis at x1 where the y is '0'

says loss is zero at iteration x1, u may now wonder the the value

of X has moved from X0 to X1 minimize the value of X where the loss here y =0.

U might be still confused why all this talks don't worry will try to explain in detail

OK is i said i will start with assumption for x0

then how to move on or arrive at x1 or some other direction or some other point x2....xn

this is a blinder , according to the theory and the diagram 

there is a triangle  formed ,which is a right angled triangle ,OK then

I started with X0 ,then Y0 ,then finding the tangent 

meaning the tangent line which is a line equation ,don't worry if u don't have an idea of line equation jus stay with me 

 tangent line eq y = mx + b 

y0  = m* x0 +b 

 what is ' m ' ,it is called the slope of the above line 

by formula m = y2-y1/x2-x1

here we have x0 , y0 , next iteration x1 , corresponding y 1

so m = y1- y0 /x1- x0  

 

alright the equation of the curve is y = x -a

what is the slope of the equation which is the partial derivative 

so dy/dx  = 1 

now the step is to find the next iteration value that is x1 , y 1 

from the know values that is x0 , y0 and the slope which we found using

derivative ,so why all these drama ,the idea to find

line equation of the tangent line y = mx +b that passes through (x0, y 0) 

ok then y0 = m * x0 + b

since the slope is 1 which is m =1 

y 0 = x0 +b ,here value of b is the y -intercept ,don't worry y- intercept is nothing

but the tangent line cuts the y -axis at some value which is our case is 'b ' 

see the diagram below ,how we values are arrived one by one

jus a recap 

1 . random  x0 , get the yo from y = x-a

2. find the tangent line equation 

3. find x1 

since slope m = y1-y0/x1- x0 

re arranging the equation x1 -x0  = y1 -y0 /m

m is the derivative dy/dx or f '(x) ,so 

x1 = x0 +  (y1- y0)/f' (x)

since from the findings we have y1= 0 from the graph 

x1 = x0 + (0 - y0)/f '(x)

y0 = f(x0) 

x1 = x0 - f(x0)/ f '(x) is the Newton–Raphson method.

check out the sample page too for a better understanding of the equation

and the results.

in the upcoming lessons we will talk about failures of the method.