Skip to main content

What happens if gradient is set at 1 in "Gradient descent algorithm"

So, What is gradient descent algorithm ?

 - Given a feed forward netword we apply gradient descent as a fundamental function of Operation :
1. Randomly initialize : b, W1, W2, ... , Wm
2. Repeat untill convergence
3. Predict y(i) for each data point in training
4. Calculate loss J(b,w)
5. Calculate gradient of J(b,w)
     b(new)   =  b(old)   -a.gradient
    w1(new) =  w1(old)-a.gradient.w1
    w2(new) =  w2(old)-a.gradient.w2
    wm(new) =  wm(old) - a.gradient.wm
6. Update b,w1,w2,......,wm .. Simulateneously..

- When "Louis Augustin Cauchy" needed a function to find local minima he used idea of slope to iteratively move in direction guided by slope to reach local minima.
- Using the same idea in feed forward networks leads to convergence of minimum error.

Why gradient anyway ?

 So if gradient is set to 1 or in other words if we dont use gradient descent we reach to a point directly on x - axis which will not be the required local minima.

Comments

Post a Comment

Popular posts from this blog

ASCII to Decimal conversion

#include "msp430.h"                     ; #define controlled include file         NAME    main                    ; module name         PUBLIC  main                    ; make the main label vissible                                         ; outside this module         ORG     0FFFEh         DC16    init                    ; set reset vector to 'init' label         RSEG    CSTACK                  ; pre-declaration of segment         RSEG    CODE      ...
Post a Comment
Read more

Event Sourcing with CQRS.

  The way event sourcing works with CQRS is to have  part of the application that models updates as writes to an event log or Kafka topic . This is paired with an event handler that subscribes to the Kafka topic, transforms the event (as required) and writes the materialized view to a read store.
Post a Comment
Read more

Create One-Click Shutdown and Reboot Shortcuts

First, create a shortcut on your desktop by right-clicking on the desktop, choosing New, and then choosing Shortcut. The Create Shortcut Wizard appears. In the box asking for the location of the shortcut, type shutdown. After you create the shortcut, double-clicking on it will shut down your PC. But you can do much more with a shutdown shortcut than merely shut down your PC. You can add any combination of several switches to do extra duty, like this: shutdown -r -t 01 -c "Rebooting your PC" Double-clicking on that shortcut will reboot your PC after a one-second delay and display the message "Rebooting your PC." The shutdown command includes a variety of switches you can use to customize it. Table 1-3 lists all of them and describes their use. I use this technique to create two shutdown shortcuts on my desktop—one for turning off my PC, and one for rebooting. Here are the ones I use: shutdown -s -t 03 -c "Bye Bye m8!" shutdown -r -t 03 -c ...
Post a Comment
Read more