Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases now! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Conferences
Free Learning
Arrow right icon
Arrow up icon
GO TO TOP
Mastering Machine Learning Algorithms

You're reading from   Mastering Machine Learning Algorithms Expert techniques for implementing popular machine learning algorithms, fine-tuning your models, and understanding how they work

Arrow left icon
Product type Paperback
Published in Jan 2020
Publisher Packt
ISBN-13 9781838820299
Length 798 pages
Edition 2nd Edition
Languages
Tools
Arrow right icon
Authors (2):
Arrow left icon
Giuseppe Bonaccorso Giuseppe Bonaccorso
Author Profile Icon Giuseppe Bonaccorso
Giuseppe Bonaccorso
Giuseppe Bonaccorso Giuseppe Bonaccorso
Author Profile Icon Giuseppe Bonaccorso
Giuseppe Bonaccorso
Arrow right icon
View More author details
Toc

Table of Contents (28) Chapters Close

Preface 1. Machine Learning Model Fundamentals 2. Loss Functions and Regularization FREE CHAPTER 3. Introduction to Semi-Supervised Learning 4. Advanced Semi-Supervised Classification 5. Graph-Based Semi-Supervised Learning 6. Clustering and Unsupervised Models 7. Advanced Clustering and Unsupervised Models 8. Clustering and Unsupervised Models for Marketing 9. Generalized Linear Models and Regression 10. Introduction to Time-Series Analysis 11. Bayesian Networks and Hidden Markov Models 12. The EM Algorithm 13. Component Analysis and Dimensionality Reduction 14. Hebbian Learning 15. Fundamentals of Ensemble Learning 16. Advanced Boosting Algorithms 17. Modeling Neural Networks 18. Optimizing Neural Networks 19. Deep Convolutional Networks 20. Recurrent Neural Networks 21. Autoencoders 22. Introduction to Generative Adversarial Networks 23. Deep Belief Networks 24. Introduction to Reinforcement Learning 25. Advanced Policy Estimation Algorithms 26. Other Books You May Enjoy
27. Index

Optimization algorithms

When we discussed the back-propagation algorithm in the previous chapter, we showed how the SGD strategy can be easily employed to train deep networks with large datasets. This method is quite robust and effective; however, the function to optimize is generally non-convex and the number of parameters is extremely large.

These conditions dramatically increase the probability of finding saddle points (instead of local minima) and can slow down the training process when the surface is almost flat (as shown in the following figure, where the point (0, 0) is a saddle point).

Example of saddle point in a hyperbolic paraboloid

Considering the previous example, as the function is f(x,y) = x2y2, the partial derivatives and the Hessian are:

Hence, the point the first partial derivatives vanishes at (0, 0), so the point is a candidate to be an extreme. However, the Hessian has the eigenvalues that are solutions of the equation , which leads...

lock icon The rest of the chapter is locked
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Renews at €18.99/month. Cancel anytime