Deep learning in a large scale distributed system

Written by Jenaiz on 14 Jun 2015

Deep learning is interesting in many ways. But when you consider to do it in thousands of cores that can process millions of parameters, then the problem is more interesting and complex at the same time.

Google Datacenter (via Google)

Google was doing an interesting experiment, training a deep network with millions of parameters in thousands of CPUs. The goal was to train very large datasets without to limit the form of the model.

The paper describes the use of DistBelief, a framework created for distributed parallel computing applied to deep learning training. A collection of the features that the framework manage by itself are:

The framework automatically parallelizes computation in each machine using all available core, and manages communication, synchronization and data transfer between machines during both training and inference.

I couldn't find too much information about it, only what it is written in the paper.

They have applied two algorithms: SGD (Stochastic Gradient Descent) and L-BFGS. These algorithms usually works well, but they doesn't scale with very large data sets. That is because they introduce some modifications to them. The paper gives you more details about the optimizations in both algorithms that you can find interesting.

I was found really interesting the idea of distributed parallel computing working for very large datasets in such algorithms.

You can read "Large Scale Distributed Deep Networks", or if you are interested in the pdf version. Have fun!

Tags: Machine Learning papers Deep Learning google Distributed systems DistBelief

Deep learning in a large scale distributed system

Two interesting books to start with Machine Learning

There are a lot of books in the field of Machine Learning, just a fast search in Amazon gives you more than 25.ooo books. I wanted to filter all those books an choose the most useful. I was looking in google, quora and reading some post that I found around internet. There a lot of people giving a list of 10 - 20 books about machine learning, statistical learning, reinforcement learning... I just wanted to find the two interesting books to go into the field.

Smell the code by Jesús Navarrete