Federated Learning: An Introduction

Original article can be found here (source): Deep Learning on Medium

Machine Learning on Decentralized Data

Enter: Federated learning.

Federated learning is a model training technique that enables devices to learn collaboratively from a shared model. The shared model is first trained on a server using proxy data. Each device then downloads the model and improves it using data — federated data — from the device.

The device trains the model with the locally available data. The changes made to the model are summarized as an update that is then sent to the cloud. The training data and individual updates remain on the device. In order to ensure faster uploads of theses updates, the model is compressed using random rotations and quantization. When the devices send their specific models to the server, the models are averaged to obtain a single combined model. This is done for several iterations until a high-quality model is obtained.

source

Compared to centralized machine learning, federated learning has a couple of specific advantages:

  • Ensuring privacy, since the data remains on the user’s device.
  • Lower latency, because the updated model can be used to make predictions on the user’s device.
  • Smarter models, given the collaborative training process.
  • Less power consumption, as models are trained on a user’s device.

In order to ensure that an application’s user experience is not hampered, the model training happens when the user’s device is on free WiFi, is idle, and connected to a power supply.

TensorFlow Federated

TensorFlow enables the application of federated learning by leveraging its own framework.

TensorFlow Federated (TFF) is an open-source framework for machine learning and other computations on decentralized data.

TFF has two layers: the Federated Learning (FL) API and the Federated Core (FC) API. The Federated Learning (FL) API allows developers to apply federated training and evaluation to existing TensorFlow models.

The Federated Core (FC) API is the core foundation for federated learning. It is a system of low-level interfaces for writing federated algorithms in combination with distributed communication operations in strongly-typed functional programming environments.

An example application of federated learning is in Google’s Gboard keyboard.

In this case, a recurrent neural network language model is trained using decentralized on-device datasets. Its aim is to predict the next word on smartphone keyboards.

source

Training the model on client devices using the federated averaging algorithm is shown to perform better than server-based training using stochastic gradient descent. The algorithm is used on the server to combine updates from the clients and produce a new global model.

In this case, the use of federated learning led to a 24% increase in next-word prediction accuracy. Other improvements to the Gboard experience resulting from this are enhanced emoji and GIF predictions. This means that users are now seeing more relevant emojis and GIFs when using Gboard.