Federated Learning

Original article was published by Subhaditya Mukherjee on Deep Learning on Medium

Can I add this to my model?

Yes! There are many ways but the easiest by far is by using a library in Pytorch called Syft. It is as easy as adding a few lines to your code.

(Since the entire code is huge, I will only put snippets here. For the entire code, refer to my repository)

  1. Do look at this article by Open Mined
  2. Let us take a simple architecture with 2 Conv layers and 2 linear (fully connected) layers
  3. Import the library(along with the other pytorch ones) and add 2 users
import syft as sy
hook = sy.TorchHook(torch)
bob = sy.VirtualWorker(hook, id="bob") # person1
alice = sy.VirtualWorker(hook, id="alice") # person2
  1. Modify the pytorch train dataloader in this way. The test does not change because it is local.
train_loader = sy.FederatedDataLoader( # this is now a FederatedDataLoader
datasets.MNIST('/home/eragon/Documents/datasets', train=True, download=True,
transforms.Normalize((0.1307,), (0.3081,))

.federate((bob, alice)), # This will simulate a federated learning system
shuffle=True, **kwargs )
  1. Modify the main training loop like this
def train(args, model, device, train_loader, optimizer, epoch):
model.train() # Setting model to train
device = torch.device("cuda") # Sending to GPU
for batch_idx, (data, target) in tqdm(enumerate(train_loader)):
model.send(data.location) # Send to location (worker)
data, target = data.to(device), target.to(device)
optimizer.zero_grad() # Reset grads
output = model(data) # Passing batch through model

loss = F.nll_loss(output, target )

loss.backward() # Backprop
optimizer.step() # Pass through optimizer
model.get() # Get it back

if batch_idx % args.log_interval == 0:
loss = loss.get()
if args.dry_run:
  1. The test loop remains the same
  2. Train the model!!
  3. Write a blog post about it and cry when nobody reads it