Combine Image & Tabular data in one Model using PyTorch

Original article was published by Soumo Chatterjee on Deep Learning on Medium


Combine Image & Tabular data in one Model using PyTorch

The following blog will help the readers to understand how we can combine image and tabular data together in PyTorch using deep learning and generate predictions from the model . I will go step by step to understand the whole end to end scenario easily : →

#1# Create a custom dataset class →

I am going to use here OpenVaccine: COVID-19 mRNA Vaccine Degradation Prediction dataset . After cleaning and processing the tabular data I will create the dataset class for our data as follows……

As there are a lot of codes and syntaxes , I recommend reading the codes with respect to the comments given beside each syntax written . The steps involved are: →

  1. First we declared some of the specific libraries that we will need .
  2. we will define the constructor for the dataset class that we created.
  3. then we will define the __len__ function to calculate the length of the dataset
  4. then we will define the __getitem__ function extract the input and the targets that we will be feeding into our model that we will create.

#2# Create object for dataset class and dataloaders →

#3# Create model class →

I have commented the code syntaxes as much as possible , but reader if you still feel that you are unable to understand any part of the code please please please feel free to comment on this article , I will be very glad to answer your doubts .

Still there are 3 major syntaxes in this present in this at line no. 15, 21 & 56 where we combine our tabular and image data layers to generate predictions out of it .

15 self.image_dense_layer_2 = nn.Linear(512, 1) # in constructor 21 self.tabular_dense_layer_4 = nn.Linear(4, 1) # in constructor56 x = torch.cat((x, tab), dim=1) # in forward function

#4# Define loss function , optimizer and training loop for our model.

After model creation we will define the following things easily .

Please consider going through my notebook I have written on this topic once , so that you can move ahead with ease .

I hope you have understood the logic behind how we can combine image data & tabular data in a deep learning model using pytorch . Let me know , if you have any Questions , Comments or Concerns regarding this in the comment section ; Until then enjoy LEARNING……………….

For more awesome data science related blog posts check out the top data science bloggers on medium.