Source: Deep Learning on Medium
Building a mixed-data neural network in Keras
In order to iterate on model versions, it’s good practice to do this in the form of functions. I wrote the following functions to pre-process the structured data and create the mixed-data neural network architecture.
- Pre-processing the structured data:
- Using the Keras Sequential API to create a two-layer multi-layer perceptron (a simple feedforward neural network) for the structured data branch:
- Creating a convolutional neural network for the image data branch (adapted from here):
After train-test splitting, labelling and pre-processing the data, the final steps are to build the branches, concatenate the outputs, and use them as inputs in the final neural network layer head:
Iteration and evaluation
I experimented with a few different versions of this model, using different numbers and sizes of layers, types of optimisers and regularisation, and training lengths.
Behold, my final mixed-data neural network model architecture:
The best model I was able to produce had an average F1 score (the harmonic mean of precision and recall — chosen because both measures of accuracy are important in this case) of 0.8, and no overfitting.
This mixed-data model performed slightly better than the best version of the CNN-only model (0.77), which only used satellite images as inputs and did not include the extra structured data. Augmenting the satellite imagery data with the structured data that was also available for the areas did indeed improve accuracy.
However, if a model had to be chosen to put into production, based on the current best versions, it might be preferable to use the CNN-only model, due to the extra data processing and computation power involved in the mixed-data model.
However, there is plenty of room for improvement in the mixed-data model, including tuning of both the CNN and MLP components (e.g. using pre-trained weights in the CNN), as well as the final fully-connected layers. So the accuracy could be increased even further.
Keras has some cool functionality in its Functional API for building neural networks that can take multiple different forms of data as inputs. I’ve shown an example here of combining both structured data and image data to predict the locations of traffic accidents.
There’s plenty to play around with in Keras beyond this — there’s no reason to limit yourself to only two branches, or to structured and image data. Feel free to fork my GitHub repo for this project and have a play around, or have a read through the Keras docs to learn more.
Although mixed-data neural networks are a bit more complicated to build — and are certainly a more niche area of general data science — they can be a great solution to your more complex neural network needs.