Building Your First Deep Learning Web Application With FastAI 2 (Local Ubuntu)

Original article was published by Christopher McBride on Deep Learning on Medium


Building Your First Deep Learning Web Application With FastAI 2 (Local Ubuntu)

I recently just completed the first couple of lessons of Jeremy Howard’s new fastai course (https://course.fast.ai/) where he shows how you can develop a powerful image classification deep learning neural network web application with the popular package. While he does an excellent job of going through the steps, there are always going to be some aspects that don’t exactly work as advertised, either due to having become outdated since the original publication or simply due to working in a different environment. Provided here is a step-by-step walk through of the full process along with potential pitfalls you may come across when building your own app.

Setting Up Your Environment

The first detail I’d like to point out is that the fast.ai package really does not work well in a Windows environment. I can tell you I already attempted to complete the initial setup on my Windows machine so that I could develop locally, and found it confusing and tedious. The recommendation in the course that you utilize a Notebook server such as Colab or Gradient is well-founded, and either choice is a very simple solution to getting started with fast.ai as quickly as possible. However, in my opinion, if you really intend to make full use of fast.ai I recommend setting up a local Linux system. For me, I setup a dual boot environment on my desktop so that I could switch over to Ubuntu 18.04. Setting this up is a process in of itself so if that’s your preferred approach I recommend following an existing tutorial on that before coming back here. The course recommends not doing this for good reason; it’s a long process, can be stressful if you don’t know what you’re doing, and it will undoubtedly take up more of your time to setup than actually building your deep learning web application. However, I maintain that it is worth it — especially if you have your own GPU.

From an Ubuntu environment, it’s very straightforward to acquire all the software dependencies for fast.ai. Note that the steps here describe the procedure in the simplest terms possible.

1. Install NVIDIA Drivers

This can be done by opening up Ubuntu terminal and using the command listed below and then rebooting your machine.

sudo ubuntu-drivers autoinstall

2. Install Anaconda

The Anaconda ‘sh’ file can be downloaded here: https://www.anaconda.com/products/individual and then installed by navigating to the downloaded file in the terminal (using the ‘cd’ command) and running the bash command (example below — version number/date may change).

cd /Downloads/
bash Anaconda3-2020.07-Linux-x86_64.sh

3. Install fastai

This can be done by running the command below pulled from the fastai GitHub.

conda install -c fastai -c pytorch -c anaconda fastai gh anaconda

And that’s it! You now have a working environment for development with fastai. With your environment, you should now be able to run jupyter notebook as a command in terminal and run all of the introductory code for the fastai course (found here https://github.com/fastai/fastbook in the /clean/ folder).

Signing Up For Azure Cognitive Services

If you do not intend to follow the data retrieval model set forth in the course, you may skip this section. But, the Bing Image Search API does make acquiring data extremely convenient.

Signing up for Azure Cognitive Services can be done here: (https://azure.microsoft.com/en-us/services/cognitive-services/). You should see an option to “Start free”. You can then

  1. Sign up for Azure Cognitive Services. In my sign up, I encountered a payment step and was required to input my card information despite only using their free services. Rest assured, as long as you select the “Free” options wherever asked, you will not be charged.
  2. Log into the Azure portal and select “Cognitive Services”.
  3. You will then be able to select “Create a resource”.
  4. Search for “Bing Image Search” in their library and create it.
  5. You will move through several forms, select the Free payment tier option to ensure you won’t be charged for this particular resource.
  6. After filling out all the forms, it will deploy and you can select “Go to Resource” where it will display an API key (or two). This key is all you will need to utilize the Bing image search functionality in your code.

Gathering Data

For my application, I decided to attempt to classify different tomato varietals, as I had several types growing in my garden at the time, and I decided it would be interesting to test my application with them. The code for training the model and the web application code that I deployed on mybinder.org can be found here: https://github.com/QuantumAbyss/TomatoApp. All code used comes directly from the fastai course material with modifications to classify tomatoes instead of bears.

My code begins with a pip install command to ensure I have azure-cognitiveservices available followed by four import statements that make the fastai functions available in the notebook.

pip install azure-cognitiveservices-search-imagesearch
from fastai.vision.all import *
from fastai.vision.widgets import ImageClassifierCleaner
from azure.cognitiveservices.search.imagesearch import ImageSearchClient as api
from msrest.authentication import CognitiveServicesCredentials as auth

The course material includes a utils.ipynb notebook which contains the search_images_bing function that I included directly into my workbook. Note that the “count” argument corresponds to how many images you’d like to download for each search query.

def search_images_bing(key, term, min_sz=128):
client = api('https://api.cognitive.microsoft.com', auth(key))
return L(client.images.search(query=term, count=150, min_height=min_sz, min_width=min_sz).value)

With that you will be able to input your own API key and use that function to search for images of your choice. The code below defines the different types of tomatoes I wanted my app to distinguish between (just the varietals I had in my garden — tomato_types). It will then create a directory “tomatoes” along with several sub-directories labeled with the tomato type. These sub-directories will be populated with images based on your search query.

key = 'XXX'tomato_types = 'wild red cherry','pink brandywine','san marzano','cherokee purple','green zebra','amish plum','yellow pear'
path = Path('tomatoes')

if not path.exists():
path.mkdir()
for o in tomato_types:
dest = (path/o)
dest.mkdir(exist_ok=True)
results = search_images_bing(key, f'{o} tomato')
download_images(dest, urls=results.attrgot('content_url'))

The fastai library then has a very useful function, verify_images() which checks to make sure the data you acquired is in fact image data and you can then “unlink” any invalid data.

failed = verify_images(fns)
failed.map(Path.unlink);

Building And Training A Model

The main element of building a neural network with fastai is the DataBlock object. The DataBlock describes the different elements of your data and how you’d like it to be handled.

  • In the code below, the second line defines two block types, an ImageBlock (which primes the library to load image data) and a CategoryBlock (which tells the code to expect categorical labels assigned to each of the images).
  • The get_items argument is filled in with the fastai get_image_files function which will prime the library to load images from a specific directory (provided by the path variable in the code). Here, it is critical that our tomato folder and varietal sub-directories were created in the way that they were as the get_image_files function knows how to handle data saved in this way.
  • The splitter argument will simply specify how you’d like your data to be separated into training and validation sets. In this case we want it to be random and have validation data make up 30% of the total data set.
  • get_y needs to be supplied with a getter function for how it acquires the CategoryBlock labels.
  • item_tfms is an optional argument that is supplied with different types of data transform functions. In this case, we supplied it with the RandomResizedCrop function which will duplicate the images but with randomly selected cropped subsets of those images. This serves to augment our data such that our model will be better at handling images taken at different scales.
  • batch_tfms applies a transformation on batches of images. Here we use the aug_transforms function to provide more extensive data augmentation by distorting/shading/stretching the images.
tomatoes = DataBlock(
blocks=(ImageBlock, CategoryBlock),
get_items=get_image_files,
splitter=RandomSplitter(valid_pct=0.3, seed=25),
get_y=parent_label,
item_tfms=RandomResizedCrop(128, min_scale=0.2),
batch_tfms=aug_transforms(mult=2))
batch_tfms = aug_transforms effect on a single image

After creating the DataBlock you can then create a data loader which will feed into a convolutional neural network (cnn_learner). The fastai code below has the incredibly useful feature of utilizing transfer learning by taking an already trained image classification architecture (resnet34) along with all of its’ weights and “fine tuning” it (for seven epochs) based on the newly introduced tomato data.

dls = tomatoes.dataloaders(path)
learn = cnn_learner(dls, resnet34, metrics=error_rate)
learn.fine_tune(7)

You will then be able to review the error rate metric for each epoch, or utilize a confusion matrix to determine cases where your model may be failing. Gathering this information and understanding why it comes about is essential to optimizing your models predictive power

interp = ClassificationInterpretation.from_learner(learn)
interp.plot_confusion_matrix()
Confusion matrix for our tomato classifier

Fastai has several other features available for understanding the areas in which your model is failing in addition to procedures for cleaning/removing bad data. Details on these can be found in the code repository.

Once you feel you’ve gotten your model to a good point — where the error metric is low. You can then use the export function to save a .pkl file which contains all the model architecture and weight information to be used in a production environment.

learn.export()

Building A Web Application with Binder and Voila

Now we get to part where you get to deploy your classifier using

  • Voila — which builds python notebooks but ignores all code cells and displaying only markdown and widget objects.
  • Binder — a web service that allows you to launch a python notebook from its’ GitHub page and then access that active notebook from a browser. Note, they only really keep these web applications live for a couple hours, so you may need to look for another deployment option if you are seeking a more permanent solution.

The Tomato App code is here again for reference, but it begins by installing voila and importing the fastai libraries.

!pip install voila
from fastai.vision.all import *
from fastai.vision.widgets import*

Then we get into the app features which include a description of what the app does, an upload button, an on_click function describing what the upload button does, an image display section, and an output label which will be the result of our models prediction.

path = Path()
learn_inf = load_learner(path/"export.pkl", cpu=True)
btn_upload = widgets.FileUpload()
out_pl = widgets.Output()
lbl_pred = widgets.Label()
def on_click(change):
img = PILImage.create(btn_upload.data[-1])
out_pl.clear_output()
with out_pl: display(img.to_thumb(128,128))
pred, pred_idx, probs = learn_inf.predict(img)
lbl_pred.value = f'Prediction: {pred}; Probability: {probs[pred_idx]:.04f}'
btn_upload.observe(on_click, names=['data'])
display(VBox([widgets.Label('Select your tomato!'), btn_upload, out_pl, lbl_pred]))

Once you’ve tested your app you can then commit your changes to your GitHub. Note that binder can be buggy and it seemed to respond better when I included a “requirements.txt” in my GitHub repository. Also note that if you make your repository public, you should remove your API key in the public version.

Launching with Binder is extremely straightforward. Simply

  1. Go to https://mybinder.org/
  2. Enter your GitHub repository URL. For me that looked like “https://github.com/QuantumAbyss/TomatoApp
  3. Enter the path to a notebook URL (select URL in the dropdown). For me that was “/voila/render/TomatoApp.ipynb”
  4. Select “launch”. It will take several minutes to build.
  5. While launching, copy the link and paste it somewhere.

Once the application has launched you will be able to navigate to that link on your phone to upload photos directly from your camera or send it to friends to test out for fun. That’s it! You’ve now completed the introductory project for the FastAI course.