How to run Tensorflow.js on a serverless platform : deploying models

Original article can be found here (source): Artificial Intelligence on Medium

How to run Tensorflow.js on a serverless platform : deploying models

This is the last part of a 3 articles serie.

In the first part, we introduced neural networks and TensorFlow framework basics.

In the second part, we explained how to convert existing models from Python to TensorFlow.js

Finally, we present, through an example, how to use an online TensorFlow.js model and deploy it rapidly using our WarpJS JavaScript Serverless Function-as-a-Service (FaaS).

Using an online model with TensorFlow.js

Many public models can be retrieved from web databases.

We’ll use the “toxicity” pre-trained model in the next sections as an example.

The toxicity model detects whether text contains toxic content such as threatening language, insults, obscenities, identity-based hate, or sexually explicit language. The model was trained on the civil comments’ dataset: https://figshare.com/articles/data_json/7376747 which contains ~2 million comments labeled for toxicity. The model is built on top of the Universal Sentence Encoder (Cer et al., 2018).

Browser-based usage:

The model can be directly loaded for use in JavaScript at:

https://cdn.jsdelivr.net/npm/@tensorflow-models/toxicity

In the html, add:

<script src=”https://cdn.jsdelivr.net/npm/@tensorflow/tfjs"></script><script src=”https://cdn.jsdelivr.net/npm/@tensorflow-models/toxicity"></script>

Then, in the JS code:

// sets the minimum prediction confidenceconst threshold = 0.9// load and init the modelconst model = await toxicity.load(threshold);. . .// apply an inferenceconst predictions = await model.classify(inputText);. . .

Node-based usage:

Toxicity is also available as a NPM module for Node.js (package that actually loads the model from the storage link above):

$ npm install @tensorflow-models/toxicity

Then, in the JS code:

const toxicity = require(‘@tensorflow-models/toxicity);// sets the minimum prediction confidenceconst threshold = 0.9 // sets the minimum prediction confidence// load and init the modelconst model = await toxicity.load(threshold);. . .// apply an inferenceconst predictions = await model.classify(inputText);. . .

Deploying a model with WarpJS

As discussed before, inference on big data sets in the browser comes rapidly short in terms of performance due to model and data loading time as well as computing capabilities (even with accelerated backends).

Node.js allows to push further the performance limit by deploying on a high performance GPU engine and in the network neighborhood of the dataset, but the user will face complexity when trying to address distributed processing for the next performance step.

The WarpJS JavaScript FaaS enables easy serverless process distribution with very little development effort.

Example: toxicity model serverless deployment

WarpJS installation guidelines can be found here: Getting started with WarpJS

You can register to WarpJS with my invitation code (as I write those lines, WarpJS is currently in private Beta).

This article also provides a good tutorial on all steps to operate WarpJS.

In our WarpJS serverless operation, the browser acts as the primary input/output interface, through an index.html file.

It contains a text box to submit the input text to be analyzed and a “classify” button triggering the inference process.

<!DOCTYPE html><body><h1>TensorFlow.js toxicity demo with WarpJS</h1><form id=”form”><input id=”classifyNewTextInput” placeholder=”i.e. ‘you suck’” required><button>Classify</button></form><p id=”result”></p></body></html>

The index.js file (see below) contains the inference function (highlighted) to be run at each user input and distributed on serverless infrastructure. WarpJS is a function-as-a-service platform for JavaScript. Instead of creating HTTP endpoints and use HTTP calls to do a remote inference, we just have to tell WarpJS to manage the execution of a JavaScript function on its FaaS just by Warp calling it (warp.call), which is very similar to a JavaScript call. So in our case we Warp-call the classify function that will be run on the WarpJS FaaS.

index.js:

/** Copyright 2020 ScaleDynamics SAS. All rights reserved.* Licensed under the MIT license.*/‘use strict’// import WarpJS moduleimport { defaultWarper as warper } from ‘@warpjs/warp’import engine from ‘@warpjs/engine’// init WarpJSengine.init()// warp prediction functionconst classify = async inputs => {‘warp +server -client’// predict with tensorflow modelconst predictions = await model.classify(inputs)// check toxicity resultsconst toxic = predictions.some(({ results }) => results[0].match !== false)return toxic}// main script// listen to button click eventdocument.getElementById(‘form’).addEventListener(‘submit’, async event => {event.preventDefault()result.innerHTML = ‘<h2>Remote inference running</h2>’// scan textbox contentconst text = classifyNewTextInput.value// invoke inferenceconst toxic = await warper.call(classify, [text])// render result on html pageif (toxic) {result.innerHTML = `<h2 style=”color:red”>Your sentence is TOXIC :(</h2><img src=”/img/Pdown.png” alt=””>`} else {result.innerHTML = `<h2 style=”color:green”>Your sentence is NON TOXIC :)</h2><img src=”/img/Pup.png” alt=””>`}})

As all FaaS, functions executed serverless are stateless. Instead of loading and initializing the model at each function request, WarpJS provides an easy way to add some initialization on the server that will run the functions.

To do it, we add in init-server.js the initialization of TensorFlow.js and toxicity model. When done, we set global.model, so the functions can use the model directly in their code.

init-server.js:

/** Copyright 2020 ScaleDynamics SAS. All rights reserved.* Licensed under the MIT license.*/require(‘@tensorflow/tfjs’)require(‘@tensorflow/tfjs-node’)const toxicity = require(‘@tensorflow-models/toxicity’)// The minimum prediction confidenceconst threshold = 0.9// Load the modellet modelLoaded = falsetoxicity.load(threshold).then(model => {global.model = modelmodelLoaded = true})// Force waiting for the async TensorFlow model load.// The “deasync” lib turns async function into sync via JS wrapper of Node event loop.// The “loopWhile” function will wait for the condition resolution to continue.require(‘deasync’).loopWhile(() => !modelLoaded)

Deploying to the WarpJS FaaS is straightforward, just use “npm run deploy” to get the url of the deployed site and start playing with TensorFlow.js.

Feel free to access https://warpjs-744h4bixx1x93pg3oxc3hr4cf.storage.googleapis.com/index.html url to see the demo in action.

#MadeWithTFJS

Thanks!

About the author

Dominique d’Inverno holds a MSC in telecommunications engineering. After 20 years of experience including embedded electronics design, mobile computing systems architecture and mathematical modeling, he joined ScaleDynamics team in 2018 as AI and algorithm development engineer.