AI at the Edge- An introduction to Intel OpenVINO Toolkit

Source: Deep Learning on Medium

AI at the Edge- An introduction to Intel OpenVINO Toolkit

We now live in the age of cloud computing. Everything is on the cloud now, Your data, your processing power- everything. Cloud services like AWS, Azure, GCP etc have made it very easy for your IoT devices to make up for the lack of processing power in local machine and use the processing power in the cloud and serve you with the amazing capability of AI. But not in every case, you can rely to cloud services. There is always a risk of leaking your sensitive personal data if you send it to the cloud. There might be network issue or latency issue where you want to deploy your AI model for real-time decision making, for example, a self-driving car. You really don’t want your self-driving car to wait for a response from the server while it is driving. Or there might be even situation where network is not available at all. That’s where the AI at the edge concept comes in to help.

What is AI at the Edge

The growth of IoT devices has increased the edge application of AI. We are now surrounded by many smart devices- mobile phones, smart speakers, smart lock and so on. Though they are pretty smart, they do not possess powerful processing capacity. So there are lots of target devices with limited hardware resources where you might want to deploy an AI model. The edge means local processing. That means you can use an AI model in a device and use its processing power to make decisions without connecting to a cloud service.

Edge is not a replacement of cloud computing though. You may still need to train your model in a cloud environment when high processing power is needed (or in a powerful desktop). After training is complete, you can optimize your trained model for your device, deploy it in your edge application and perform inference without the need for a connection to the server. Optimization software can greatly help in optimizing the trained model for the multiple target hardware to increase its efficiency and maximize its performance. In this (series of) blog, I will talk about Intel OpenVINO toolkit software which makes edge deployment of different computer vision models very easy and efficient.

What is Intel OpenVINO

The name OpenVINO stands for Open Visual Inferencing and Neural Network Optimization. It is an open source software developed by Intel. The main focus of OpenVINO is to optimize neural networks for fast inference across various Intel hardware like CPUs, GPUs, VPUs, FPGAs, IPUs etc. with a common API. OpenVINO software optimizes the size and speed of a model so that it can run at the edge with the limited hardware resources. It does not increases the model accuracy though. That kind of optimization is done in the model training step. Rather, you might sometimes need to opt for a reduced accuracy for higher performance at the edge. For example, selecting 8-bit integer precision instead of 32-bit floating point precision.

Getting Set up

For using OpenVINO you must know Python or C++. You can follow the instruction for installation of OpenVINO toolkit on your local machine in the official documentation. I use a windows machine. Here was my steps,

1. I had python installed using anaconda distribution (I select “add to path” option when prompted during installation, even though the installer will not recommend it. This option allows you to access python from command prompt which we will use.).

2. I installed keras previously running the following command in anaconda prompt which installed tensorflow 2.0 with it. (note this. you will see in point 8 why I have mentioned it.)

conda install -c conda-forge keras

3. I had PyTorch installed running the following command in anaconda prompt

conda install -c pytorch pytorch

4. Then I downloaded the OpenVINO installer exe file going to this link, selecting my operating system which is windows and then registering for Intel.

5. The installer pointed out my missing requirements which were CMake and Microsoft Visual Studio.

6. I installed CMake following the instructions in this video.

7. Downloaded Microsoft Visual Studio 2019 and installed it. Followed the steps mentioned in this page (you have to install some workloads with VS otherwise things won’t work, so must go through the instructions in the linked page)

8. Now, after that I followed the documentation that opens after finishing the installation of openVINO. But then I noticed my TensorFlow version downgraded to 1.15.2. because, at the time of this writing, openVINO supports TensorFlow versions that are bellow 2.0. So, here is what you gonna do if you don’t want openVINO to downgrade or change other packages. Create a virtual environment first.

(if you don’t have “virtualenv” package installed already, run the first command in the following code to install it. otherwise, skip to the 2nd line which create the virtual envirnoment. “openvinoenv” is the name of the environment that I chose, you can name it differently if you want)

pip install virtualenvvirtualenv --system-site-packages -p python openvinoenv

9. Activate the virtual environment.


10. Change your directory to <openVINO_installation_directory>\openvino\deployment_tools\model_optimizer. In my case,

cd "C:\Program Files (x86)\IntelSWTools\openvino\deployment_tools\model_optimizer>"

11. You will see, the folder has some text files, one of them is “ requirements.txt”. Run the following command,

pip install -r requirements.txt

12. If you check (by running “pip list”), you will see that TensorFlow is downgraded to 1.15. But if you check the TensorFlow version outside of you Virtual environment, the version will remain the same as earlier (2.0 in my case). This is the benefit of using the virtual environment which will isolate the requirements of openVINO in a separate python sandbox and install them separately without modifying your global packages.

13. Next, change your directory to one step back and go to “demo” folder.

cd ..
cd demo

14. Run the “demo_benchmark_app.bat” file. This is important as it will install some other missing components for you the first time you run it. Type this in the command line.


You will see a benchmark result. That means, everything worked fine. Your openVINO setup is ready to use. If an error occurred, there is probably a package missing. You will see the information of the missing package in the error message. Install the package manually.

Remember, every time you use OpenVINO, you have to run the setupvars.bat file in the <installation_directory>\openvino\bin folder. So, don’t forget to run the following command every time you start working with openVINO (if you installed it in a different location, use the location of setupvars.bat in your machine) after activating your virtual environment.

"C:\Program Files (x86)\IntelSWTools\openvino\bin\setupvars.bat"

When finished, deactivate the environment by simply running,


Workflow with OpenVINO

There are four steps to work with OpenVINO.

1. Obtain a pre-trained model.

2. With model optimizer, optimize the pre-trained model and create intermediate representation (IR)

3. Perform the actual inference (predicting) with the Inference Engine

4. Handle the output of the Inference Engine properly

A pre-trained model can be obtained from the OpenVINO’s model zoo, which I will talk about explicitly in the next post, part 1. Some models are already prepared in the intermediate representation, for them, you don’t need to perform any optimization and can directly jump to inference step by feeding the model to the inference engine. Some models on the other hand need further optimization before transforming into the intermediate representation. If you created your own model with TensorFlow or Caffe or other deep learning libraries, you can convert your pre-trained model with the model optimizer. I will talk about this conversion in part 2. Then finally, in part 3, we will use the inference engine to perform the inference and handle the output to an example app.

So without further adieu, get your environment set up and follow me to the next post.