Intelligent Defect Inspection Powered by Computer Vision and Deep Learning

Source: Deep Learning on Medium

Originally published on the Infopulse blog.

Quality control has been a key factor in manufacturing since Henry Ford first introduced the assembly line system. The idea was that quality could be achieved by a mass production assembly line, so long as every individual did their part in the process. We have obviously come a long way since then. The traditional assembly line still exists in many forms, even though machinery has replaced many manual processes.

However, quality control remains an issue. And it’s a costly one: many companies experience quality-related costs as high as 15%-20% of sales revenue. In some cases, those can even go as high as 40% of the total number of operations. The European Commission also estimated that in some industries 50% of production can be scrapped due to defects; the defect rate can reach a staggering 90% in complex manufacturing lines.

The biggest problem for manufacturers is that even a slight variance in production processes or materials (invisible to the human eye) can make the entire production run defective. Surely, those parts will not reach the end users due to extensive post-production quality checks. However, relying on current (mostly manual) defect inspection practices means that thousands of items can be manufactured at significant expense before the defect is discovered.

Moving from Manual to Automated Optical Inspection

As mentioned already, manual inspection of products, parts, and components can be a cumbersome and expensive process. Firstly, it involves significant training for human experts to perform such inspections. Secondly, such inspections can cause bottlenecks in the production/time to market timelines. Thirdly, manual inspections do not scale as products do — further training is required to have enough specialists for performing timely inspections.

As a replacement for more cumbersome and error-prone manual inspection emerged the automated optical inspection (AOI), powered by machine vision. The new types of AOI systems are equipped with multi-cameras ranging from simple XGA (Extended Graphics Array) units to high-resolution, multiple-megapixel video sensors. Depending on the camera type, an AOI system can either provide monochrome or color images of the inspected items, and the captured images can span a wide range, from mere thousands of data points to millions of data points.


The benefits of automated optical inspection are multifold:

  • Such systems enable early error detection in the manufacturing processes and help ensure high quality of the item before it’s moved to the next manufacturing step.
  • AOI helps gather historical and production statistics used to improve manufacturing lines.
  • As a result, it will help reduce material waste, repair and rework costs, as well as added manufacturing labor time and expenses.

AOI systems can be programmed to use a different technique for quality assurance and defect inspection such as:

  • Template matching: the system is programmed to compare the obtained item image with the image of perfectly made, non-defective items. The system first learns about all the correct attributes of a certain part of the item and then assesses the quality of a produced item according to the estimated standards.
  • Pattern matching: the system stores information of both good and bad assemblies, comparing and contrasting the actual product versus available patterns.
  • Statistical pattern matching: In this case, the system stores the results of several products and several types of defects, so that it becomes capable of greenlighting acceptable minor deviations without flagging errors.

Automated optical inspection systems have been a major breakthrough in quality control, providing more accurate and rapid inspections throughout the production process. However, it is still not the pinnacle of automated defect inspection.

By incorporating AI and deep learning, not only can the optical inspection process images of already produced products, but also identify the defects and over time, learn more about different types of defects (without explicit programming). Ultimately, it will be targeted at conducting predictive analyses and thus, reaching error-free production.

How AI and Deep Learning Can Further Improve the Visual Inspection Process

While automated optical systems are a powerful method of inspecting defects, they are still relatively slow, inaccurate and expensive in maintenance. With the rapid transition towards Industry 4.0., most companies can no longer lose time and resources on long-term setups. For instance, final assembly verification is extremely difficult to program due to numerous variables that can be hard for an AOI to isolate (e.g., lighting, changes in curvature, color, etc.). Though statistical pattern matching can help tolerate some variability in items’ appearance, complex surface textures and image quality issues can pose some serious inspection challenges.

In addition, machine vision systems typically cannot cope with the following issues:

  • They cannot properly distinguish the variability and deviation between visually similar parts.
  • Also, they struggle to properly differentiate between “functional” defects — that are almost always a cause for rejection; and “cosmetic” defects — some mild issues with the part’s overall look that are not viewed as critical by the manufacturer.

Computer vision and deep learning-based systems have emerged as a powerful alternative to AOI systems, addressing the aforementioned shortcomings.

What is computer vision?

Computer vision is a subfield of artificial intelligence and machine learning based on specific algorithms and other methods allowing computers to understand the content of digital images. In short, computer vision software attempts to reproduce the capability of human vision.

There are two specific tasks most computer vision tools are aimed at solving:

  • Object classification: a model is trained on a dataset of specific objects (e.g., images of defects) and then it classifies new as belonging to one or more of your training categories.
  • Object identification: a model is trained to recognize a specific instance of an object. For instance, when it determines two components in an image it can tag one as a circuit board and another as a microcontroller.

Here’s how computer vision works in simple terms:


The interpreting device (computers + software) is the element that performs most of the work in this case. Models, trained with the help of machine learning and deep learning techniques, deconstruct the incoming visual data into pixels, then assess them according to various parameters and compare to other images in the dataset to find the best “match” and make a valid prediction of what it is.

At this point, it’s important to recap what is deep learning.

“Deep learning is a subdivision of machine learning with a strong emphasis on teaching computers to learn like humans: by being presented with an example.”

Unlike machine learning, deep learning models do not need to be constantly programmed with explicit instructions for analyzing data. Typically, such models are only presented with a large dataset of relevant information and some initial parameters for operationalizing the data. They can churn such data and self-learn to predict which output (e.g., a classification) is accurate or not.

If you want to learn more about deep learning, refer to our Executive Guide to Deep Learning and Neural Networks for Businesses.

Neural networks and deep learning models can help overcome the current limitations within AOI systems.


Deep learning-based defect inspections are particularly effective when it comes to assessing complex surfaces and detecting cosmetic defects such as scratches or dents. As well, such systems have the ability to inspect more precisely or classify features of certain items based on their defining characteristics — even if those characteristics vary in subtle but acceptable ways.

Additionally, they are more adaptable when it comes to analyzing hard-to-capture visual elements. A good example is one of the recent projects Infopulse team developed together with a German manufacturer. They managed to come with an IoT device capturing and recognizing digits on old gauges despite image defects from 7-segment LCD display, such as glares, white spots, reflections of objects or people, etc. For this, our team had to build a convolutional neural network (CNN) that could capture and process the image in less than 2 seconds, saving the client a lot of operational expenses on manual data collection. Similar computer vision solutions can also be implemented to analyze different types of visual defects.

Computer vision algorithms and applications can be powered both by deep learning or machine learning. The latter is often a more suitable choice for performing assessments related to gauging, measuring, and performing a precision alignment. However, deep learning systems can greatly complement ML-based ones as it allows computerizing visual and sound inspections that previously required specific human expertise. This technology broadens the limits of what a computer and camera/sensors can accurately inspect.

Read more on the Infopulse blog.