Using Metal Performance Shaders for RealTime Fast Style Transfer.

Original article was published on Deep Learning on Medium

iOS and iPhones

Currently iPhones it’s enough powerful devices, they contains GPU and API for heavy computations on it.

Metal Performance Shaders

Neural Networks can approximate any function into matrix multiplications, for accelerate these massive calculations we should use GPU. Direct access to GPU can provide Metal.

Metal — API for direct access to GPU. We can use it for advanced hi-performace graphics, 3D, heavy computations.

Metal Performance Shaders it’s collection of different shaders implementations which we can use in our’s application.

This library also contains Neural Networks implementations.

Why not CoreML?

CoreML is universal framework, but it’s internal approach cover all very decrease performance, because it on every layer perform load/unload results of computation from GPU to CPU and back.

It’s not so big problem for image classification task, because for classification on every layer you have less and less. For Style Transfer on some layer you decrease data, but then neural network upscale resolution. In CoreML you can processing only via MLMultiArray object as adapter, it’s CPU memory.

At practice CoreML processing one frame for this task about ~20 seconds. It’s very slow.

Also I got few problems with generation from PyTorch model to CoreML model. Some layers was absent.