Building Blocks of Neural Networks

Source: Deep Learning on Medium

Prayers of Deep Learning: Part3

Building Blocks of Neural Networks

An Introduction to tensors and its operations

In the previous article, we have discussed the basic architecture of Machine Learning/Deep learning projects. Now, we are going to concentrate more on deep learning concepts. So if you have missed my previous articles, here is the link to my profile to read those contents — IJAS A H

Prayers of Deep Learning Cover

We know that data is the basic requirement to build-up any learning model. But simply raw data doesn’t meet our requirements. We need to represent data in some specific format to feed as input to neural networks. So let’s discuss Data Representations for Neural Networks.

Data Representations

In deep learning, whatever data we process(text, audio, video, etc.) we first convert data into tensors. This process of converting the data into tensors is called data vectorization. So what are tensors? Tensors are data stored in multidimensional-array.

Note: Here data stored in the multidimensional array will be a numerical representation of actual data.

Tensors can have different dimensions based on the data. So let’s explore some tensors with different dimensions. Here we use the Numpy library in python for creating tensors.

0-Dimensional tensor: 0-dimensional tensors are also called scalar. In Numpy, float32 or float64 number is a scalar-tensor.

Here ndim is used to display the number of axes.

import numpy as np
x = np.array(100)
print("Array",x)
print("dimension",x.ndim)
output
Array 100
dimension 0

One dimensional tensor: An array of numbers can be considered as one-dimensional tensor(1D tensor, also called vector).

import numpy as np
x = np.array([1,5,2,7,11,24,25,12])
print(“Array:”, x)
print(“Dimension:”, x.ndim)
Output
Array: [ 1 5 2 7 11 24 25 12]
Dimension 1

Two-dimensional tensor: A two-dimensional tensor is a collection of one-dimensional tensors.

import numpy as np
x = np.array(
[
[1,5,2,7,11,24,25,12],
[1,2,3,4,5,6,7,8]
]
)
print(“Array:”, x)
print(“Dimension”, x.ndim)
Output
Array: [[ 1 5 2 7 11 24 25 12] [ 1 2 3 4 5 6 7 8]] Dimension 2

Multidimensional tensor: Collection of the two-dimensional tensors produce three-dimensional tensor, again the collection of three-dimensional tensors produces a four-dimensional tensor and this goes on.

Fig 1: Graphical Representation of tensor with different dimensions

Real-world tensors

We have already seen the representation of data as tensors, now let’s explore some real-world tensor representation to understand the concept further.

Image data — Images are 4D tensors. So in image related deep learning problems image data will be represented as 4D tensors with shape(samples, height, width, channels)

Video data — Videos are represented as 5D tensors with shape(samples, frames, height, width, channels)

Tensor Operations

Fig 2: Element-Wise Tensor Operations

As shown in the above figure tensors we can add, subtract multiply and divide the tensors in an element-wise fashion.

The element-wise operation of two tensors with the same dimensions results in a new tensor with the same dimensions where each scalar value is the element-wise operation(add, subtract, multiply, divide) of the scalars in the parent tensors.

Now, let’s see how we implement these tensor operations using numpy.

import numpy as npA = np.array([
[[1,2,3], [4,5,6], [7,8,9]],
[[11,12,13], [14,15,16], [17,18,19]],
[[21,22,23], [24,25,26], [27,28,29]],
])
B = np.array([
[[1,2,3], [4,5,6], [7,8,9]],
[[11,12,13], [14,15,16], [17,18,19]],
[[21,22,23], [24,25,26], [27,28,29]],
])
print(“Array A”,A)
print()
print(“Array B”,B)
print()
print("Array A+B:",A+B)
print()
print("Array A-B:", A-B)
print()
print("Array A*B:", A*B)
print()
print("Array A/B:", A/B)
OutputArray A [[[ 1 2 3] [ 4 5 6] [ 7 8 9]] [[11 12 13] [14 15 16] [17 18 19]] [[21 22 23] [24 25 26] [27 28 29]]]Array B [[[ 1 2 3] [ 4 5 6] [ 7 8 9]] [[11 12 13] [14 15 16] [17 18 19]] [[21 22 23] [24 25 26] [27 28 29]]]Array A+B: [[[ 2 4 6] [ 8 10 12] [14 16 18]] [[22 24 26] [28 30 32] [34 36 38]] [[42 44 46] [48 50 52] [54 56 58]]]Array A-B: [[[0 0 0] [0 0 0] [0 0 0]] [[0 0 0] [0 0 0] [0 0 0]] [[0 0 0] [0 0 0] [0 0 0]]]Array A*B: [[[ 1 4 9] [ 16 25 36] [ 49 64 81]] [[121 144 169] [196 225 256] [289 324 361]] [[441 484 529] [576 625 676] [729 784 841]]]Array A/B: [[[1. 1. 1.] [1. 1. 1.] [1. 1. 1.]] [[1. 1. 1.] [1. 1. 1.] [1. 1. 1.]] [[1. 1. 1.] [1. 1. 1.] [1. 1. 1.]]]

Broadcasting

We have discussed numpy operation on tensors with identical shapes. Now let’s discuss the operation on tensors with different shapes. Subject to certain constraints, the smaller array is “broadcast” across the larger array so that they have compatible shapes. Broadcasting provides a means of vectorizing array operations so that looping occurs in C instead of Python. It does this without making needless copies of data and usually leads to efficient algorithm implementations.

Numpy operations are usually done in pairs of arrays on an element by element fashion. In other words, two tensors must have the same shape.

import numpy as np
a = np.array([1.0, 2.0, 3.0])
b = np.array([2.0, 2.0, 2.0])
print(a * b)
Output
array([2., 4., 6.])

Numpy broadcasting rule relaxes this when shapes of arrays meet certain constraints. A simple example is the multiplication of a scalar value with vector.

import numpy as np
a = np.array([1.0, 2.0, 3.0])
b = 2
print(a * b)
Output
array([2., 4., 6.])

Here, the above two code snippet produces the same output, the first one is by using simple tensor multiplication and the second one is by making use of tensor broadcasting.

for more information on broadcasting please visit: Broadcasting

Tensor Product

The tensor product operation is not the same as element-wise tensor multiplication. Given, a tensor with q dimension and another tensor with r dimension the product of two tensors will produce a new tensor having dimension q + r.

Fig 3: Tensor dot Operation

To compute the tensor product of A and B, each element in tensor A is multiplied with tensor B.

Now, let’s see how we can implement tensor dot operation using numpy

we use tensordot(tensor A, tensor B, axes =’Value’) function in numpy for this purpose

Three common use cases for axes value are:

axes = 0 : tensor product

axes = 1 : tensor dot product

axes = 2 : (default) tensor double contraction

Since we are trying to find the tensor product we specify axes value as ‘0’

import numpy as np
A = np.array([2,24])
B = np.array([5,8])
C = np.tensordot(A,B,axes = 0)
print(C)
Output
[[ 10 16] [120 192]]

References

  1. Tensor on Wikipedia
  2. Tensor product on Wikipedia
  3. Tensor algebra on Wikipedia
  4. A Gentle Introduction to Tensors for Machine Learning with NumPy
  5. numpy.tensordot