How To Implement Custom Regularization in TensorFlow(Keras)

Original article was published on Artificial Intelligence on Medium

Putting It All Together

To see how well the implemented custom regularization technique works, we will build a simple neural network and perform the trivial task of image classification.

This section will include snippets of code that perform some common neural network implementation tasks, and also a GitHub repo link is available here that includes all code within this article.

To implement our neural network and load datasets, we will be utilizing the following tools and libraries: Keras, TensorFlow and NumPy.

import tensorflow as tf
from tensorflow import keras
import numpy as np

Load and partition dataset into testing, validation and training.

It is also required that the pixel intensity of the images within the dataset are normalized from the value range 0–255 to 0–1.

An array ‘class_names’ is initialized to hold the item labels of the pieces of clothing represented in the images within the dataset.

(train_images, train_labels),(test_images, test_labels) = keras.datasets.fashion_mnist.load_data()train_images = train_images / 255.0
test_images = test_images / 255.0
validation_images = train_images[:5000]
validation_labels = train_labels[:5000]
class_names = ["T-shirt/top", "Trouser", "Pullover", "Dress", "Coat", "Sandal", "Shirt", "Sneaker", "Bag", "Ankle boot"]

Here we define the custom regularizer as explained earlier.

def custom_l2_regularizer(weights):
return tf.reduce_sum(0.02 * tf.square(weights))

Next step is to implement our neural network and its layers. An assignment of the appropriate parameters to each layer takes place here, including our custom regularizer.

model = keras.models.Sequential([
keras.layers.Flatten(input_shape=[28,28]),
keras.layers.Dense(200, activation='relu', kernel_regularizer=custom_l2_regularizer),
keras.layers.Dense(100, activation='relu', kernel_regularizer=custom_l2_regularizer),
keras.layers.Dense(50, activation='relu', kernel_regularizer=custom_l2_regularizer),
keras.layers.Dense(10, activation='softmax')
])

To ensure that our neural network trains properly, we utilize the stochastic gradient descent optimization algorithm and set some arbitrary hyperparameters such as epochs and learning rates.

sgd = keras.optimizers.SGD(lr=0.01)
model.compile(loss="sparse_categorical_crossentropy", optimizer=sgd, metrics=["accuracy"])
model.fit(train_images, train_labels, epochs=60, validation_data=(validation_images, validation_labels))

After training, an evaluation of the trained model is conducted on the test dataset. The evaluation data must contain data that the model has not encountered during training.

model.evaluate(test_images, test_labels)

To verify the accuracy of our model, we can take a subset of our test images and run inference to verify their results.

practical_test_images = test_images[:10]
predictions = model.predict_classes(practical_test_images)
print(predictions)
print(np.array(class_names)[predictions])