Source: Deep Learning on Medium
Colorization of grayscale images is a simple task for the human imagination. A human need only recall that sky is blue and grass is green; for many objects, the mind is free to hallucinate several plausible colors. The high-level comprehension required for this process is precisely why the development of fully automatic colorization algorithms remains a challenge. Colorization is thus intriguing beyond its immediate practical utility in graphics applications. Automatic colorization serves as a proxy measure for visual understanding. Our work makes this connection explicit; we unify a colorization pipeline with the type of deep neural architectures driving advances in image classification and object detection.
CNN Architecture for Colorization
The architecture proposed by Richard Zhang, Phillip Isola and Alexei A. Efros is a VGG-style network with multiple convolutional blocks. Each block has two or three convolutional layers followed by a Rectified Linear Unit (ReLU) and terminating in a Batch Normalizationlayer. Unlike the VGG net, there are no pooling or fully connected layers.
Each conv layer refers to a block of 2 or 3 repeated conv and ReLU layers, followed by a BatchNorm  layer. The net has no pool layers. All changes in resolution are achieved through spatial downsampling or upsampling between conv blocks.
1.) Learning Representations for Automatic Colorization
Gustav Larsson , Michael Maire and Gregory Shakhnarovich
2.) Colorful Image Colorization
Richard Zhang, Phillip Isola and Alexei A. Efros