Source: Deep Learning on Medium
Unveiling Biology with Deep Microscopy
An AI-inspired Revolution in the Life Sciences
The scientific revolution was ushered in at the beginning of the 17th century with the development of two of the most important inventions in history — the telescope and the microscope. With the telescope, Galileo turned his attention skyward, and advances in optics led Robert Hooke and Antonie van Leeuwenhoek toward the first use of the compound microscope as a scientific instrument, circa 1665. Today, we are witnessing an information technology-era revolution in microscopy, supercharged by deep learning algorithms that have propelled artificial intelligence to transform industry after industry.
One of the major breakthroughs in deep learning came in 2012, when the performance superiority of a deep convolutional neural network combined with GPUs for image classification was revealed by Hinton and colleagues  for the ImageNet Large Scale Visual Recognition Challenge (ILSVRC). In AI’s current innovation and implementation phase, deep learning algorithms are propelling nearly all computer vision-intensive applications, including autonomous vehicles (transportation, military), facial recognition (retail, IT, communications, finance), biomedical imaging (healthcare), autonomous weapons and targeting systems (military), and automation and robotics (military, manufacturing, heavy industry, retail).
It should come as no surprise that the field of microscopy would ripe for transformation by artificial intelligence-aided image processing, analysis and interpretation. In biological research, microscopy generates prodigious amounts of image data; a single experiment with a transmission electron microscope can generate a data set containing over 100 terabytes worth of images . The myriad of instruments and image processing techniques available today can resolve structures ranging in size across nearly 10 orders of magnitude, from single molecules to entire organisms, and capture spatial (3D) as well as temporal (4D) dynamics on time scales of femtoseconds to seconds.
Deep learning algorithms are tackling a wide range of computer vision problems applicable to biological images. These include image classification and image segmentation; object detection and tracking; image style transfer, colorization and reconstruction; image super-resolution and image synthesis. Many of these tasks are extremely well-suited to deep learning approaches. Spectacular results have been achieved for super-resolution localization microscopy, where a newly developed deep learning technique called ANNA-PALM can perform super-resolution image reconstruction on vastly fewer images versus standard protocols .
Another powerful example of deep learning in biological imaging comes out of the application of cryoelectron microscopy (cryo-EM), used to produce 1 to 5 Angstrom resolution molecular models for structural biologists. To get a sense of the value of cryo-EM technology, simply picture exquisite 3D models of viruses, enzymes, and macromolecular complexes, as well as supramolecular assemblies. To generate an atomic scale model, tens or even hundreds of thousands of images are required, and image processing workflows are complicated and difficult to fully automate. At one step called particle picking, hundreds of thousands of particles need to be selected (for all orientations to be represented) to identify molecular images, which are then collectively used to build the structure. Artificial neural networks (CNNs) are now showing promise in solving this daunting piece of the workflow .
Research involving deep microscopy can leverage architectures found in commercial applications for image classification and segmentation, two of the main use cases in biological applications. What is lacking at present are annotated training data sets. As a result, transfer learning has played a big role in jumpstarting image classifiers on biological data [5,6]. For example, by replacing the final layer of an image classifier trained on another (non-biological) data set with one suitable for the new classification task, the model can subsequently be retrained on a smaller set of annotated data. The Cell Cognition Explorer (https://software.cellcognition-project.org/ is a Python-based tool for classifying cellular phenotypes with deep learning.
New schemas are emerging for image segmentation to predict and identify distinct objects in images, such as cells, nuclei, filaments, and all types of cell edges and boundaries. A few examples are: DeepCell http://www.deepcell.org, http://github.com/vanvalenlab/deepcell-tf a software library for deep-learning-enabled single-cell analysis  and U-Net http://github.com/lmb-freiburg/Unet-Segmentation a ImageJ plug-in for single-cell image segmentation [8,9]. Both of these software libraries approach segmentation as a pixel-level classification task and produce pixel-level predictions for a set of features. Deep learning-based segmentation is making an impact on cancer research, as was shown by an elegant approach to quantify immune cell behavior in the complex tumor microenvironment of breast cancers .
Nowhere is the impact of deep learning going to be displayed more dramatically than across the domain of neuroscience. Many outstanding problems related to neuronal circuitry, models of synaptic plasticity, live imaging of networks and others are on the verge of being confronted by researchers armed with AI-based methods.
A landmark paper published last week in the journal Science  provides a glimpse into the possibilities of biological insight and discovery resulting from deep microscopy. The research group’s aim was to reconstruct the entire connectivity pattern — axons, dendrites, pre- and postsynaptic processes, and especially synapses — of a small brain tissue volume residing within layer 4 of the mouse cortex using 3D electron microscopy. From a set of 3,400 images totaling 194 gigabytes of data, the analysis pipeline generated a vast trove of information and led to the reconstruction of 89 neurons with a connectivity matrix built of 34,221 presynaptic processes and 11,400 postsynaptic processes. This neuroanatomical tour de force required crowdsourcing efforts from about 100 students and used several AI algorithms to process and generate the results. The pipeline involved SegEM (using CNN and watershed) to generate features from 15 million volume segments; these were fed into ConnectEM, a connectivity classifier and TypeEM, an object classifier (axons, dendrites, spine head, glia). SynEM was used to classify pre- and postsynaptic segments. Quantitative analyses conducted on the output yielded important insights into the formation of neuronal circuits and described how learning (i.e. long-term potentiation and long-term depression) might be imprinted in the neural network.
Deep microscopy is enabling new paradigms in experimentation across a field that once was driven primarily by descriptive studies. In addition to the examples above, imaging-based AI-driven protocols are beginning to impact drug discovery operations. Companies such as Recursion Pharmaceuticals are using deep learning in morphological analysis (known as image-based profiling) to identify biological features and variables associated with drug response and disease-related conditions.
In a similar fashion, the dense brain region reconstruction achieved by the 3D electron microscopy study opens up the possibility that there are connectomic phenotypes that can now be characterized and predicted using image-based assays. For example, changes in neural circuits at the synaptic level could be tracked after environmental or drug exposures, in disease states and as a consequence of genetic changes and lead to new therapies. It is also apparent that further explorations of the resulting connectivity maps could elucidate connectivity rules and serve as a way to test mathematical models of brain information processing operations. We may be close to a virtuous cycle whereby discoveries from deep microscopy lead to new artificial neural network architectures, which can more deeply explore biology.
 A. Krizhevsky, I. Sutskever, and G. Hinton, ImageNet classification with deep convolutional neural networks (2012), In Proc. Advances in Neural Information Processing Systems 25 1090–1098
 E. Moen, D. Bannon, T. Kudo, et al., Deep learning for cellular image analysis (2019), Nature Methods 16:1233–1246 https://doi.org/10.1038/s41592-019-0403-1
 W. Ouyang, A. Aristov, M. Lelek et al., Deep learning massively accelerates super-resolution localization microscopy (2018) Nat Biotechnol 36, 460–468 https://www.nature.com/articles/nbt.4106
 T. Bepler, A. Morin, M. Rapp et al., Positive-unlabeled convolutional neural networks for particle picking in cry electron micrographs. (2019), Nature Methods 16: 1153–1160 https://doi.org/10.1038/s41592-019-0575-8
 W. Zhang, R. Li, T. Zeng et al., Deep model based transfer and multi-task learning for biological image analysis (2016), IEEE Trans. Big Data https://doi.org/10.1109/ TBDATA.2016.2573280
 N. Pawlowski, J. C. Caicedo, S. Singh et al., Automating morphological profiling with generic deep convolutional networks (2016), Preprint available at https://www.biorxiv.org/content/ early/2016/11/02/085118
 DA Van Valen, et al. Deep learning automates the quantitative analysis of individual cells in live cell imaging experiments, (2016), PLoS Comput. Biol. 12, e1005177. doi:10.1371/journal.pcbi.1005177
 O. Ronneberger, P. Fischer, and T. Brox, U-net: convolutional networks for biomedical image segmentation (2015), In Medical Image Computing and Computer-Assisted Intervention — MICCAI 2015 Lecture Notes in Computer Science, vol 9351. Cham, Switzerland: Springer.
 T. Falk, D. Mai, R. Bensch et al., U-Net: deep learning for cell counting, detection, and morphometry (2019), Nature Methods 16: 67–70 https://doi.org/10.1038/s41592-018-0261-2
 L. Keren, M. Bosse, D. Marquez et al., A structured tumor-immune microenvironment in triple negative breast cancer revealed by multiplexed ion beam imaging (2018) Cell 174(6):1373–1387.e19. doi:10.1016/j.cell.2018.08.039
 A. Motta, M. Berning, K. M. Boergens, et al., Dense connectomic reconstruction in layer 4 of the somatosensory cortex (2019), Science 366, eaay3134