Carbon Footprint of Neural Network Training — Is it that bad?

Source: Deep Learning on Medium

Carbon Footprint of Neural Network Training — Is it that bad?

In a recent work, researchers at the University of Massachusetts, Amherst, performed a carbon footprint analysis of the training of several common large AI models. The findings are quite interesting and somewhat eyeopening — The carbon dioxide (CO2) emission for the training of a neural network model can be 10 times more than the CO2 produced by a human in a year. When I read this article, I was baffled, yet intrigued to find out how true this claim is. Sure! training a single neural network model can produce 10x CO2 compared to a human, but not every human is doing deep learning. What if the number of training instances are much smaller than the number of humans on earth? In that case, the carbon footprint of CO2 from deep learning will be negligible compared to that of humans. Moreover, how does the CO2 production by a single human compare to the overall CO2 emitted in the environment?

In this article, I want to dig deeper to find out how “bad” is the neural network training. From 49th HPC User Forum, 2013, Tuscon presentation by Nvidia, it can be seen that the total number of compute GPUs that are out there are 430M in 2013. The average power consumption of a GPU is around 100–200 W, so I will take 150W as a rough estimate of power for all the GPUs. Now, if we run all 430M GPUs 24×7 for 365 days at 150W, and use the power usage effectiveness coefficient (PUE) of 1.58, the total energy consumption will be roughly 8.9 x 10¹¹ KWhr.

Using the U.S. Environmental Protection Agency (EPA) provided data from U.S. (EPA, 2018), the energy consumption can be converted to estimated CO2 emissions using:

CO2 emission (in lbs) = 0.954 x energy in KWhr

which in our case will be 8.5 x 10¹¹ lbs or 0.4 gigatonnes. Thus, if we run all the GPUs that exist today continuously 24×7 for 1 year, it will produce 0.4 gigatonnes of CO2 while the total carbon dioxide production in 1 year all over the world is 36.2 gigatonnes (source). Thus, GPUs are already producing 1 % of the total carbon emission. Note that in the analysis, I assumed the total number of GPUs as 430M, which is the data from 2013. The number of GPUs out there today, in 2019, is a lot more and the number is supposed to grow in upcoming years.