Home Artificial Intelligence Early Bird: Greener Way of Training Deep Neural Networks

Early Bird: Greener Way of Training Deep Neural Networks


Deep Neural Networks (DNNs), or Deep Learning, is a sub-field of Artificial Intelligence that functions for specific tasks that involve unstructured data types that are easy for humans. It has billions of artificial neurons that can make it perform and decide like humans or even better.

We may not realize it, but Deep Neural Networks work in our everyday high-tech applications. They are present in infamous virtual assistants like Google and Siri, self-driving cars, product recommendations on Netflix and Amazon, cat and dog filters in Snapchat, and even tag suggestions on Facebook’s photos.

Deep Neural Networks’ Pruning System and the Need for a Less-Costly Method

Deep Neural Networks’ training requires a lot of energy, and it can be costly. One of the ways for training Deep Neural Networks involves a step-by-step process of progressive pruning:

  1. training a vast, dense network strengthens the network of artificial neurons, and it reveals which neurons are sufficient for pruning in the next step.
  2. removing neurons that are not relevant reduces the model size and computed cost that can benefit even small devices with limited capacity. It also selects only the neurons that can help do the task.
  3. restoring performance by retraining the pruned network
  4. doing the process many times to achieve good performance.

Lin mentioned that the first step is the costliest, so they need to search for a fully functional pruned framework to skip the step.

Additionally, researchers conducting a study at UMA found out that the carbon trace of training an elite Deep Neural Networks equates to five automobiles’ lifetime emissions of Carbon Dioxide in the U.S.

The innovation this A.I. would require researchers to find ‘greener ways’ to deflate financial and environmental concerns.

Introducing the Early Bird Ticket to a Greener Network

Lin and his team developed a network termed as the Early Bird Ticket to address the costs of the first step in the pruning process by searching for crucial network patterns early in training. It trims dense Deep Neural Networks training networks into thoroughly pruned ones, and the researchers can also automatically spot these tickets in the first 10 percent or less in the Deep Neural Networks training. Also, Early Bird detects greenhouse gas emissions.

On April 29, 2020, researchers revealed Early Bird in a famous paper at the International Conference on Learning Representations (ICLR 2020). The research comprised of the following study:

Rice’s Efficient and Intelligent Computing Lab Study

  • It is authored by leading Haoran You and Chaojian Li.
  • It is led by Yingyan Lin (EIC Director), Richard Baraniuk (Rice), and Zhangyang Wang (Texas A&M).
  • Results show Early Bird could use 10.7 times less energy in Deep Neural Networks Training towards the same or better accuracy level than the former training.

Benefits of Early Bird Training and Expectations for the Future

Early Bird training poses many benefits aside from its main focus and motivation to address environmental concerns. Apart from making it more environmentally friendly, their goal also includes making Artificial Intelligence more inclusive. Green A.I. can open opportunities for researchers with limited computer resources to explore these innovations. It is beneficial in saving both costs and energy.

How does Deep Neural Networks inspire your thesis?



Please enter your comment!
Please enter your name here

Exit mobile version