Orthogonalized SGD and Nested Architectures for Anytime Neural Networks (2020)

This paper describes a process for creating and training anytime neural networks.

These networks are dynamically configurable during inference in that they support a number of exit points—in contrast, most neural networks must be run from input to a single exit point.

Earlier exits represent require fewer computational resources and produce less accurate answers, while later exits improve accuracy at the cost of more computation.

This paper presents techniques for designing such exit points and then a novel technique (the orthogonalized stochastic gradient descent) for training such a network.

This training technique requires users to pick a preferred exit point. Somewhat surprisingly, we find that even with the added flexibility of anytime networks, the accuracy for the preferred exit is even higher than that which can be achieved using traditional networks with a single exit points and similar computational resources.

While this paper did not show how to use the dynamic adaptation capabilities, it did show how to build them and train a network to produce accurate results.

Read abstract

We propose a novel variant of SGD customized for training network architectures that support anytime behavior: such networks produce a series of increasingly accurate outputs over time. Efficient architectural designs for these networks focus on re-using internal state; subnetworks must produce representations relevant for both immediate prediction as well as refinement by subsequent network stages. We consider traditional branched networks as well as a new class of recursively nested networks. Our new optimizer, Orthogonalized SGD, dynamically re-balances task-specific gradients when training a multitask network. In the context of anytime architectures, this optimizer projects gradients from later outputs onto a parameter subspace that does not interfere with those from earlier outputs. Experiments demonstrate that training with Orthogonalized SGD significantly improves generalization accuracy of anytime networks.