A convolutional neural network ought to have translational symmetry, which should lead to a generalized version of momentum. If I understood the article correctly the conserved quantity would be <gx, dx>, where dx is the finite difference gradient of x.
This gives a vector with dimensions equal to however many directions you can translate a layer in and which is conserved over all (convolutional) layers.
Exactly right! In fact, because that symmetry does not include an action on the parameters of the layer, your conserved quantity <gx, dx> should hold whether or not the network is stationary for a loss. This means that it'll be stationary on every single data point. (In an image classification model, these values are just telling you whether or not the loss would be improved if the input image were translated.)
This gives a vector with dimensions equal to however many directions you can translate a layer in and which is conserved over all (convolutional) layers.