On: A stack of feed-forward layers does surprisingly w...

[BobbyJo]

Wouldn't attention networks use less computation for the same number of weights? Feed forward networks have higher connectivity no?

If a network is the same size, less computationally demanding, and gives you 3% improvement, it seems extremely worthwhile, especially given the (mostly) diminishing returns of just adding more weights/layers.

[jeffreyrogers → BobbyJo]

It is less general though, so this is interesting because it shows you can learn a lot without adding very much structure.