After looking at the map I am a little confused on which river. Sounds like he took off in a NE direction and the ATC's command to "follow the river" meant follow the Anacostia river NE. Maybe the pilot was confused by the Washington Channel and started to follow that when he turned left?
They say they lined up on runway 18 which is now runway 19 so they were flying almost directly south. That doesn't really clear up how they had any ambiguity about what direction they should go to follow the river though because for a decent ways you're already following the river downstream.
