I think that is actually a huge factor, and one of the key obvious things that the OP overlooked. Latvia (and I suspect the whole Baltic region) was at an artificially inflated population peak when the Soviet Union collapsed. It was one of the most economically prosperous areas of the Soviet Union, and had huge Soviet military bases. Retirement on the Baltic coast was the dream for military brass as well as many other Soviets in the 80s. Additionally, the freedom movements and greater autonomy in the second half of the 80s may have attracted others from other Soviet Republics.
Since independence, and particularly recently, Russians (who once made up over half of Latvia's population) have left, and in many cases not been welcome to stay. One of the major tensions in Latvian politics today is the status of Russian Latvians.
With that backdrop (and given the other factors, such as migration to the rest of the EU) it isn't really that surprising that population levels are returning to that of the 60's.
Since independence, and particularly recently, Russians (who once made up over half of Latvia's population) have left, and in many cases not been welcome to stay. One of the major tensions in Latvian politics today is the status of Russian Latvians.
With that backdrop (and given the other factors, such as migration to the rest of the EU) it isn't really that surprising that population levels are returning to that of the 60's.