Calorie count _is_ the key. This is incredibly well established in studies. Weight loss does not occur without caloric deficit, no matter what you eat. Likewise, weight loss will occur in caloric deficit even when eating what people would consider "junk", including sugar. See The Twinkie Diet, for example: http://www.cnn.com/2010/HEALTH/11/08/twinkie.diet.professor/...
Avoiding food because "we didn't evolve to eat" is not sound science, either.
Avoiding food because "we didn't evolve to eat" is not sound science, either.