This is not what was actually implemented, and would have finished the Times, due to SEO effects alone. (There are numerous other issues, but that's the one that will crater their revenue tomorrow.)
Seriously, while I appreciate the desire of programmers to think that we're smarter than the average bear, this is like saying StackOverflow is a clone-it-in-a-weekend site. It betrays a fundamental lack of understanding as to what is actually going on with the business model.
You obviously didn't get the sarcasm. Having said that, $40m is a lot of money, especially considering it shouldn't have required any extra hardware or even a complete platform change (with expensive licensing).
Even at $200k/year that's 100 developers for 2 years to implement what is essentially a subscription-based service. Even if you take into account the requirement to offer some level of free access for search engines and the like it still doesn't account for why this project cost this much money.
It's a little more complicated than that. From the Bloomberg article:
"Developing the technology for paywalls is challenging because publishers are trying to strike the most profitable balance between charging some online readers and letting others in free to generate advertising and attention. Though Times Co. has said it will charge visitors after they’ve read a certain number of stories, for example, the company plans to let people coming to the website from social networks such as Facebook Inc. view an unlimited number of those stories for free."
I suspect that almost all of that budget went into market research. This isn't quite an A/B testing scenario. If you drive away potentially paying customers 50% of the time (in a worst case A/B test) then you are losing money. The potential for lost revenue in a "release early, release often" iteration approach may justify such a big budget. With so many factors, quite a bit of strategizing and research need to happen before putting things to the test.
Indeed, a bunch of New York executive wages over an 18 month period start to add up. Khoi Vin (former art director) pondered this on his blog a few weeks ago [1], citing it as one of the reasons he resigned his post.
Result: