I've had phones stolen enough that I'd say "yes, too accessible" - your mileage may vary.
Timestamped travel information generally equates to "hey, this guy returns home between 7 and 8:30 most nights, works in the office five days a week, and here's where his home is". It's no great stretch to equate that to "he will not be home during this time frame" and suddenly a lost phone turns into a much larger burglary.
There's also plenty of larger privacy concerns, but even just the out-and-out-crook scenarios should be concerning enough.
The iPhone is not storing travel history information. It's only storing a "last seen" timestamp for each cell tower. And again, the most accurate they'd be able to get was "this guy was within 500 meters of these dozens of cell towers at 5PM, and with 500 meters of these dozens of other cell towers at 7PM." They'd in no way be able to get your physical address, unless you lived alone in the countryside.
But the data is not that important, relatively. The assumption is that some "Bad Guy" has gotten a hold of your phone. Here's a list of other vectors that this person could use to identify you, your residence, etc.
1) Your cell number is going to be readily available from the device itself. Using this, he'll be able to look up your address information in the Address Book.
2) Unless you notice the missing device right away and change your passwords, this guy also has access to your email accounts. Ever order anything online? Your address is most likely in your emails.
3) Ever use Google Maps? They can look at your recent searches. "3 different searches originating at 123 Main St? Might be his home address"
4) Phone app. Recent calls/favorites. With a little social engineering, your address is known. "Hi, I found this phone on the ground, can you tell me where they live so I can drop it off?"
Timestamped travel information generally equates to "hey, this guy returns home between 7 and 8:30 most nights, works in the office five days a week, and here's where his home is". It's no great stretch to equate that to "he will not be home during this time frame" and suddenly a lost phone turns into a much larger burglary.
There's also plenty of larger privacy concerns, but even just the out-and-out-crook scenarios should be concerning enough.