OK, I'll give an example. My team leader in a previous job used to regularly turn up at 11am (officially we were supposed to be in by 9.30, but it was academia and no one complained too much). I was usually in before 9.15 - like the majority of the staff.
So sometimes I would need to ask him something, and he wasn't around - that would hold me up.
Sometimes other people would need to ask him something. He wasn't around, so they asked me instead - usually when I was trying to get in the zone.
I am all for giving staff flexibility but it should not come at an expense to productivity.
Remote, timezone-distributed teams deal with this all the time. We always ensure everyone has a bunch of work they can do, and we use email/chat to handle the asynchronous communication issue. What you described isn't a scheduling issue, it's a communication issue. Not on your part, either, BTW.
So sometimes I would need to ask him something, and he wasn't around - that would hold me up. Sometimes other people would need to ask him something. He wasn't around, so they asked me instead - usually when I was trying to get in the zone.
I am all for giving staff flexibility but it should not come at an expense to productivity.