From my understanding, this only happens when the GC is run. Currently (Go1.3) the GC runs every 5 minutes or unless you manually invoke it. Currently the GC will stop the world and do the entire cleanup at once. From my understanding of this, the GC will do the cleanup in small 10ms bursts, so you won't have a 200ms pause every 5 minutes.
"Go, the language with zero nines of availability."