You're right. It does have an impact on performance. I haven't done a comparison, but the performance is still acceptable in my experience. It depends on the use case, but the local cache that borg maintains helps a lot in simple cases, because any unchanged files simply get skipped.