This is surprising and counter to what most others are experiencing.
Many of the performance optimizations have tradeoffs and while they increase performance for the majority of cases, some cases will experience slight degradation.
This is also a surprisingly long time for initialization. My site with a few hundred pages on a rather old mac (with an SSD) has
Go initialization:
48.00164ms (48.842363ms) 2.14 MB 18299 Allocs
Do you have your site source available publicly so we can do more testing and benchmarking with it? We'd love to understand the places where it's not improving so we can address them further.
Started building sites ... Go initialization: 5.722103073s (5.722526117s) 2.66 MB 22083 Allocs
The rest of the steps take 115ms. What is happening in go init, and why does it take so long?