Experience teaches you the noise to ignore. Same thing applies to all sorts of disciplines, even debugging or fixing a code.
An experienced dev can look at a large code base and spot a bug or just from the structure of the code tell if it's "badly written" code.
An experienced sysops can look at large streams of system log files and tell you what's wrong with the system because they can ignore the noise. Someone else might be fixated on the high cpu load when the issue is a bad device.
Experience is knowing what qualifies as noise, and when.
Just because a system is running doesn't mean it's running well. Just because code is orderly in appearance doesn't mean it's effective at performing the desired task.
Perfection is the enemy of good.
Sometimes a shambling mess of spaghetti is that way because of a lot of underlying and tangentially-related issues with the operating environment. Code can be perfect looking yet fail because developer expectation of the compiler/toolchain and the underlying OS don't actually function as documented. Of course this can also lead to things being brittle when the underlying issues change. Experience is knowing when those issues are involved.
An experienced dev can look at a large code base and spot a bug or just from the structure of the code tell if it's "badly written" code.
An experienced sysops can look at large streams of system log files and tell you what's wrong with the system because they can ignore the noise. Someone else might be fixated on the high cpu load when the issue is a bad device.