In fact you SHOULD have a layer of clay between the surface and the ground water you are drinking. You don't want a fast path between surface contamination and your well water.
You want a compromise. You need water to get to the well somehow. It needs to get there fast enough to replace what you use, but not so fast that it isn't filtered first.
The simpler and more likely explanation is that you have a clay or otherwise mostly impermeable substrate.