He said you can have privacy without secrets. But suppose there exists a room with a door where you don't know what someone is doing behind it. The activity behind the door is the secret. The door is still representing privacy (not secrecy). That was the point of the analogy.
You're adding an element of your own subjective knowledge/awareness. You could not know a secret or something kept private. The only thing that is the same is your not knowing. What you don't know could still be a secret, something private, or even public knowledge for that matter. The point is, you not knowing doesn't make anything anything.
A secret is an attribution to the contents of a description which are withheld for security reasons. Or at least, that's what it should be. And that's what it appears to be with the military.
Privacy is a right to withhold information based on a fair desire to not have that information be known.
Invasion of privacy is someone pulling the curtains when you're showering or opening the door when you're in the bathroom.
That should not be confused with invasion of secrecy.
Either way, the main point is that these two terms are confusing, and even the law confuses them. And yes, people would consider something they don't know a secret also, and your use of the word is intuitively correct. But a clearer distinction could and should exist, and we should all work towards that clearer distinction. I believe the words are already more than adequate, since we already see the correct (most practical and fitting) distinctions used where they are most needed in practice (military).
In the real world, though, 99% of the time, we DO know. And even if we don't know exactly, we can narrow it very closely. We either shit, take a bath, masturebate or whatever. We still want privacy of any of those things.