Not to be too pedantic here -- but what do you mean by "PRIVATE CRYPTOGRAPHIC CERTIFICATES"? Certificates are intended to be public. Private keys are intended to be private. They are not equivalent although they are certainly related.
Public and private keys are just very large numbers; they are stored in various formats. Certificate may have been the wrong word, but the gist is, you should never copy your private key anywhere.
I disagree about the copying. You should protect your private key wherever you take it, but copying it is certainly a reasonable thing to do under some circumstances. With an appropriate passphrase, storing your private key in a PKCS#12 file is just as secure as storing the private key in software-based OS or application keystore. In fact it may be identical depending on the OS or application. You are subject to the same attacks (password jacking, in memory key copying) in both cases. I keep some of my lower-value private keys on a USB stick for exactly this scenario.
The only way (IMHO) to get around those attacks is to never decrypt the key on a machine with untrusted software running and accessible memory. The only device that comes close to this is a smartcard or TPM type scenario, which uses a separate CPU and protected memory to do the RSA operations.
They work as a pair. If you access from a different machine you need the private key on that machine. You can't just copy the public key over to that machine and expect it to work.
It depends on what you are doing. You can do public key operations without the private key (for example signature verification). You can do private key operations without the public key (for example signing). In practice the certificate is always available near the private key, but the reverse is usually not true -- there are many cases where you will have the certificate but not the private key.
