A quick note on how to do this, partly so I don’t forget.

I used to think (and Google says) that the answer is to compare the output of the following two commands:

cat /path/to/public/key | openssl x509 -noout -modulus
cat /path/to/private/key | openssl rsa -noout -modulus

Links may suggest md5sum on the output to make the moduli easier to compare.

Well, this is all very well as far as it goes, but it won’t work with ECDSA certificates. Here’s a much more reliable way that works with all certificates, assuming you’re using a modern OpenSSL:

cat /path/to/public/key | openssl x509 -noout -pubkey
cat /path/to/private/key | openssl pkey -pubout

The openssl pkey command actually extracts the public key from the private key. You can then compare them (or their MD5 if you pass them through md5sum).