vSphere 6.0 and 6.5 SSL certificate Replacement / Implementation using the Certificate-Manager automation tool
vSphere 6.x Architecture
vSphere Certificate replacement and implementation is much easier than Center Server 5.1 or 5.5. In the past, you would have to replace each out of the endpoint certificates, for example vCenter Server, Single Sign On, Inventory Service, Web Client, and so forth.
To simplify the process, VMware now uses a Reverse HTTP Proxy which will route traffic accordingly, meaning we only need to replace one certificate, instead of replacing all them in the previous version. There are 4 Solution Users in vSphere 6.x – vpxd, vpxd-extention, vsphere-webclient, and machine and you can replace each solution user certificate if you would like, however it’s no longer necessary thanks to the reverse proxy.
As you may know, there have been architectural changes to vSphere 6.x as well. There are now two different deployment options, an Embedded Platform Services Controller (PSC), or an external PSC). With embedded nodes, you will have one Reverse HTTP proxy endpoint to replace, and with an external PSC you will have two endpoint certificates to replace.
vSphere 6.x also ships with its own internal certificate Authority called the VMCA – VMware Certificate Authority. The VMCA will issue or validate certificates and has two different implementation methods. You can either use it as your Root CA, which is the default configuration, or it can be used as a Subordinate CA which will be signed by an Enterprise CA. To manage the VMCA, you will use the certool.exe located in the following directories:
- <Install Directory>\Program Files\VMware\vCenter Server\vmcad\
- /usr/lib/vmware-vmca/bin/certool
Trusts are handled by the VMware Endpoint Certificate Store (VECS). This is great news because we no longer have to update the trusts between the endpoint when we replace the certificate, the VECS will do it all for us! VECS holds stores that contain certificates and their keys.
By default there are three stores, as shown above, each store has an entry for a Certificate + Key.
To manage the VECS we will use VECS-CLI which is located in the following directories. To learn more about vecs-cli, please click here: vecs-cli usage.
- <Install Directory>\VMware\vCenter Server\vmafdd\
- /usr/lib/vmware-vmafd/bin/vecs-cli
With these changes you have three different types of certificates which can be replaced.
- Machine SSL / Reverse Proxy Certificate
- CA Certificates (Trusted Root Certificates)
- Solution User Certificates
Now we get to the good stuff. The certificate replacement.
Certificate Replacement with ‘Certificate-Manager’ the new SSL Automation tool
The new certificate automation tool is called “Certificate-Manager.bat” and is installed with vCenter by default. It’s located in \Program Files\VMware\vCenter Server\vmcad
If you are using this tool, you do not have to interact with vecs-cli.exe or the certool.exe; when you run the Certificate-Manager tool you will see the options pictured below. The most widely used will be the first two options, which I will go through step by step.
- Replace Machine (Reverse HTTP Proxy) Certificate with Custom Certificate
- Replace VMCA Root certificate with custom signing certificate and replace all Certificates.
Replace Machine (Reverse HTTP Proxy) Certificate with Custom Certificate
Step 1. Edit the certool.cfg file – template file for CSR
The file is located here: <InstallDirectory>\Program Files\VMware\vCenter Server\vmcad\certool.cfg
I would leave the IP address blank since VMware will be dropping support for this soon. IPs are supposed to change, so you really don’t want this in your certificate.
Step 2. Run the Certificate-Manager.bat tool
- <Install Directory>\Program Files\VMware\vCenter Server\vmcad\certificate-manager.bat
- Select Option 1 to “Replace Machine SSL certificate with Custom Certificates”
- Enter your SSO password
- Select Option 1 to “Generate Certificate Signing Request(s) and key(s) for Machine SSL certificate”
- Choose the path to write your CSR and Key
Step 3. Sign your CSR
- Your new CSR is in the folder you specified titled “machine_ssl.csr” with it’s corresponding key file.
- You then want to go get your CSR signed by your CA. In my case, I am signing it with an internal Microsoft Certificate Authority. I will not provide these steps as they are the same for any previous version, but I will provide a KB article that outlines this process. Please click here and go to the section titled “Obtaining the certificate” steps 1-10.
- You will also need to download your root certificate or certificate chain which is outlined in the same KB above section “Obtaining the certificate” steps 14-20.
Step 4. Import Custom Certificate in place of Machine SSL certificate
- Provide the path to the new certificate
- Provide the path to the key
- Provide the path to your Root certificate
That’s it! All end points will communicate through the Reverse HTTP proxy which uses this certificate. This is much more simple than before right!?
Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates (Using VMCA as a subordinate CA)
Step 1. Edit the certool.cfg file – template file for CSR
The file is located here: <InstallDirectory>\Program Files\VMware\vCenter Server\vmcad\certool.cfg
I would leave the IP address blank since VMware will be dropping support for this soon. IPs are supposed to change, so you really don’t want this in your certificate.
Step 2. Run the Certificate-Manager.bat tool
- <Install Directory>\Program Files\VMware\vCenter Server\vmcad\certificate-manager.bat
- Select Option 2 to “Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates”
- Enter your SSO Password
- Select Option 1 to “Generate Certificate Signing Request(s) and Key(s) for VMCA Root Signing certificate”
- Choose the path to write your CSR and Key
Step 3. Sign your CSR
Note: Please see Creating Microsoft CA Templates to create a template for the signing certificate.
- Your new CSR is in the folder you specified titled “root_signing_cert.csr” with it’s corresponding key file.
- When you sign this certificate, make sure you select the template “Subordinate Certificate Authority” or an altered copy of this.
- Select Base 64 encoded and “Download certificate chain”
- Open the new .p7b certificate and export both certs as base 64.
- Click Next, Select Base-64 encoded, give the certificate a name, click Finish.
- Create a chain file called chain.cer by running the following command to concatenate the new leaf (vmca) certificate, and the root certificate.
copy vmca.cer+root64.cer chain.cer
Step 4. Import Custom certificate(s) and key(s) for VMCA Root Signing certificate
- Go back to the certificate-manager tool and select option 2 to “Import Custom certificate(s) and key(s) for VMCA Root Signing certificate”
- Provide the chain.cer
- Provide the root_signing_cert.key
- Select ‘Y‘
Note: If you are using an embedded PSC then you are done, otherwise, please proceed to the steps below.
Step 5. (Only necessary if External PSC) Recycle your vCenter Server services by running the following commands:
c:\Program Files\VMware\vCenter Server\bin> service-control --stop --all c:\Program Files\VMware\vCenter Server\bin> service-control --start --all
Step 6. Go to your vCenter Server and run the certificate manager tool (C:\Program Files\VMware\vCenter Server\vmcad) Select Option 3 – Replace Machine SSL certificate with VMCA Certificate
Step 7. (Only necessary if External PSC) Go to your vCenter Server and run the certificate manager tool (C:\Program Files\VMware\vCenter Server\vmcad) Select Option 6 – Replace solution user certificates with VMCA Certificate
Note: You may need to add your VMCA signing certificate to Trusted Publishers as shown below; I didn’t have to run through this step, but a few customer’s have had to do this.
Once this is done your VMCA will act like a subordinate CA and provide CA signed certificates for your services. During the replacement, it will also regenerate all other certificates. There were a few more steps to this option, but it is still much easier than in vSphere 5.1 or vSphere 5.5. If you want to replace your ESXi certificates using VMCA as a subordinate CA, please check out my other post!
Please feel free to leave questions, comments, or suggestions. Good Luck! 🙂
109 Comments
Hi Sean, this is an excellent article and provides better details than VMware’s own documentation.
However do you have to replace the machine “Reverse Proxy” certificate if you are going to replace the VMCA Root certificate with one signed by a MS Enterprise CA? Can you just replace the VMCA certificate and then it generates a new one for the “Reverse Proxy” and all other vCenter components plus the ESXi hosts?
Hi Anthony,
Thank you!
Good question. When you replace the VMCA certificate with a MS Enterprise CA (signing certificate) it tells you that it will regenerate and push out CA certificates for all the services. So yes, you can just replace the VMCA certificate.
Sean
Thanks Sean, appreciate the reply. This will help me greatly when I setup my evaluation Vsphere 6 environment for testing before moving to live.
How about replace Wildcard Certificate of Global Sign?
Hi Jax,
Last time I was in support, we didn’t support wildcart certs. This may have changed but I would contact GSS to confirm.
Sean
“Create a chain file called chain.cer byrunningthefollowingcommandtoconcatenate the new leaf (vmca) certificate, and the root certificate.
copy vmca.cer+root64.cer chain.cer”
I can’t understand this mean.
Can you explain for me?
Thank you.
Hi Sean,
Sure, I had to fix the spacing as it was all messed up for some reason.
To clarify, you need to create a chain file. The file I created is called chain.cer and it is a combination of both the signing certificate from your CA (I called mine vmca.cer) and the root certificate or enterprise CA certificate.
The command you can run to combine the files in windows is below:
copy vmca.cer+root64.cer chain.cer
Let me know if that helps clarify.
Sean
Thank you very much for this helpful guide! Creating a subordinate CA for vSphere components works fine.
There is one small mistake in Step 4. It is option 2 not 1 😉
But Without your post I did not finished the configuration so fast. So Thanks!
Philipp
I have fixed that, thank you for pointing it out 🙂
thanks for this article, there is scarce information out there for this. One question:
I completed the steps, and after my vcenter starts again if I try the web client, it is still untrusted. Viewing the certificate and going to the chain I see
The issuer of this certificate could not be found.
But I can see that the issuer is the same as I setup in the certool.cfg so I kno it was replaced.
Do you have to push the vmcad subordinate certificate to all computers, or shouldn’t it validate?
Hi Brian,
Is your PSC embedded or external? If it’s external you still have to run the certificate-manager tool on the vCenter Server and run through Options 3 and 6.
Sean
embedded. i run everything on the same vcenter server including the psc, vcenter service and web client. Just to verify, when I view the web client certificate, i should see the complete chain all the way up to my microsoft enterprise CA right?
Hrm Odd. If I go to the PSC webpage at https://my.vcenter.local/websso , the certificate seems fine. It does mention that it has no audit trail but i can see the whole chain of
[enterprise ca]
|____ PSC SubCA
|______ Cert
But if i go to https://my.vcenter.local:9443 to get to the vsphere web client i get an untrusted cert that has no chain
Hi Brian,
Yes, you may need to publish your VMCA signing certificate. It should end up under “Trusted Publishers” in the certificates snap-in.
Sean
Hi,
I have the same issue, seems like a bug?
I have the root CA installed in trusted store, shouldn’t need to install intermediate certificate if server sends the entire chain.
Only for that particular server (on port 9443) does it not send complete chain (it sends root and leaf, but not the intermediate VMCA cert that I generated).
Other servers (like port 443) don’t seem to have this issue.
Can confirm, I have the same problem. The web client itself (on port 443) shows the full chain up to my root CA, so it is trusted. The service running on port 9443 (opening a console for example) does not show the full chain and obviously isn’t trusted.
According to VMwares KB, there is no resolution. https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2129180
I am having the exact same issue on 6.0 Update 3 with external PSC… connecting to the PSC with my browser works fine, but connecting to the vCenter server throws the same error that the certificate could not be verified.
If I open the certificate properties in Windows, I get a certificate status of “The issuer of the certificate could not be found”. All of the CA certificates in the chain are in my trusted store, and the certificate presented PSC web app works fine.
Hi Brian,
Did you ever figure this out? I’m having the exact same issue.
When I do this in a lab on a fresh vCenter 6.02 U2a install, it works fine, but in production I get the same issue as you.
I’m also wondering if it could be caused by the fact that my production vCenter has been upgraded through versions 4, 5.1, 5.5 to 6.0.
Thanks for the great guide I really appreciated this.
Thanks! I’m happy to help the community!
Great guide, great job. i’m using the linux appliance and almost exactly the same.
I’m stuck though on importing the new certificates, seems to fail on importing the root certificate. I wonder if my windows encryption is too high (which follow current best practice). The offline root ca is RSA 4096 and sha256. The online issuing is not as high.
Followed the advice on the templates and copying the offline root and online issuing into a combined file, but no joy. Everything was exported to base64. Any ideas?
——————————————————–
You are going to replace Machine SSL cert using custom cert
Continue operation : Option[Y/N] ? : y
Status : 0% Completed [Publishing Root cert…]
Status : 0% Completed [Operation failed, performing automatic rollback]
Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.
——————————————————–
Log file shows nothing useful that i can see, except it dies on the first command after creating the backup store.
Played around a bit more, the import worked this time. You’re a legend! Thanks
Thanks, Dingo! Glad to hear you got it to work, what did you have to do to resolve it?
Hi Sean, I tried a few things to get vCenter appliance to accept the certificates. Similar error posted https://communities.vmware.com/thread/504942 seems to indicate a possible problem with chains. I did his patch whether had any effect I don’t know.
Finally, this a big one. vCenter accepts certificates into the store that use dhe-rsa (PKCS#1 v2.1) instead of plain rsa. It gives no error but services fail to start. I think they should check the certificate on entry into the system for compatible algorithms.
Unfortunately dhe-rsa is the default in Windows PKI, so i had to downgrade my security slightly by making AlternateSignatureAlgorithm=0 in capolicy.inf. Then reissue root and subordinate CAs and eventually – voila!!! I have green padlock on webclient.
many many thanks for this excellent guide.
Good to know, thank you!
This answer really helped me! THANKS
“AlternateSignatureAlgorithm=0 in capolicy.inf.”
I think by default the capolicy.inf file does not natively exist because in a lab on a fresh install of AD CA Enterprise I didn’t find it, but on the environment where I was having the issues with this failure getting certificate-manager to install the machine cert (fought this for 2 weeks), I did find the file there.
Funny how one simple switch can stop everything. Want to say thanks again as I would’ve never found that on my own. VMware script gives no indication of the cause for failure.
We have a two tiered Enterprise PKI in our AD environment with our root CA offline and essentially locked away. Our CAPolicy for AlternateSignatureAlgorithm is AlternateSignatureAlgorithm=1 so we’re running into the situation described above. We’re a little gun-shy about firing up the root CA and changing the policy. Has anyone found an alternative to this? What are the risks to my existing production PKI?
Also, when we created the new template for vSphere 6 in our existing CA, I see there under the “Cryptography” tab there are multiple choices for “Providers”, some are checked and others not. Has anyone tried these as a solution?
Thanks in advance
What did you do to get it working I cannot get it working at all always the same message… I try to downgrade, upgrade same issue…. does it need to be sha256 / 4096 ?
psc is 6.5 version.
What did you do to get it working I cannot get it working at all always the same message… I try to downgrade, upgrade same issue…. does it need to be sha256 / 4096 ?
vcsa is 6.5 windows ca is 2012
[…] Beg's Useful Thoughts – vSphere 6: VMware Certificate Authority (VMCA): Design Decisions 4. vSphere 6 SSL certificate Replacement / Implementation using the Certificate-Manager automation tool There are a few VMware videos about this too, but a couple of the above links do a better job than […]
I too had an issue like Dingo. I am using DigiCert for certificate signing, and they were able to help me out with the issue. After signing when you go to download the certificate…the type of certificate you are downloading is important. Once i chose “Apache” certificate it properly concatenated the intermediate and the trusted root certificates into one. Then it worked like a charm.
Thanks for the information, it should help anyone else that hits that issue!
Second time in two days your blog has help me out!! Great stuff.
I do have question regarding Step 1 of the section “Replace VMCA Root certificate with Custom Signing…..”
How is this step performed if vCenter is an appliance?
Hi Al,
You can still use the certificate-manager tool and follow the same steps. The program is located in /usr/lib/vmware-vmca/bin/certool
Glad to have helped out!
Thanks Sean.
In our environment we have both a MS Root CA and MS Subordinate CA. In following the section “Replace VMCA Root…”
What would you do in case like this?
I would then have a chain (P7B) that contains the Root > Sub CA > VMCA Sub CA. Would I then need to create the chain.cer with all three of the certs? Would this be a normal practice? What would you suggest?
Hi Al,
Yes, the chain.cer will include all three but in the reverse order that you listed. It’s common to have an intermediate certificate like you have in your environment. You should be able to run through the same steps, but create the chain.cer with the command below.
copy vmca.cer+subca.cer+root64.cer chain.cer
Let me know if you have any other questions!
Sean
Thanks again Sean. I have applied the chain.cer and machine ssl successfully.
AI
Fantastic guide! One question – I have a certificate from Go Daddy I want to install. How do I get the certificate onto the vCenter Server Appliance (VCSA)? I have a nice .pfx file with the private key embedded, or I can certainly convert it to any format needed. But I can’t seem to figure out how to actually get the file onto the appliance!
WinSCP used to work with the 5.5 appliance, but VCSA 6.0 hates it. I think it has to do with the new frontend VMWare set up, that makes you type “shell.set –enabled true” before you can type shell, and get access to bash. Seems kind of silly to me, having the directions on how to get into the shell right there, but whatevs.
ftp is not in the bash on the VCSA. sftpd is also not listening. SCP doesn’t work either. The VCSA doesn’t have links to the datastores that I can find (no /vmfs/ directory).
Thanks again!
Hi Jay,
I actually wrote a post on this as well because I hit similar issues when I first ran through the HA configuration for PSCs.
The login bash for root was changed to /bin/appliancesh. To resolve this, you will need to change the login shell for root to /bin/bash using the change shell command below.
chsh -s “/bin/bash” root
You can check the article out below!
Unable to SCP into PSC or VCSA 6
Thanks,
Sean
Yusssss! You da MAN!
Awesome, glad to help!
Hi Sean,
I am planning to deploy vRealize Automation 6.2.1 using the Custom certificates generated through vmware certificate authority as subordinate authority (VMCA), Could you please confirm if that model is feasible and how?
Thanks
I don’t believe the VMCA has that functionality yet. The only supported use cases that I know of are ESXi, Solution User Certificates, and Machine SSL certificate.
Sean
the Machine SSL updates went very easy
but the VMCA replacement of ssl certs is not going at all.
i keep getting error 70037 VMCAAddRootCertificatePrivate () failed
error message cert/key pair does not match
I have remade them 10 times they are the only one in the driectroy as I clear all others out each time and still get this error
was getting the Certificate Chain is not completebut . Add the certificate to the VMware Endpoint Certificate Store
any suggestions
Like your writeup got me a lot further then trying to read vmware garbage.
Hi Bruce,
You can confirm that the key/cert pair matches by running the following openssl commands.
openssl x509 -noout -modulus -in your.crt | openssl md5
openssl rsa -noout -modulus -in your.key | openssl md5
When you run that does the modulus match? If so, when you are running option 2 “Import Custom certificate(s) and key(s) for VMCA Root Signing certificate” you are providing the chain.cer, not the actual certificate right? Are you using intermediates?
Thanks,
Sean
Hi Sean,
Do you know that when the VMCA gives out a cert (eg. vCenter)
that is can assign a short name as well as the FQDN
I have everything working but when i go to my vCenter Shortname i will get a cert mismatch as it only contains the FQDN
I have been unable to find any documentation on this thus far
Appreciate the work and effort you put in..
thanks
James
Hi James,
I haven’t attempted it yet, but you may be able to edit the certool.cfg file to reflect the short name as well:
In previous version the template we used was below: I am not sure if certool.exe will like the changes, but you can attempt and let me know, the subject alt name contains the shortname in the example below.
[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS: vc55-1, IP:10.0.0.10, DNS:vc55-1.vmware.com
[ req_distinguished_name ]
countryName = US
stateOrProvinceName = NY
localityName = New York
0.organizationName = VMWare
organizationalUnitName = vCenterUniqueServer
commonName = vc55-1.vmware.com
Thanks Sean,
I am going to give this a try in the lab and get back to you.
i dont think the cert tool will allow it to be honest… not from the hours i have tried with the .cfg file
Hi James,
I haven’t attempted this in my lab, so please let me know, I will be curious to see if this works.
Thanks,
Sean
Hi James,
Hi Jean,
in my case the variables IPAddress and Hostname of the certool.cfg were interpreted as SubjectAltName. Our CA generated a certficate file with: alt_names = DNS: , DNS: , DNS: . However, i had to run the certool directly with:
/usr/lib/vmware-vmca/bin/certool –genkey –privkey /root/machine_ssl.key –pubkey /root/machine_ssl.pub
/usr/lib/vmware-vmca/bin/certool –gencsr –privkey /root/machine_ssl.key –pubkey /root/machine_ssl.pub –csrfile /root/machine_ssl.csr –config /root/certool.cfg
In my case the certificate-manager command did not take the default certool.cfg into account.
Green padlock!!! Thanks again!!!
So, once I replace the Machine (Reverse HTTP Proxy) certificate (Option 1 in the manager), is there a reason to replace the rest (Option 2)?
From the log the manager generated, I gather that a different certificate would be needed for the 2nd option.
Hi Jay,
Depends what you are looking to do. The solution user endpoints go through the reverse proxy, so your traffic is all going through the certificate anyway. At that point, you really don’t have to replace your endpoints.
Thanks,
Sean
Well, something broke overnight. vMotion is dead. VMs are stuck in their folders, and can’t be migrated. vmcad-syslog.log has the following entries:
2015-06-18T12:43:37.295953+00:00 info vmcad t@140674804377344: VMCACheckAccessKrb: Authenticated user vcenter.my.tld@vsphere.local
2015-06-18T12:43:41.009782+00:00 info vmcad t@140675108685600: VM Certificate Service exiting…
2015-06-18T12:43:41.010711+00:00 warning vmcad t@140674982287104: error code: 0xffffffff
2015-06-18T12:43:41.010833+00:00 err vmcad t@140674982287104: VMCAListenRpcServer failed (-1)
2015-06-18T12:43:41.010942+00:00 info vmcad t@140674982287104: VMCAListenRpcServer is exiting
2015-06-18T12:43:41.011319+00:00 info vmcad t@140674999072512: Directory sync thread exiting
2015-06-18T12:45:52.772103+00:00 warning vmcad t@140361288795936: error code: 0x00009efc
2015-06-18T12:45:52.772149+00:00 warning vmcad t@140361288795936: error code: 0x00009efc
2015-06-18T12:45:52.772986+00:00 warning vmcad t@140361288795936: error code: 0x00009efc
2015-06-18T12:45:52.773008+00:00 warning vmcad t@140361288795936: error code: 0x00009efc
2015-06-18T12:45:52.788612+00:00 info vmcad t@140361288795936: Initializing database: [/var/lib/vmware/vmca/certs.db]
2015-06-18T12:45:52.808197+00:00 info vmcad t@140361179182848: Directory sync thread starting
2015-06-18T12:45:52.811525+00:00 info vmcad t@140361288795936: rpc_server_use_protseq_ep() succeeded, protoSeq (ncalrpc), endPoint(vmcasvc).
2015-06-18T12:45:52.811537+00:00 info vmcad t@140361288795936: rpc_server_use_protseq_ep() succeeded, protoSeq (ncacn_ip_tcp), endPoint(2014).
2015-06-18T12:45:52.811993+00:00 warning vmcad t@140361179182848: error code: 0xffffffff
2015-06-18T12:45:52.812003+00:00 err vmcad t@140361179182848: Failed to update root certs due to error [4294967295]
2015-06-18T12:45:53.812395+00:00 info vmcad t@140361288795936: VM Certificate Service started.
I’m rolling back. 🙁
Hi Jay,
I don’t think these two are related, but I suggest opening an SR with VMware support.
Thanks,
Sean
Hi Sean, i was looking with an issue while upgrading my VMware SRM giving me an error message Internal error: unexpected error code: -1. based on the upgrade release notes: https://www.vmware.com/support/srm/srm-releasenotes-6-0-0.html it says to
Replace the Platform Services Controller certificate and attempt installation again.
Does this apply here???
i am planning to upgrade from SRM 5.1 to 6 which is a 2 step upgrade 5.1 to 5.5 to 6.0
Hi Nelson,
This only applies if your PSC certificate has expired or is not yet valid; which I haven’t seen because the PSC is relatively new still. Are you using default certificates? You might be able to reset your certificates using step 8 of the certificate-manager tool, then run a modify install of SRM.
Thanks,
Sean
Is it possible that I use a UCC Cert that has up to 30 Subject Alternative Names?
I got one from GoDaddy – and I generated my CSR from an Exchange Server – added the SAN’s for all my servers. I imported the certs provide by GoDaddy – and then exported a cert with private key from the Exchange Server.
I’ve used this certificate on all the servers listed in the Subject Alternative Name line with no issues, but I’m confused on how to use it on the vCenter server. This is a new cluster we are firing up for VDI – and it seems Horizon View requires vCenter server to have a non self-signed cert. In the past I only used the GoDaddy cert on my View Connection/Security servers.
My confusion comes from where do I get the key file if I’m not generating the CSR from the vcenter appliance?
Hi John,
I am not sure the use case for having 30 subject alt names, nor is that supported by VMware. I also don’t think it’s necessary that vCenter is CA for Horizon, I think you may get a warning, but it doesn’t affect anything.
Wherever you generate your CSR, it should generate a keyfile as well, it doesn’t have to be the vCenter Server appliance, it can be any windows box using openssl.
Thanks,
Sean
Hi John,
I’m using the same thing from GoDaddy, but with only 9 SANs (the actual name makes 10 devices). I have it working on 10 devices, including the green padlock on vCenter, thanks to Sean’s brilliant thread here.
I should say, I *had* it working, but it blew up. I had to roll back, and I haven’t had time to play with it since. I don’t think the problems were related to the certificate, but until I get it working again, I can’t be sure. I give it a 95% probability though – certainly enough to give it a shot.
Just be sure the DNS (public and private) FQDN of your vCenter server is one of the SANs. The physical hosts need not be on the list.
getting this error…
Continue operation : Option[Y/N] ? : Y
Status : 0% Completed [Replacing Root Cert…]
Status : Failed
Error Code : 70063
Error Message : Invalid Certificate Chain was gives as input
Hi Mark,
Are you using intermediates? Make sure you use the chain.cer and not just the certificate. Also, make sure that the certificate is first, then root.
Sean
[…] 6 SSL Cert Replacement / Implementation using the Cert Manager automation tool This is useful information if you are going to do Cert work in vSphere 6 – it is different then previous […]
Hi Sean,
thanks for the great article.
After sucessfully replacing the certificates I still see an untrusted certificate without a chain when starting a console session (VMRC, from web client or fat client). Do i have to add certificates to hosts to get this resolved?
You shouldn’t have to, did you make sure to publish the cert?
Hi Rudi,
I had this exact same error and it was due to not having the full certificate chain in the Machine SSL certificate.
Make sure you concatenate the Machine, intermediate and root certificate and use this cert on the option “Please provide valid custom certificate for Machine SSL”
Also, make sure there are no spaces in the cert. Otherwise the certificate tool will error out. I originally copy/pasted all the certificates from a email that had spaces. VECS will not accept these certs
Hello Sean,
Such a wonderful article it is, thanks for sharing.
I am planning to have PSC behind HA, should I mention all hostnames in certool.cfg file i.e. all PSC VMs, vCenter server, dns of PSCVIP e.g. xxx-psc01 /02 /03, xxxpscvip, xxxVC01. As we usually do this in vCAC certs.
One more thing, I chose option 2 (Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates (Using VMCA as a subordinate CA) on primary psc node, I believe it will replace the vmca and all certificates with custom certs, Do i need to replace the Machine SSL separately on the same node or option will take care of certs replacement.
And do i need to run option 2 on secondary psc node as well (using external psc) to replace certs there as well.
I acknowledge the option 3 to 6 will be used on external vCenter.
Thanks beforehand,
It will take care of the machine cert for you, you will need to perform the same steps on the secondary PSC, yes.
Sean
I’ve stepped into a role at a new organization. It looks like my predecessor partially configured VMCA as a sub CA but now I’m having all kinds of issues joining other appliances to the PSC due to lack of chain verification. Is it possible to “re-do” this process without having a significant outage?
Hi Bill,
Yeah, you should be able to just reset the certificates using option 8. Then reimplement them, worst case you may have to reregister some solutions like SRM, View, etc. if you are using them.
Thanks,
Sean
Sean,
I have a situation where I have deployed a brand new VCSA 6.0U1.. chosen to use IP address instead of FQDN.
But then post build, notice ‘localhost.localdom’ applied as a hostname of the appliance.
If I wanted to change that, I can change it via ‘/opt/vmware/share/vami/vami_set_hostname’ and then need to re-issue all certs with cert manager.
Problem is, OVF deployments fail at point of validating storage location during OVF deployment wizard.. and I notice vpxd.log still references the old cert for the STS. (Where the ‘Retrieved trusted STS certificate: O=localhost.localdom,C=US,DC=local,DC=*my SSO site name*,CN=CA,’)
One can change the hostname of an VCSA deployed via IP ?
This is for lab purposes and I have no DNS. I just want to change the ugly presence and DNS propagation of ‘localhost.localdom’ of the OS.
Hi Michael,
You should be able to do this, I have done it before in my lab but you might be hitting some other issue. Have you been able to work around this yet? If no, I suggest opening an SRC with VMware Support.
Thanks,
Sean
Fair enough…
The thing is.. OVF deployment is quite a niche/nested activity .. to only then notice that the hostname change was the cause.
With cert manager you were definitely able to reissue all certs and still deploy OVF with Client Integration Plugin ?
I’ll speak to VMware..
Are you sure you can change it ?
https://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vsphere.install.doc%2FGUID-86FD9381-88B0-40F1-B516-6CC782ED9A14.html
Sentence towards lower part of page..
———————————————————————————————————————————————————————-
If you use an IP address as a system name, you cannot change the IP address and update the DNS settings after deployment.
———————————————————————————————————————————————————————-
Hi Michael,
I guess you cannot.. that’s a good catch. I haven’t been working with certificates a lot lately, mostly doing work surrounding NSX and VIO.
Sean
Hi Sean,
Thank you for the great article. I had a quick question. When I went to generate my CSR and sign it by my CA, I noticed the common name was different than FQDN for my server name. It is generating a CN based on .vsphere.local, instead of the proper domain name we use. Have you encountered this before? Is that the expected behavior? Thanks
Hi Jay,
How are you creating the CSR? You could always try generating it with openSSL.
Thanks,
Sean
Hi Sean, thanks for the helpful walk through! I’ve recently stood up a new 6U1 environment and setup the PSC as a subordinate. This process works but when I replace the machine and solutions cert on vcenter it causes an error message every time I log into vcenter. This message appears related around the CEIP. Any thoughts? Thanks!
Hi Gary,
Can you paste the exact error message here?
Thanks,
Sean
Hello, very nice article
i reinstallted VMCA certificate however after that i’m receving error in my vcenter 6 U1 web client: Error occurred while processing request. Check vSphere WebClient logs for details.
vmware tells: “To resolve this issue, ensure that you have updated the SSL Trust anchors of the vCenter Server and the Platform Services Controller stored in the Lookup Service after you have updated your SSL certificates”
Question:
do replacement VMCA Root certificate with Custom Signing Certificate and replace all Certificates also replaces Lookup Service SSL ?
Hi Gela,
The lookup service port is now exposed through 443, or the reverse proxy. So when you go to the lookup server URL it will pull up the Machine SSL certificate which is replaced during the VMCA subordinate process in my article. It doesn’t actually change the certificate though, if youw ant to replace that you will follow the KB article below.
* Replacing the Lookup Service SSL certificate on a Platform Services Controller 6.0 (2118939)
http://kb.vmware.com/kb/2118939
Thanks,
Sean
Hello.
I’ve been using this walkthrough in lab in order to configure my windows vSphere server as a subordinate CA. My problem is that despite being able to generate a maching_ssl.cer based on the info supplied in certool.cfg I’m not able to generate vmCA.cer with correct info.
I’m in Norway and my CA-certificate insist C=US (The tool – certificate-manager does not seem to read certool.cfg? I’m using fqdn.)
Perhaps I’m facing the same problem as Jay? (September 25.)
Thanks!
Hi Selma,
I’ll have to check but I think there is a known issue with this. You may be able to just run through creating the CSR through openssl then getting in signed with your CA.
Thanks,
Sean
Thanks. I’ll look into that workaround. 🙂
Hi
I ended up using Derek Seaman’s toolkit script.
http://www.derekseaman.com/2015/02/vsphere-6-0-install-pt-1-introduction.html
That worked and by using the methods outlined for windows PSC + online windows root CA (by using part 1, 8, 9, 10, 11 & 12) I both got a correctly signed subordinate CA-certificate, and finally successfully replaced the machine ssl with one that had c = no.
😀
Regards!
Hi Selma,
Very nice, Derek does great work. It’s much easier than the manual process 🙂
Sean
Hi Sean, thanks for this awesome guide. I do have one problem with the console connection to the VM.
When i connect to vCenter everything is “green”, however, as soon as i open a console connection to a VM I am getting a certificate error. The certificate is exactly the same as on vCenter but this is the only difference: http://i.imgur.com/8u8wSVk.png
The certificate somehow cant be checked for its validity.
Do you also have this problem, or do you know how to fix this?
Hi Eagleman,
This is a known issue with 9443 and the console, please subscribe to kb.vmware.com/kb/2129180 when the fix is realeased.
Thanks,
Sean
Sean,
I upgraded my existing vCenter from 5.5 to 6.0 U1b. I had installed custom certs on the 5.5 version for the following services: vCenter, Web Client, Log Browser & VUM. They expire next week. Can I merely request a new, single cert for all services (via the reverse HTTP Proxy), run option 8 on the certificate manager tool (Reset All Certificates), then run option 1 (Replace Machine SSL certificatate with Custom Certificates). Or, do you have a better idea?
Hi Matt,
I apologize for the delayed response, it looks like they are already past they expiration date based on your post. Did you end up finding the answer or getting help from VMware support?
Thanks,
Sean
Hi Sean,
Great article !!. I followed your article and it all worked after many hours trying VMware doco. I have added a second PSC to the existing PSC domain and I am getting certificate issues ( not the full chain) Do I just treat like a vCenter and run through options 3 and 6 like you mentioned in Brian’s post ?
Many Thanks
Hi Sean,
Many, many thanks for the great articles (also replacing certificates on ESXi servers).
For me everything run fine except the certificate from VAMI (web console on port 5480). Do you know if there is an issue or if there is a separate procedure for this particular component ?
My environement :
– vSphere server appliance 6 update 2
– Custom vmware CA certificates then (from AD CS)
– ESXi hosts : vSphere 6 update 2 as well
That is my only problem., everything else run smoothly 🙂
Thanks a lot
PS : I’ve noted with a test appliance that if you take the same certificate name/description for the intermediate VMCA and all other generated certificates, it might give the problem of non trusted certificate than Brian mentionned (April 4, 2015 – 7:59 pm). You have to give a different name for the VMCA and the generated certificates. Might be useful for some others 😉
Hi Sean,
Excellent post, thank you.
I recently upgraded my existing vCenter from 5.5 to 6.0 U2 in an embedded PSC environment on Windows. I would like to know which steps must be performed in order to make the VMCA to act as a subordinate CA for the VMware infrastructure. I followed the procedure indicated above to “Replace VMCA Root certificate with custom signing” but when finished and all services restarted, I have experienced several issues on my clusters being disconnected…
The same happened with the Veeam integration; it looks like nobody was able to communicate correctly. Finally, I have had to roll back the action.
Just a side note, in my implementation of Certificate Manager I found more questions that are not posted in neither blog, for example running option 2 the app asks me if I wish to generate all certificates using the configuration file, and later (after SSO) it asks for reconfiguring all CFG files individually (MACHINE_SSL_CERT.cfg file exists and so on). Is it normal?
Thanks again.
Hi,
A question here. I have synchronized my VCSA with ESXi Host. If i change the time on esx the vcsa screws up and vpxd service fails to start. Any idea on how to solve it?
I updated the machine_ssl certificate using certificate manager which replaced the certificate. I verified using the following 2 commands:
1. vecs-cli.exe entry getcert –store machine_ssl_cert –alias __MACHINE_CERT
2. openssl.exe s_client -servername servername.company.com -connect localhost:443 -showcerts
However, only the host cert was listed and not the full chain (intermediate and root certs). I had to manually replace the certificate using the following commands:
Delete certificate: vecs-cli.exe entry delete –store MACHINE_SSL_CERT –alias __MACHINE_CERT
Create certificate: vecs-cli entry create –store MACHINE_SSL_CERT –alias __MACHINE_CERT –cert c:\certs\ca_chain.crt –key c:\certs\machine_ssl_priv.key
Restart virtualcenter.
This correct the chain issue for port 443. However, I’m having the same problem with port 9443. It appears to be successfully replaced using the certificate manager, however, only the host cert is listed, and not the full chain. This is evident when I run the following command:
openssl.exe s_client -servername servername.company.com -connect localhost:9443 -showcerts
Do you know how I can manually replace the certificate for port 9443 to insure that it uses the cert with the full chain? Thanks in advance.
Sean, this is an excellent article . good work on that.
have a question related to the design. i am working on designing vmca for our vsphere 6 stuff. we have 14 environments with 2 psc’s and 1 vcenter for each environment that belong to same domain at the moment.
in above scenario, if i say all 28 psc’s combined acts as a subordinate CA for their own environment, it might worry our security teams.
just wondering if we can have 2 dedicated appliances that does the subordinate ca job for all the environments. is that even possible?
Hi Sean,
Can you provide me specific steps to replace Custom ssl certificate in vsphere 6 without using VCMA , also certificate manager tool dont use certool.cfg file by default that is bug in vsphere 6.0 & U1b .IF CSR need to generate certool.exe need use . but for solution user what is command to generate csr ?
and also how we will replace ssoserver certificate in vsphere 6
Thanks
Aanand
Anand.. did you get your answer? I need help cleaning up and starting over. our vcenter services will not start and the cert-manager errors out when trying to create the self signed certs
Hey, great writeup! So I sucessfully did the first portion of your article “Replace Machine (Reverse HTTP Proxy) Certificate with Custom Certificate”. Now I can log into the vSphere web client without any certificate errors, great! However now if I check vSphere update manager, or use the C# client to my 6.0 vCenter server, I get an error about vSphere update manager certificate untrusted, and now VUM does not work.
Also vSphere replication now says error – configuration required, and I fear SRM is hosed too.
I did this first step with our Microsoft CA on both our vcenter and DR vcenter which we replicate to. Do I have to also do this portion “Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates (Using VMCA as a subordinate CA)”? I’m stuck here because when creating the request it seems to repeat itself over and over again (asking for country, name, organization, etc…) and I answer each question exactly the same (though everything is populated from that cfg file except for the FQDN at the end. Then it says vpxd.cfg file exisits, do you with so reconfigure? I don’t know if I should answer yes to that at this point?
Any ideas? Thanks!
Note for 6.0 U2 users, you must set each name to be unique (machine, web-client, vpxd, vpxd-e cannot match the CA Cert).
Failure to use unique names will cause a generic chain error.
Hi guys,
I have problem with deleting expired certificate in TRUSTED_ROOTS keystore. (which issued in my CA)
Thaed expired certificate trigers critical alarm in vCenter.
Example from xpxd.log there:
2016-10-14T17:22:28.570+02:00 warning vpxd[05988] [Originator@6876 sub=Main opID=CheckCertificateExpiry-427c3c55]
[Vpxd::VecsUtil::CheckCertificatesFromStore] Certificate [Subject:xxx] from store TRUSTED_ROOTS will expire on 2016-07-31 19:37:46.000.
I found that this certificate is stored in TRUSTED_ROOTS with commands:
vecs-cli store list
vecs-cli entry list –store TRUSTED_ROOTS –text
Then when I try to remove the certificate from store:
vecs-cli entry delete –store TRUSTED_ROOTS –alias a9a6c9e47d1869594157d65f176c8d83c616d61e
…certificate is deleted
…but it comes back after a moment
Do you know how to remove it permanently?
Regards
Kris
big problem with the vcsaU1 right now. I’ve created csr’s for all services and uploaded certs back to vcsa. cert-manager will not install any of them. They all fail and rollback fails both on machine certs as well as ALL solutions certs. Also NOW it fails to recreate VMCA cert and also fails on a “reset all certificates” attempt.
There must be a manual way to clean up files and regenerate self signed certs or upload those signed by our CA. We are currently unable to connect to the vcenter with our vdi and this is now our #1 priority. Any immediate help is appreciated.
Hi Tommy,
I faced the same issue as you, how did you manage to solve the certification problem? Thanks.
Regards,
Kef
Hi Sean
great article, I have a question please, if i dont want to use an internal CA at all and just wanted to use Go Daddy certs, can i do this and still get the PSC to be a subordinate or would i not be able to use that feature and have to replace every certificate on every host manually?
thanks
If you are issuing public certs you cannot be a subordinate CA.
Hello, Quick question, I have vcenter appliance 6.0 ( embedded PSC) whose custom certificate expired recently. VMware support is recommending to build a new vCenter. Is there a way I can replace expired certificate. Can I replace expired certificate with VMCA certificate using the inbuilt tool (option 4) ? Will it work.
Dear Sean,
thank you for all your work and effort you put into this. You seem to have good knowledge and understanding of this topic, which is very helpful. I have been working with vSphere for almost three years now, and unfortunately I’m new to certificates as unexceptionally all clients I’ve been working with have been using self-signed certificates and – as far as I can tell – they had no intention to change this. Until now …
On Monday I have upgraded a client’s environment with ~80 hosts and roughly 800 VMs to v6.0 U3 (latest). Now, due to security concerns, the client needs to have all self-signed certificates replaced by certificates issued by their inhouse root CA. So far, so good.
I understand that much has changed since 5.1 or 5.5 and that by the use of the tool “certificate manager” many things have become easier. My problem is, that I don’t understand the difference between all this. The wizard offering 1., 2., 3. and so on is not as inuitive to me as it probably is to you.
Here we use the VCSA and the VMCA is not supposed to be used as a SubCA. So I understand from KB article 2097936 that in this case the internal CA is bypassed as all certificates will have to be issued directly by the inhouse root CA and there are no cert chains.
Now when I need to replace all certificates, which I’m afraid I’ll have to do, then what? I really tried to understand what I need to do, but I just couldn’t. In this context I can’t even tell what the Machine SSL certificate is, or the Solution user certificate. Everything I did btw. produced .csr files that only had 2048 bit encryption, but our inhouse root CA guy says he needs 4096 bit and I have failed to find a solution for that, too. Have been looking for this but couldn’t find an answer so maybe we’ll have to go with 2048 bit – it’s becoming more and more annoying :-/ so I was hoping and thus would kindly ask you what needs to be done in order to have all certificates replaced in terms of selecting what number in the wizard exactly.
Sorry for this long post, and thank you in advance.
Tobi
Hello Sean,
excellent article, thanks, I have an issue in my environment that I hope you can help me in someway to figure it out,
environment:
1 windows server vcenter server 6.0u2 with the PSC embedded.
after upgrade to vsphere 6.0u3 I got PSC visualization issues:
1- at PSC page –> Certificates –> Certificates Store
I’m not able to visualize anymore all certificates, machine_ssl_cert, trusted_roots, etc…
2- at PSC page –> Certificates –> Certificate Management
I’m not able to visualize anymore all certificates at “solution user certificate” neither at “trusted root certificates”
before the upgrade I double check that all certificates were there, I already try to regenerate a new VMCA Root Certificate and replace all certificates with the certificate manager, all the process completed without errors.
I took some snapshots during the process just in case.
Thanks for your time,
Iesoua