Single Sign On or Platform Services Controller (PSC) fails to install or upgrade with Error code 1603

 

vCenter Single Sign-On Setup Wizard ended prematurely because of an error

 
Ah, the famous 1603 error code when installing or upgrading the PSC or SSO server. Unfortunately, this is not a clear cut error code, nor is there one way to fix the issue. In fact this error code is so generic you typically need to review other log files to understand what may be happening. That being said, I wanted to compile a one stop shop for troubleshooting this error code, what errors the logs show and any resolutions I have found.
 
The typical error messages you see are below.
 

vCenter Single Sign-On Setup Wizard ended prematurely because of an error
VMware Single Sign-On :  VmSetupUpdateVmdirCert error: 1603
Product: vCenter Single Sign-On -- Installation failed. MSI (s) MSI Closing MSIHANDLE (79) of type 790531 for thread 4416 CustomAction BootstrapAll returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox) MSI (s) 
MainEngineThread is returning 1603

 

Log Locations

 
Typically, these error messages you see are in vminst.log or vim-sso-msi.log in %temp%. However, in order to troubleshoot this further, you will need to investigate other helpful logs shown below.
 
C:\ProgramData\VMware\CIS\logs\vmdird\vdcpromo.log
C:\ProgramData\VMware\CIS\logs\vmdird\vdcsetupldu.log
C:\ProgramData\VMware\CIS\logs\vmdird\vmdir.log
C:\ProgramData\VMware\CIS\logs\vmware-sso\vmware-sts-idmd.log
 

Error Codes and Possible Resolutions

 

1. Unsupported character in SSO Password

 
Error:

vminst.log
Cannot authenticate user
Return code is: InvalidCredentials

Resolution: The password for your SSO Administrator account, administrator@vsphere.local or admin@system-domain, is using an unsupported character. If this is an install, try specifying a password that does not contain any of the characters below. If this is an upgrade, try changing the SSO password, then attempting the upgrade again.

Unsupported Characters in password
Non-ASCII characters
& ; ” ‘ ^ \ ! % or a space

It seems like the friendliest character to use is the ‘@’ symbol.

 

2. Unsupported or non-ASCII character in SSO or PSC install directory

 
Error:

vminst.log
VMware Single Sign-On-build vdcpromo path [c:\Program Files\VMware\Infrastructure\é><\VMware\cis\vmdird\\vdcpromo.exe]
VMware Single Sign-On-build  --- CustomAction execution: VmSetupRollback
 
vmdir.log
ERROR: VmDirAllocateMemory failed, (0) requested.

Resolution: Specify an install directory that does not contain non-ASCII or unsupported characters in the install path.

Unsupported Characters in install path
Non-ASCII characters
A high ASCII character count
Special characters, such as
^ * $ ; ” ‘ ) < Note: I have also heard of issues with installing on the D: drive, but I haven’t confirmed this for the 1603 error. The only time I have seen the D: drive cause issues is when installing the vSphere Web Client.

 

3. Stale files or Prerequisites need to be run first

 
Error:

vim-sso-msi.log
MSI (c) (74:9C) [17:27:13:806]: Note: 1: 1708  
MSI (c) (74:9C) [17:27:13:806]: Product: vCenter Single Sign-On -- Installation failed. MSI (s) MSI (s): Closing MSIHANDLE (79) of type 790531 for thread 4416 CustomAction BootstrapAll returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox
Action : VM_InstallOpenSSL_RB . Rollback VMware OpenSSL is performed ...
Action  : VM_InstallPython_RB . Rollback VMware Python is performed ......
postinstall script Postinstall script: postinstall script BootstrapAll Custom Action returned actual error code 1603 ( note this may not be 100 % accurate if translation happened inside sandbox
 
vminst.log
ERROR: Failed to run the command, return code is 1

Resolution:

To work around this issue, clean up stale SSO files and install the SSO prerequisites before retrying the installation.
 
Open %Temp% in windows explorer
 
Note: If you are navigated to a temp\[1,2,3] directory, navigate back one directory to \temp.
Create a backup folder and move all files in temp to the new directory
Rename the C:\ProgramData\VMware\CIS folder

Uninstall and then reinstall the SSO prerequisites for OpenSSL and Python in the following order:

vCenter_Server_Installation_directory\Single Sign-On\prerequisites
1. VMware-OpenSSL.msi and click Install.
2. VMware-python.msi and click Install.
3. kfw.msi and click Install.

Attempt SSO install again but from the SSO directory, not using Auto-Run.
 

4. Invalid Registry Entries

 
Error:

vim-sso-msi.log
Action TIME: PostInstallScripts. Importing Lookupservice data...
CustomAction DoUpdateAndMigrateTasks returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)
MSI (s) (0C:E8) [TIME]: Hello, I'm your 64bit Elevated custom action server.
Action 09:32:18 PostInstallScripts. Configuring SSO Components...
PostInstallScripts: PostInstallScripts
PostInstallScripts: PostInstallScripts
PostInstallScripts: PostInstallScripts
CustomAction BootstrapAll returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)
Action ended 09:32:18: InstallFinalize. Return value 3.
MSI (s) (0C:B4) [08:56:09:908]: User policy value 'DisableRollback' is 0

Resolution: Open the registry and navigate to HKEY_LOCAL_MACHINE\Software\VMware, Inc.\VMware Infrastructure\SSOServer\FQDNIP
Set the FQDN value found in the Subject Alt Name field of the SSO certificate when browsing to https://FQDN:7444/lookupservice/sdk

 

5. Bad Java Version

 
Resolution: It has been noted that an outdated version of Java may cause this issue too. In one customer’s case they uninstalled Java version 7 update 40 (build 1.7.0_40-b43) and installed the newest version to resolve.
 

6. UAC enabled or not running the installer as administrator

Resolution: If UAC is enabled, I would recommend disabling it and then rebooting the Server. Once that is done, reopen the installer media, go to AutoRun, right click and select “Run as Administrator”
 

 

7. Secondary NICs

Resolution: I have also seen secondary NICs on the SSO or PSC server cause the installation to fail. Be sure to disable these other NICs and attempt the install or upgrade again.
 

 

8. IPv6 Enabled

Resolution: Another common resolution to this error is to disable IPv6 on server. I typically use the Microsoft Fix It program here: DisableIPv6
 

 

7. Linked Mode was not uninstalled before upgrading

Resolution: It’s common for users to leave linked mode in place when upgrading SSO, Inventory Service, vSphere Web Client and vCenter Server. I always recommend that linked mode is broken before any type of upgrade is performed.
 

I am going to continue to update this article as I see more errors and resolutions to this problem, especially as more customer’s attempt to install vSphere 6. If you have any other resolutions, please post them in the comments and I can update the article!
 

Posted by:

Sean Whitney

1 Comment

Leave A Comment

Your email address will not be published. Required fields are marked (required):

Back to Top