Manual creation of the Ambari Server p12 keystore

If you have Ambari Server configured for SSL, the certs Ambari uses for this communication will eventually expire if they are not renewed. If you do let them expire, this is the process you would take to have them generated again from a self signed CA.

1. Stop the Ambari server
   

   ambari-server stop

2. Backup /var/lib/ambari-server/keys and its child directories

   cp -R /var/lib/ambari-server/keys/* ambari-server.bak/

3. Delete the following files from /var/lib/ambari-server/keys:
   • ca.key
   • ca.csr
   • ca.crt
   • pass.txt
   • keystore.p12
   • *.csr
   • *.crt
4. Stop the Ambari Agents
   

   ambari-agent stop

5. Backup /var/lib/ambari-agent/keys and its child directories

   cp -R /var/lib/ambari-agent/keys/* ambari-agent.bak/

6. Delete the following files from /var/lib/ambari-agent/keys:
   • *.key
   • *.csr
   • *.crt
7. Delete the following files from /var/lib/ambari-server/keys/db/
   • index.txt.old
   • index.txt.attr.old
   • serial.old
8. Truncate the following files from /var/lib/ambari-server/keys/db/ by using the command shown
   

   : > /var/lib/ambari-server/keys/db/index.txt

   • • index.txt
     • Index.txt.attr
9. Edit the contents of /var/lib/ambari-server/keys/db/serial, to be exactly as shown.
   

   00

10. Delete all files under /var/lib/ambari-server/keys/db/newcerts

    rm -rf /var/lib/ambari-server/keys/db/newcerts

    
    Normally you would start Ambari Server at this stage and have it generate the keystores. But what if the generated keystore does not work or has issues being created? This can happen if you lose the password for the old SSL key that was used. This is the scenario we will cover going forward.

     

11. Create your CA key, csr, crt and the Ambari Server p12 keystore. If you have HTTPS enabled, you will need to do this for the /var/lib/ambari-server/keys/https.keystore.p12 keystore as well.

    openssl ca -create_serial -out /var/lib/ambari-server/keys/ca.crt -days 365 -keyfile /var/lib/ambari-server/keys/ca.key -selfsign -extenions jdk7_ca -config /var/lib/ambari-server/keys/ca.config -batch -infiles /var/lib/ambari-server/keys/ca.csr

    openssl pkcs12 -export -in /var/lib/ambari-server/keys/ca.crt -inkey /var/lib/ambari-server/keys/ca.key -certfile /var/lib/ambari-server/keys/ca.crt -out /var/lib/ambari-server/keys/keystore.p12 -password pass:YourPasswordHere -passin pass:YourPasswordHere

     

12. Ensure the password you used for the keystore is stored in /var/lib/ambari-server/keys/pass.txt , without any newlines. For HTTPS the file would be /var/lib/ambari-server/keys/https.pass.txt

    tr -d '\n' < /var/lib/ambari-server/keys/pass.txt

     

13. Restart Ambari Server and all Ambari Agents.

    ambari-server restart
    ambari-agent restart

    After restarting Ambari Server, you should see similar entries in /var/log/ambari-server/ambari-server.log

    INFO [main] ShellCommandUtil:63 - Command openssl genrsa -des3 -passout pass:**** -out /var/lib/ambari-server/keys/ca.key 4096  was finished with exit code: 0 - the operation was completely successfully
    INFO [main] ShellCommandUtil:63 - Command openssl req -passin pass:**** -new -key /var/lib/ambari-server/keys/ca.key -out /var/lib/ambari-server/keys/ca.csr -batch was finished with exit code: 0 - the o
    peration was completely successfully.
    INFO [main] ShellCommandUtil:63 - Command openssl ca -create_serial -out /var/lib/ambari-server/keys/ca.crt -days 365 -keyfile /var/lib/ambari-server/keys/ca.key -key **** -selfsign -extensions jdk7_ca -config /var/lib/ambari-server/keys/ca.config -batch -infiles /var/lib/ambari-server/keys/ca.csr was finished with exit code: 0 - the operation was completely successfully.
    INFO [main] ShellCommandUtil:63 - Command openssl pkcs12 -export -in /var/lib/ambari-server/keys/ca.crt -inkey /var/lib/ambari-server/keys/ca.key -certfile /var/lib/ambari-server/keys/ca.crt -out /var/lib/ambari-server/keys/keystore.p12 -password pass:**** -passin pass:****
     was finished with exit code: 0 - the operation was completely successfully.
    INFO [main] ShellCommandUtil:63 - Command find /var/lib/ambari-server/keys -type f -exec chmod 700 {} + was finished with exit code: 0 - the operation was completely successfully.
    INFO [main] ShellCommandUtil:63 - Command chmod 600 /var/lib/ambari-server/keys/pass.txt was finished with exit code: 0 - the operation was completely successfully.
    INFO [qtp-ambari-agent-34] CertificateManager:200 - Signing agent certificate
    INFO [qtp-ambari-agent-34] CertificateManager:220 - Validating agent hostname: ambari.apache.org
    INFO [qtp-ambari-agent-34] CertificateManager:232 - Verifying passphrase
    INFO [qtp-ambari-agent-34] ShellCommandUtil:63 - Command openssl ca -config /var/lib/ambari-server/keys/ca.config -in /var/lib/ambari-server/keys/ambari.apache.org.csr -out /var/lib/ambari-server/keys/ambari.apache.org.crt -batch -passin pass:**** -keyfile /var/lib/ambari-server/keys/ca.key -cert /var/lib/ambari-server/keys/ca.crt was finished with exit code: 0 - the operation was completely successfully.

    After restarting Ambari Agent, you should see similar entries in /var/log/ambari-agent/ambari-agent.log

    INFO security.py:55 - Server require two-way SSL authentication. Use it instead of one-way...
    INFO security.py:179 - Server certicate not exists, downloading
    INFO security.py:202 - Downloading server cert from https://localhost:8440/cert/ca/
    INFO security.py:187 - Agent key not exists, generating request
    INFO security.py:258 - openssl req -new -newkey rsa:1024 -nodes -keyout "/var/lib/ambari-agent/keys/ambari.apache.org.key" -subj /OU=ambari.apache.org/ -out "/var/lib/ambari-agent/keys/ambari.apache.org.csr"
    INFO security.py:195 - Agent certificate not exists, sending sign request
    INFO security.py:93 - SSL Connect being called.. connecting to the server
    INFO security.py:77 - SSL connection established. Two-way SSL authentication completed successfully.

Written by Ryan St. Louis

This is a modification of the Cloudera documentation found here – https://docs.cloudera.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-security/content/regenerating_ssl_certificates.html