Fixing Orphaned Configs in the Ambari DB

When upgrading Ambari Server, in our case from version 2.6.0 to 2.6.2.0, database inconsistencies can occur. For us, Ambari Server would fail to start, and would report database inconsistencies.

Database inconsistency log file:

/var/log/ambari-server/ambari-server-check-database.log

Fixing these issues will require the Ambari DB to have records altered. Please be sure to have a valid backup of your Ambari database before proceeding.

After reviewing the log file, here are some of steps taken to resolve our issue;

First we tried restarting Ambari Server with it’s auto-fix option. This will run some generic SQL to try and cleanup what it can find.

# ambari-server restart --auto-fix-database

In our case Ambari was able to clean up a majority of the WARN’s in the log, but some remained. There were some old configs that seemed to be orphaned now, and were not mapped to any services.

Here is an example such warning messages:

"WARN - You have config(s): kerberos-env-version1500071764352,krb5-conf-version1500071764352 that is(are) not mapped (in serviceconfigmapping table) to any service!" occurs as the Ambari Server fails to start

For some context, our Ambari is using an MySQL database.

With issues persisting, manual changes of the records in Ambari were needed. To fix the orphaned configs, you will have to login to your Ambari DB and perform the following queries.

1. List of orphaned configs using the following query:

SELECT config_id,type_name,version_tag FROM clusterconfig WHERE unmapped != 1 AND type_name NOT IN ('cluster-env') AND config_id NOT IN (SELECT cc.config_id FROM clusterconfig cc, serviceconfig sc, serviceconfigmapping scm WHERE cc.type_name != 'cluster-env' AND cc.config_id = scm.config_id AND scm.service_config_id = sc.service_config_id);

2. Create a Temporary table for the orphaned configs.

CREATE TEMPORARY TABLE orphaned_configs AS (SELECT config_id,type_name,version_tag FROM clusterconfig WHERE unmapped != 1 AND type_name NOT IN ('cluster-env') AND config_id NOT IN (SELECT cc.config_id FROM clusterconfig cc, serviceconfig sc, serviceconfigmapping scm WHERE cc.type_name != 'cluster-env' AND cc.config_id = scm.config_id AND scm.service_config_id = sc.service_config_id));

3. Verify if the table “orphaned_configs” has the same number of records mentioned in step 1.

SELECT * FROM orphaned_configs;

4. Update the orphaned configs as “unmapped = 1” using the following query:

UPDATE clusterconfig SET unmapped = 1 WHERE config_id IN (SELECT config_id FROM orphaned_configs);

5. Restart Ambari Server

This should clear up the last of Ambari DB inconsistency warnings, regarding any orphaned configs, and Ambari Server should now be able to start successfully. Still not having success? Here is a community article similar to this post.

It would also be best practice to cleanup Ambari Server historical data during this time. Hortonworks documentation for this step, provided here!
Performing this operation analyzes tables in the Ambari Server database and removes rows that can be deleted which have a –create-date after the –from-date specified in your command.

As per the documentation, here are the steps to purge your Ambari Server historical data;

1.  Stop the Ambari Server by using ambari-server stop.

   # ambari-server stop
   Using python /usr/bin/python
   Stopping ambari-server
   Waiting for server stop
   Ambari Server stopped

2. Run db-purge-history .

   # ambari-server db-purge-history --cluster-name CLUSTER_NAME --from-date 2020-12-01
   Using python  /usr/bin/python
   Purge database history...
   Ambari Server configured for Embedded Postgres. Confirm you have made a backup of the Ambari Server database [y/n] y
   Ambari server is using db type Embedded Postgres. Cleanable database entries older than 2020-12-01 will be purged. Proceed [y/n] y
   Purging historical data from the database ...
   Purging historical data completed. Check the ambari-server.log for details.
   Ambari Server 'db-purge-history' completed successfully.

3. Start the Ambari Server.

   # ambari-server start
   Using python  /usr/bin/python
   Starting ambari-server
   Ambari Server running with administrator privileges.
   Organizing resource files at /var/lib/ambari-server/resources...
   Ambari database consistency check started...
   Server PID at: /var/run/ambari-server/ambari-server.pid
   Server out at: /var/log/ambari-server/ambari-server.out
   Server log at: /var/log/ambari-server/ambari-server.log
   Waiting for server start........................
   Server started listening on 8443
   
   DB configs consistency check: no errors and warnings were found.
   Ambari Server 'start' completed successfully.

Happy Hadooping!

Written by Ryan St. Louis