Installing IBM BigInsights 4.2 with Ambari Server

IBM BigInsights and Apache BigTop

In this guide, we will show you how to setup and install IBM’s BigInsights 4.x stack and manage it with Apache Ambari. This should also be possible with Apache BigTop, so stay tuned for a future post where we cover that. So, let’s get into BigInsights in the mean time.

CentOS 7 Security Setup

First, we need to prep the security settings on the machine, just as you would for any Hadoop installation. You can use root or a sudo enabled user for the configuration.

Install iptables services, flush the rules and disable/stop the service.

sudo yum install -y iptables*
sudo systemctl start iptables
sudo iptables -F
sudo systemctl restart iptables
sudo systemctl disable iptables
sudo systemctl stop iptables
sudo iptables -L

Set selinux to permissive.

sudo setenforce 0
sudo getenforce

Install yum utilities, DNS tools, wget and Java tools.

sudo yum install -y yum-utils
sudo yum install -y bind-utils
sudo yum install -y wget
sudo yum group install -y "Development Tools"

With security and basic packages taken care of, we can now move onto our installation.

Installing Apache Ambari

Ambari 2.6.1.0 is actually made available through the Apache BigTop 1.5.0 repo, so we will go ahead and add the repo

sudo wget https://downloads.apache.org/bigtop/bigtop-1.5.0/repos/centos-7/bigtop.repo -O /etc/yum.repos.d/bigtop.repo

Install Ambari Server and Agent packages from the BigTop 1.5.0 repo

sudo yum install -y ambari-server
sudo yum install -y ambari-agent
sudo yum -y install mysql-connector-java

Setup Ambari Server

sudo ambari-server setup

Force the ambari-agent to use a specific HTTPS TLS version by editing the ambari-agent.ini file and adding the line below to the security section of the config.

force_https_protocol=PROTOCOL_TLSv1_2

Preparing BigInsights 4.2

Now that Ambari Server has been setup, we want to modify some repo xml files used by the BigInsights 4.2 stack. The Ambari packages we downloaded actually come with BigInsights 4.x builds and we will be using BigInsights 4.2. We need to enable the stack we want first.

sudo vi /var/lib/ambari-server/resources/stacks/BigInsights/4.2/metainfo.xml

Change the BigInsight 4.2 metainfo.xml file active flag from “false” to “true”.

<metainfo>
    <versions>
              <active>true</active>
    </versions>
    <extends>4.1</extends>
    <minJdk>1.7</minJdk>
    <maxJdk>1.8</maxJdk>
</metainfo>

Next we need to change the repo URLs used by this stack as they are outdated. Here are the updated repo URL’s we will be using;

IBM BigInsights 4.2 for RHEL7 (CentOS-7)
https://ibm-open-platform.ibm.com/repos/IOP/rhel/7/x86_64/4.2.x/Updates/4.2.0.0_20180124/

IBM IOP Utils for RHEL7 (CentOS-7)
https://ibm-open-platform.ibm.com/repos/IOP-UTILS/rhel/7/x86_64/1.2/

Edit the repoinfo.xml file

sudo vi /var/lib/ambari-server/resources/stacks/BigInsights/4.2/repos/repoinfo.xml

Change the baseurl flags for the IOP-4.2 and IOP-UTILS-1.2 repos to their new URLs shown here and listed above

<reposinfo>
  <mainrepoid>IOP-4.2</mainrepoid>
  <os family="redhat7">
    <repo>
      <baseurl>https://ibm-open-platform.ibm.com/repos/IOP/rhel/7/x86_64/4.2.x/Updates/4.2.0.0_20180124/</baseurl>
      <repoid>IOP-4.2</repoid>
      <reponame>IOP</reponame>
    </repo>
    <repo>
      <baseurl>https://ibm-open-platform.ibm.com/repos/IOP-UTILS/rhel/7/x86_64/1.2/</baseurl>
      <repoid>IOP-UTILS-1.2</repoid>
      <reponame>IOP-UTILS</reponame>
    </repo>
  </os>
</reposinfo>

Last thing we need to do is remove the ambari-log4j package from the HDFS service, since unfortunately, it is not available in the latest IOP repo.

To make this change edit this file.

sudo vi /var/lib/ambari-server/resources/stacks/BigInsights/4.2/services/HDFS/metainfo.xml

Remove the following content from the file

<package>
  <name>ambari-log4j</name>
</package>

Now we can restart Ambari Server and Agent, then start installing BigInsights 4.2 inside of Ambari.

sudo ambari-server restart
sudo ambari-agent restart

Congratulations, you can now login to your new Ambari Server instance and setup your cluster!

Happy Hadooping!

Written by Ryan St. Louis