Manually installing Spectrum Scale software on Red Hat

There are a few ways to install Spectrum Scale on a cluster. Since version 4.1.1, when Spectrum Scale introduced protocol nodes, the software bundle has included a toolkit to help with installation. Over time, this toolkit has improved considerably — but it takes a bit of finessing to get an environment ready for the toolkit.

Installing from RPMs is not really any more complex. This guide will walk through the steps needed to install the GPFS file system portion of Spectrum Scale.

Preparation

There are few things that need to be prepared on systems that will be running Spectrum Scale:

1. DNS and networking are configured. Spectrum Scale needs to be able to resolve hostnames to IP addresses, as well as reverse look-ups of IP addresses to hostnames. These mappings need to be consistent on all nodes of the cluster.
2. We need synchronized system clocks. This is commonly handled by running some sort of time service like NTP or chronyd on all the systems.
3. We need to be able use the ssh command from at least one of the Spectrum Scale nodes to be able to connect to every other node in the cluster as root without using a password. Commonly all “server” nodes are configured to be able to use ssh to connect to every cluster node. (We will call these nodes that can issue ssh commands the “administrative” nodes.)  (It is possible to configure Spectrum Scale to work with some a non-root account.)
4. We will need a working system software repository. It is rare that a system will have all of the software packages required, and there are so many interdependencies that this really can’t be determined in advance. It is much more reliable to use the package management dependency system to bring in additional components as needed. To summarize: Make sure that yum is working.
5. We need to be careful of firewalls. If all the systems can be collectively protected, then system firewalls can be disabled altogether. Otherwise the appropriate firewall rules need to configured to ensure that Spectrum Scale operates correctly.
6. We probably want to turn off SElinux, or at least put it into permissive mode.

Unpacking the software

The Spectrum Scale software is shipped as a giant shell script with a name like: Spectrum_Scale_Protocols_Advanced-4.2.3.6-x86_64-Linux-install

This can be run on an administrative node:

    # bash Spectrum_Scale_Protocols_Advanced-4.2.3.6-x86_64-Linux-install \
        --silent --text-only

The packages will be unpacked in /usr/lpp/mmfs/4.2.3.6 (or whatever version was used):

    # cd /usr/lpp/mmfs/4.2.3.6
    # ls
    ganesha_debs gpfs_debs gui license object_debs smb_debs zimon_debs
    ganesha_rpms gpfs_rpms installer manifest object_rpms smb_rpms zimon_rpms

The *_rpms directories contain RPMs for systems using rpm as the package manager (Red Hat and clones, SLES). The *_debs directories contain DEBs for systems using dpkg as the package manager (Ubuntu, Debian). Most installs will require packages from several of these directories.

The software packages will be needed on all the nodes of the cluster. You can use a tool like rsync to push the unpacked directory (in this example, /usr/lpp/mmfs/4.2.3.6) to every node of the cluster. Alternatively, you can push the installer script to every node, and unpack on every node. Some people like to put the unpacked directory into a shared location and make it available to every node of the cluster.

Creating the repository

Here is where some complexity comes in: we need to build a repository file we can use to install the packages we need using the yum command. There is a little guidance in the fine manual on creating the yum repository file at https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1ins_manualobjinstall.htm

Or you can use a tool to make a “spectrum_scale” repository file. One such tool can be downloaded from
https://github.com/drrltodd/scale-tools/raw/master/build-scale-repo.sh
You would invoke this tool like this:

    # bash build-scale-repo.sh -d /usr/lpp/mmfs/4.2.3.6 -o rhel7

This will create the file /etc/yum.repos.d/spectrum_scale.repo for Red Hat 7. You need to copy or generate this file on each node of the cluster.

Installing the packages

On each node of the system, run:

    # yum -y install gpfs.base gpfs.docs gpfs.ext gpfs.gpl gpfs.gskit \
    gpfs.msg.en_US gpfs.gss.pmsensors

On the node that will run the GUI, run:

    # yum -y install gpfs.gui gpfs.gss.pmcollector

Making sure commands are on the PATH

The Spectrum Scale commands are all stored in /usr/lpp/mmfs/bin. These are not on the default PATH environment variable. When you issue Spectrum Scale commands, you will need to give the full path name of each command. This gets old very quickly!

Better is to ensure we have the proper directories on the PATH variable. You can do this for your current login session using this command:

    # export PATH=/usr/lpp/mmfs/bin:$PATH

To automatically put this directory on your PATH when you log in, you can put a “scriptlet” in /etc/profile.d that is sourced automatically each time you log in. Put this on each node of the cluster where you are likely to be running Spectrum Scale commands. An example of a scriptlet is at https://github.com/drrltodd/scale-tools/raw/master/scale.sh

Building the kernel modules

Spectrum Scale has a few kernel modules, provided in source code form, that need to be compiled for the running kernel.  These will require a kernel development toolchain, which can be installed by installing the kernel-devel and kernel-headers RPMs. The mmbuildgpl command manages building the kernel. On each system with Spectrum Scale, run:

    mmbuildgpl

This step will need to be repeated each time either the kernel or Spectrum Scale is updated. (Be sure that if you update the kernel, you reboot before rebuilding the kernel modules. The mmbuildgpl tool builds for the running kernel, not the latest kernel installed.)

Next steps

You have now installed the basic Spectrum Scale software on you cluster!  Some systems in the cluster may need additional components, such as any of the “protocols” or “cloud tiering” — installing those are beyond the scope of this article.

With all the software installed, you can begin configuring your actual cluster.