There are a few ways to install Spectrum Scale on a cluster. Since version 4.1.1, when Spectrum Scale introduced protocol nodes, the software bundle has included a toolkit to help with installation. Over time, this toolkit has improved considerably — but it takes a bit of finessing to get an environment ready for the toolkit.
Installing from RPMs is not really any more complex. This guide will walk through the steps needed to install the GPFS file system portion of Spectrum Scale.
There are few things that need to be prepared on systems that will be running Spectrum Scale:
- DNS and networking are configured. Spectrum Scale needs to be able to resolve hostnames to IP addresses, as well as reverse look-ups of IP addresses to hostnames. These mappings need to be consistent on all nodes of the cluster.
- We need synchronized system clocks. This is commonly handled by running some sort of time service like NTP or
chronydon all the systems.
- We need to be able use the
sshcommand from at least one of the Spectrum Scale nodes to be able to connect to every other node in the cluster as
rootwithout using a password. Commonly all “server” nodes are configured to be able to use
sshto connect to every cluster node. (We will call these nodes that can issue
sshcommands the “administrative” nodes.) (It is possible to configure Spectrum Scale to work with some a non-root account.)
- We will need a working system software repository. It is rare that a system will have all of the software packages required, and there are so many interdependencies that this really can’t be determined in advance. It is much more reliable to use the package management dependency system to bring in additional components as needed. To summarize: Make sure that
- We need to be careful of firewalls. If all the systems can be collectively protected, then system firewalls can be disabled altogether. Otherwise the appropriate firewall rules need to configured to ensure that Spectrum Scale operates correctly.
- We probably want to turn off SElinux, or at least put it into permissive mode.
Unpacking the software
The Spectrum Scale software is shipped as a giant shell script with a name like:
This can be run on an administrative node:
# bash Spectrum_Scale_Protocols_Advanced-184.108.40.206-x86_64-Linux-install \ --silent --text-only
The packages will be unpacked in
/usr/lpp/mmfs/220.127.116.11 (or whatever version was used):
# cd /usr/lpp/mmfs/18.104.22.168 # ls ganesha_debs gpfs_debs gui license object_debs smb_debs zimon_debs ganesha_rpms gpfs_rpms installer manifest object_rpms smb_rpms zimon_rpms
*_rpms directories contain RPMs for systems using
rpm as the package manager (Red Hat and clones, SLES). The
*_debs directories contain DEBs for systems using
dpkg as the package manager (Ubuntu, Debian). Most installs will require packages from several of these directories.
The software packages will be needed on all the nodes of the cluster. You can use a tool like
rsync to push the unpacked directory (in this example,
/usr/lpp/mmfs/22.214.171.124) to every node of the cluster. Alternatively, you can push the installer script to every node, and unpack on every node. Some people like to put the unpacked directory into a shared location and make it available to every node of the cluster.
Creating the repository
Here is where some complexity comes in: we need to build a repository file we can use to install the packages we need using the
yum command. There is a little guidance in the fine manual on creating the yum repository file at https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1ins_manualobjinstall.htm
Or you can use a tool to make a “spectrum_scale” repository file. One such tool can be downloaded from
You would invoke this tool like this:
# bash build-scale-repo.sh -d /usr/lpp/mmfs/126.96.36.199 -o rhel7
This will create the file
/etc/yum.repos.d/spectrum_scale.repo for Red Hat 7. You need to copy or generate this file on each node of the cluster.
Installing the packages
On each node of the system, run:
# yum -y install gpfs.base gpfs.docs gpfs.ext gpfs.gpl gpfs.gskit \ gpfs.msg.en_US gpfs.gss.pmsensors
On the node that will run the GUI, run:
# yum -y install gpfs.gui gpfs.gss.pmcollector
Making sure commands are on the
The Spectrum Scale commands are all stored in
/usr/lpp/mmfs/bin. These are not on the default
PATH environment variable. When you issue Spectrum Scale commands, you will need to give the full path name of each command. This gets old very quickly!
Better is to ensure we have the proper directories on the
PATH variable. You can do this for your current login session using this command:
# export PATH=/usr/lpp/mmfs/bin:$PATH
To automatically put this directory on your
PATH when you log in, you can put a “scriptlet” in
/etc/profile.d that is sourced automatically each time you log in. Put this on each node of the cluster where you are likely to be running Spectrum Scale commands. An example of a scriptlet is at https://github.com/drrltodd/scale-tools/raw/master/scale.sh
Building the kernel modules
Spectrum Scale has a few kernel modules, provided in source code form, that need to be compiled for the running kernel. These will require a kernel development toolchain, which can be installed by installing the
kernel-headers RPMs. The
mmbuildgpl command manages building the kernel. On each system with Spectrum Scale, run:
This step will need to be repeated each time either the kernel or Spectrum Scale is updated. (Be sure that if you update the kernel, you reboot before rebuilding the kernel modules. The
mmbuildgpl tool builds for the running kernel, not the latest kernel installed.)
You have now installed the basic Spectrum Scale software on you cluster! Some systems in the cluster may need additional components, such as any of the “protocols” or “cloud tiering” — installing those are beyond the scope of this article.
With all the software installed, you can begin configuring your actual cluster.