Splunk 9.0: Directory Naming Change

With the introduction of Splunk Enterprise 9.0, Splunk has changed the language used for certain directories in indexer-clustering. Therefore, let’s explore how this works in practice.

Some history and background

In versions of Splunk prior to 9.0, the configurations sent to the indexers would be stored in the $SPLUNK_HOME/etc/master-apps directory on the cluster manager. Configurations pushed to pre-9.0 indexers would end up in the $SPLUNK_HOME/etc/slave-apps directory. This changes with the Splunk Enterprise 9.0 release.

There are now two possible locations for storing apps on a cluster-manager: master-apps and manager-apps. The manager-apps location is the preferred location for new installs, and the master-apps directory is maintained for backwards compatibility. Note that you can use only one of these locations at a time.

On Splunk Enterprise 9.0 indexers, apps received from a cluster manager will end up in the $SPLUNK_HOME/etc/peer-apps directory. This change is automatic and is only managed by the cluster manager, so there shouldn’t be anything you need to account for when upgrading to Splunk 9.0.

However, since the choice of location for apps on the CM is something that the user controls, I had to find out: what happens if someone were to attempt to push configuration from the wrong location?

Pushing a bad config

One immediate question I had when reading about the configuration change: What might happen if someone were to accidentally put a new app in the incorrect directory and attempt to push a bundle to the indexers?

The good news: Splunk figured someone would mess this up, and there is error checking in place to block a push from occurring if there are apps in both the manager-apps and master-apps directories at the same time:

splunk@splunk-cm01:/opt/splunk/etc$ ls manager-apps/
_cluster  infra_misinformed_user
splunk@splunk-cm01:/opt/splunk/etc$ ls master-apps/
_cluster              infra_indexes_base   infra_license
infra_disablekvstore  infra_indexes_local  infra_web
splunk@splunk-cm01:/opt/splunk/etc$ /opt/splunk/bin/splunk ^C
splunk@splunk-cm01:/opt/splunk/etc$ /opt/splunk/bin/splunk apply cluster-bundle
Populated manager-apps location found, but master-apps is also populated. There can be only one. This must be fixed before continuing

<repeated a bunch of times>

Populated manager-apps location found, but master-apps is also populated. There can be only one. This must be fixed before continuing
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.

Encountered some errors while applying the bundle.
manager-apps and master-apps are both populated. There can be only one. Bundle push blocked until all bundles are either in manager-apps (preferred) or master-apps

splunk@splunk-cm01:/opt/splunk/etc$ ls manager-apps/
_cluster  infra_misinformed_user
splunk@splunk-cm01:/opt/splunk/etc$ ls master-apps/
_cluster              infra_indexes_base   infra_license
infra_disablekvstore  infra_indexes_local  infra_web
splunk@splunk-cm01:/opt/splunk/etc$ /opt/splunk/bin/splunk ^C
splunk@splunk-cm01:/opt/splunk/etc$ /opt/splunk/bin/splunk apply cluster-bundle
Populated manager-apps location found, but master-apps is also populated. There can be only one. This must be fixed before continuing

<repeated a bunch of times>

Populated manager-apps location found, but master-apps is also populated. There can be only one. This must be fixed before continuing
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.

Encountered some errors while applying the bundle.
manager-apps and master-apps are both populated. There can be only one. Bundle push blocked until all bundles are either in manager-apps (preferred) or master-apps

splunk@splunk-cm01:/opt/splunk/etc$ ls manager-apps/
_cluster  infra_misinformed_user
splunk@splunk-cm01:/opt/splunk/etc$ ls master-apps/
_cluster              infra_indexes_base   infra_license
infra_disablekvstore  infra_indexes_local  infra_web
splunk@splunk-cm01:/opt/splunk/etc$ /opt/splunk/bin/splunk ^C
splunk@splunk-cm01:/opt/splunk/etc$ /opt/splunk/bin/splunk apply cluster-bundle
Populated manager-apps location found, but master-apps is also populated. There can be only one. This must be fixed before continuing

<repeated a bunch of times>

Populated manager-apps location found, but master-apps is also populated. There can be only one. This must be fixed before continuing
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.

Encountered some errors while applying the bundle.
manager-apps and master-apps are both populated. There can be only one. Bundle push blocked until all bundles are either in manager-apps (preferred) or master-apps

Need a Hand? Hurricane Labs is Here to Help?

To resolve this issue, the administrator just needs to make sure there is only one directory (either master-apps or manager-apps) that contains configuration files. If this does happen in your environment, you’ll hopefully catch it on the first cluster bundle push after the error is made.

Below is a demo showing how this happens in practice.

Migrating from master-apps to manager-apps

To perform a migration from the legacy master-apps directory to the new manager-apps directory on your cluster manager, simply move all the apps to the new manager-apps directory and push a bundle to the indexers as you normally would. Your cluster manager will now be using the new path for apps. Be sure to also update your documentation to point to the new location, and ensure any mechanisms outside of Splunk that would write to the old manager-apps directory (such as configuration management tools) point to the new location as well.

Below is a demo showing the migration process.

Wrap up

All-in-all, this is a relatively minor change in the overall Splunk administration workflow. However, if you’re used to the pre-9.0 way of doing things, it will be a change you’ll need to get used to, but there’s no need to worry about the process accidentally breaking your Splunk environment.