- Tom Kopchak
- Nov 26, 2018
- Tested on Splunk Version: 7.2
This tutorial will help you understand how Splunk’s configuration enforcement system (the deployment server) works, and how to avoid some common pitfalls when updating your apps.
Have you ever thought you knew how something worked, only to have your mind blown when you learned more? We recently ran into a scenario that forced us to re-examine our understanding of how the Splunk deployment server behaved with respect to local changes on a deployment client. This was a learning experience for me, so I wanted to share this knowledge in the hope that it can help someone else too.
The Splunk deployment server is considered a configuration enforcement mechanism: that is, it exists to keep the configuration consistent between the deployment_apps directory on the deployment server and the target app directory on the deployment clients of a given server class.
It was our original belief that this was a bidirectional relationship, whereby the client would compare the hash of its app to what was on the deployment server and, if there was a difference, automatically update the app to match the deployment server. In fact, there were some apps - such as the OPSEC LEA app - that (at least in older versions) would write data locally and were considered incompatible with a deployment server due to this mechanism.
This understanding, however, was not quite correct.
What actually happens is that the checksum for an app is calculated once (at app installation) and maintained on the deployment client, in the file serverclass.xml ($SPLUNK_HOME/var/run/serverclass.xml). We’ll cover this file in more detail below, if you’re interested.
Each app in serverclass.xml looks something like this:
<app name="all_deploymentclient" checksum="9052369574012309262" restartSplunkd="true" restartSplunkWeb="false" stateOnClient="enabled" localArchive="/opt/splunk/var/run/all_splunk/all_deploymentclient-1491319340.bundle" installed="true"/>
Note that the checksum is stored in this file. This value is not recalculated after app installation, and will not come into effect until a new version of the app in question is added to the deployment server (and the deployment server is reloaded).
Any local changes made to the app after it is deployed from the deployment server will not result in this stored checksum changing or the app being re-downloaded from the deployment server. The only exception to this is when the app is removed, it will re-download on next check-in.
The fact that checksums are only checked for changes upon a modification to the app on the deployment server can lead to configuration issues down the road. Consider this scenario:
This example illustrates the importance of tracking changes appropriately, and ensuring that any apps managed by the deployment server are not modified locally.
There’s a config file for that! Splunk allows for specific configurations to excluded from deployment server management. This is covered in Splunk Docs here: https://docs.splunk.com/Documentation/Splunk/7.2.1/Updating/Excludecontent
For those of you who like technical details and explanations, let’s dig into the config files a little bit deeper:
As mentioned, the file that controls all of this behavior is $SPLUNK_HOME/var/run/serverclass.xml. This file is included in a Splunk diag, and also can be accessed locally on any system running Splunk or a Splunk Universal Forwarder.
Digging into this file can provide a ton of information about what our deployment client is (supposed to be) doing. Let’s explore a somewhat simplified example:
~# cat SPLUNK_HOME/var/run/serverclass.xml
<?xml version="1.0" encoding="UTF-8"?>
<deployResponse restartSplunkd="false" restartSplunkWeb="false" stateOnClient="enabled" issueReload="false" repositoryLocation="$SPLUNK_HOME/etc/apps" endpoint="$deploymentServerUri$/services/streams/deployment?name=$tenantName$:$serverClassName$:$appName$">
<serverClass name="all_HeavyForwarders">
<app name="if_syslog_inputs" checksum="5279169331485055964" restartSplunkd="true" restartSplunkWeb="false" stateOnClient="enabled" localArchive="/opt/splunk/var/run/all_HeavyForwarders/if_syslog_inputs-1491319339.bundle" installed="true"/>
<app name="infra_outputs" checksum="12306752662921759682" restartSplunkd="true" restartSplunkWeb="false" stateOnClient="enabled" localArchive="/opt/splunk/var/run/all_HeavyForwarders/infra_outputs-1491319339.bundle" installed="true"/>
</serverClass>
<serverClass name="all_SplunkInfrastructure">
<app name="infra_license" checksum="12359899191485497807" restartSplunkd="true" restartSplunkWeb="false" stateOnClient="enabled" localArchive="/opt/splunk/var/run/all_SplunkInfrastructure/infra_license-1491319339.bundle" installed="true"/>
<app name="infra_authentication" checksum="14278720479217993410" restartSplunkd="true" restartSplunkWeb="false" stateOnClient="enabled" localArchive="/opt/splunk/var/run/all_SplunkInfrastructure/infra_authentication-1491319339.bundle" installed="true"/>
</serverClass>
<serverClass name="all_linux_servers">
<app name="baseline_linux_inputs" checksum="227089178054871350" restartSplunkd="true" restartSplunkWeb="false" stateOnClient="enabled" localArchive="/opt/splunk/var/run/all_linux_servers/baseline_linux_inputs-1533928043.bundle" installed="true"/>
</serverClass>
<serverClass name="all_splunk">
<app name="all_deploymentclient" checksum="9052369574012309262" restartSplunkd="true" restartSplunkWeb="false" stateOnClient="enabled" localArchive="/opt/splunk/var/run/all_splunk/all_deploymentclient-1491319340.bundle" installed="true"/>
</serverClass>
</deployResponse>
Each serverClass stanza of this XML file represents a server class in the serverclass.conf configuration on the deployment server. We can see that this example system is a member of the following server classes:
Within each serverClass stanza, we see the associated apps. In this example, this works out to the following structure:
As you can see, this is a really straightforward way to see what apps on a given deployment client are managed by the deployment server and what configuration parameters are applied.
Based on our testing, the checksum calculated on the deployment server is highly dependent on the timestamps within a given app. This means that something as innocuous as touching a file (that is, updating the timestamp) will lead to this checksum changing. Be aware of this when working with deployment apps, as it could lead to unintended Splunk restarts depending on how your server classes and apps are configured (only if you reload the server class or restart the deployment server though)!
For those of you who like proof, here you go (thanks to Brian Glenn for doing the testing). Note that simply changing the timestamp on app.conf is enough to cause a new bundle to be generated:
splunk@splunk:/opt/splunk/etc/system/local$ /opt/splunk/bin/splunk reload deploy-server
Splunk username: admin
Password:
Reloading serverclass(es).
splunk@splunk:/opt/splunk/etc/system/local$ cd /opt/splunk/var/run/tmp/test/
splunk@splunk:/opt/splunk/var/run/tmp/test$ ls -la
total 24
drwx------ 2 splunk splunk 4096 Nov 20 20:21 .
drwx------ 3 splunk splunk 4096 Nov 20 20:21 ..
-rw------- 1 splunk splunk 10240 Nov 20 20:21 test_app-1542745315.bundle
-rw------- 1 splunk splunk 412 Nov 20 20:21 test_app-1542745315.bundle.gz
splunk@splunk:/opt/splunk/var/run/tmp/test$ md5sum ./test_app-1542745315.bundle
f8cb981541713b71d8fbd3084e2b4a3f ./test_app-1542745315.bundle
splunk@splunk:/opt/splunk/var/run/tmp/test$ touch /opt/splunk/etc/deployment-apps/test_app/local/app.conf
splunk@splunk:/opt/splunk/var/run/tmp/test$ /opt/splunk/bin/splunk reload deploy-server
Reloading serverclass(es).
splunk@splunk:/opt/splunk/var/run/tmp/test$ ls -la
total 24
drwx------ 2 splunk splunk 4096 Nov 20 20:22 .
drwx------ 3 splunk splunk 4096 Nov 20 20:21 ..
-rw------- 1 splunk splunk 10240 Nov 20 20:22 test_app-1542745373.bundle
-rw------- 1 splunk splunk 414 Nov 20 20:22 test_app-1542745373.bundle.gz
splunk@splunk:/opt/splunk/var/run/tmp/test$ md5sum ./test_app-1542745373.bundle
3899c2f7bc42d9df20fef457d27bf88a ./test_app-1542745373.bundle
splunk@splunk:/opt/splunk/var/run/tmp/test$
If this functionality is not desired, it can be manipulated with the crossServerChecksum option in serverclass.conf. Setting this to true will result in the md5sum not changing if only the timestamp on a file in a deployment app is modified, as demonstrated below:
root@hdf-template:/opt/splunk/var/run/tmp/test# head -2 /opt/splunk/etc/system/local/serverclass.conf
[global]
crossServerChecksum = true
root@hdf-template:/opt/splunk/var/run/tmp/test# /opt/splunk/bin/splunk reload deploy-server
Reloading serverclass(es).
root@hdf-template:/opt/splunk/var/run/tmp/test# ls -la
total 16
drwx------ 2 root root 77 Dec 3 10:54 .
drwx------ 3 root root 18 Dec 3 10:47 ..
-rw------- 1 root root 10240 Dec 3 10:54 test_app-1543852487.bundle
-rw------- 1 root root 403 Dec 3 10:54 test_app-1543852487.bundle.gz
root@hdf-template:/opt/splunk/var/run/tmp/test# md5sum test_app-1543852487.bundle
22bd50b83c9892371675c494b356e6ae test_app-1543852487.bundle
root@hdf-template:/opt/splunk/var/run/tmp/test# touch /opt/splunk/etc/deployment-apps/test_app/local/app.conf
root@hdf-template:/opt/splunk/var/run/tmp/test# /opt/splunk/bin/splunk reload deploy-server
Reloading serverclass(es).
root@hdf-template:/opt/splunk/var/run/tmp/test# ls -la
total 16
drwx------ 2 root root 77 Dec 3 10:55 .
drwx------ 3 root root 18 Dec 3 10:47 ..
-rw------- 1 root root 10240 Dec 3 10:55 test_app-1543852517.bundle
-rw------- 1 root root 403 Dec 3 10:55 test_app-1543852517.bundle.gz
root@hdf-template:/opt/splunk/var/run/tmp/test# md5sum test_app-1543852517.bundle
22bd50b83c9892371675c494b356e6ae test_app-1543852517.bundle
While the bundle ID is regenerated (the .bundle file has a new name), the checksum remains the same. This would result in the app not being redeployed due to a difference in checksum (because there isn’t one).
Hopefully this write-up will save you some pain and suffering when making changes to your deployment apps (or at least help you understand the cause of your pain and suffering if you find out about this too late). If you found this tutorial helpful and there’s another aspect of Splunk configuration you would like to learn more about, let me know!
If you're looking for something different than the typical "one-size-fits-all" security mentality, you've come to the right place.
Sign up for our newsletter by entering your email address below.