Introduction
One of my least favorite features in Splunk is KV Store - mainly, because whenever I have to deal with it as a Splunk administrator, it’s broken in some horrible new way that I need to figure out. The goal of this post is to capture one of these troubleshooting adventures that we recently encountered in the hopes that it might help someone who runs into this same problem in the future.
Background
Beginning with Splunk Enterprise 8.1, Splunk introduced a new storage engine for KVstore (WiredTiger). When upgrading to Splunk Enterprise 9.0 or later, you are required to migrate to the new storage engine. You can also migrate to this storage engine prior to upgrading to Splunk Enterprise 9.0 if you want.
We’ve done this migration for a bunch of clients, but every once in a while, we’ve seen some issues that require additional troubleshooting, especially if there is an error or failure in the migration or upgrade process.
While I’m not sure of the exact circumstances that led to this exact error, it appears that the root cause may have been related to a Splunk version conflict where a system was upgraded to Splunk 9.0, and then an older version of Splunk 8.x was started for some reason. The end result (and where I entered this story) was a system running Splunk 9.0 with a KV Store that wouldn’t start.
Symptoms of the issue
Based on the output of splunkd.log on the broken system, it appeared that KV Store on this host was looking to start version 4.2 with the mmapv1 (legacy KV Store) storage engine. Even with storageEngine = mmapv1 in server.conf, the system was trying to migrate to WiredTiger and failing.
Furthermore, the kvstore files in $SPLUNK_HOME/var/lib/splunk/kvstore/mongo all ended with a .ns extension, which indicates that the storage engine was mmapv1 and not WiredTiger. After a conversion to WiredTiger, you’ll instead see a bunch of files with .wt extensions.
For some reason, the system was convinced that it was running a more current version of KVstore, but the data files in KVstore disagreed. When this was occurring, KV Store didn’t start or function, and there were no logs in mongod.log (at all).
Fortunately, the splunkd.log file had some more output as to what was happening:
04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Starting mongod with executable name=mongod version=kvstore version 4.2 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Setting env var LC_ALL=C 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --dbpath /opt/splunk/var/lib/splunk/kvstore/mongo 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --storageEngine mmapv1 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --port 8191 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --timeStampFormat iso8601-utc 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --smallfiles 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --oplogSize 200 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --keyFile /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --setParameter enableLocalhostAuthBypass=0 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --setParameter oplogFetcherSteadyStateMaxFetcherRestarts=0 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --replSet F50190FF-36F7-486B-B34F-FDE64B4665E9 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --bind_ip=0.0.0.0 (all ipv4 addresses) 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --sslMode requireSSL 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --sslAllowInvalidHostnames 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --sslPEMKeyFile /opt/splunk/etc/auth/server.pem 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --sslPEMKeyPasswod ***** 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --tlsDisabledProtocols noTLS1_0,noTLS1_1 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --sslCipherConfig ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --nounixsocket 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --noscripting 04-10-2023 11:42:33.828 +0000 ERROR MongodRunner [45128 MongodLogThread] - mongod exited abnormally (exit code 2, status: PID 45129 exited with code 2) - look at mongod.log to investigate. 04-10-2023 11:42:33.858 +0000 ERROR KVStoreBulletinBoardManager [32487 MainThread] - Failed to migrate KV Store storage engine to WiredTiger. This prevented running the latest version of KV Store. Migrate KV Store storage engine to WiredTiger and then upgrade KV Store. The Mmapv1 engine is deprecated. 04-10-2023 11:42:33.858 +0000 ERROR KVStoreBulletinBoardManager [45128 MongodLogThread] - KV Store process terminated abnormally (exit code 2, status PID 45129 exited with code 2). See mongod.log and splunkd.log for details. 04-10-2023 11:42:33.858 +0000 WARN KVStoreConfigurationProvider [45128 MongodLogThread] - Action scheduled, but event loop is not ready yet 04-10-2023 11:42:33.858 +0000 ERROR KVStoreBulletinBoardManager [45128 MongodLogThread] - KV Store changed status to failed. KVStore process terminated.. 04-10-2023 11:42:33.858 +0000 WARN KVStoreConfigurationProvider [32487 MainThread] - Action scheduled, but event loop is not ready yet 04-10-2023 11:42:33.864 +0000 WARN KVStoreConfigurationProvider [32487 MainThread] - Action scheduled, but event loop is not ready yet 04-10-2023 11:42:33.864 +0000 INFO loader [32487 MainThread] - SAML cert db registration with KVStore successful 04-10-2023 11:42:33.864 +0000 INFO CertStorageProvider [32487 MainThread] - Updating status from unknown to failed
04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Starting mongod with executable name=mongod version=kvstore version 4.2 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Setting env var LC_ALL=C 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --dbpath /opt/splunk/var/lib/splunk/kvstore/mongo 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --storageEngine mmapv1 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --port 8191 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --timeStampFormat iso8601-utc 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --smallfiles 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --oplogSize 200 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --keyFile /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --setParameter enableLocalhostAuthBypass=0 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --setParameter oplogFetcherSteadyStateMaxFetcherRestarts=0 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --replSet F50190FF-36F7-486B-B34F-FDE64B4665E9 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --bind_ip=0.0.0.0 (all ipv4 addresses) 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --sslMode requireSSL 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --sslAllowInvalidHostnames 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --sslPEMKeyFile /opt/splunk/etc/auth/server.pem 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --sslPEMKeyPasswod ***** 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --tlsDisabledProtocols noTLS1_0,noTLS1_1 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --sslCipherConfig ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --nounixsocket 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --noscripting 04-10-2023 11:42:33.828 +0000 ERROR MongodRunner [45128 MongodLogThread] - mongod exited abnormally (exit code 2, status: PID 45129 exited with code 2) - look at mongod.log to investigate. 04-10-2023 11:42:33.858 +0000 ERROR KVStoreBulletinBoardManager [32487 MainThread] - Failed to migrate KV Store storage engine to WiredTiger. This prevented running the latest version of KV Store. Migrate KV Store storage engine to WiredTiger and then upgrade KV Store. The Mmapv1 engine is deprecated. 04-10-2023 11:42:33.858 +0000 ERROR KVStoreBulletinBoardManager [45128 MongodLogThread] - KV Store process terminated abnormally (exit code 2, status PID 45129 exited with code 2). See mongod.log and splunkd.log for details. 04-10-2023 11:42:33.858 +0000 WARN KVStoreConfigurationProvider [45128 MongodLogThread] - Action scheduled, but event loop is not ready yet 04-10-2023 11:42:33.858 +0000 ERROR KVStoreBulletinBoardManager [45128 MongodLogThread] - KV Store changed status to failed. KVStore process terminated.. 04-10-2023 11:42:33.858 +0000 WARN KVStoreConfigurationProvider [32487 MainThread] - Action scheduled, but event loop is not ready yet 04-10-2023 11:42:33.864 +0000 WARN KVStoreConfigurationProvider [32487 MainThread] - Action scheduled, but event loop is not ready yet 04-10-2023 11:42:33.864 +0000 INFO loader [32487 MainThread] - SAML cert db registration with KVStore successful 04-10-2023 11:42:33.864 +0000 INFO CertStorageProvider [32487 MainThread] - Updating status from unknown to failed
04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Starting mongod with executable name=mongod version=kvstore version 4.2 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Setting env var LC_ALL=C 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --dbpath /opt/splunk/var/lib/splunk/kvstore/mongo 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --storageEngine mmapv1 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --port 8191 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --timeStampFormat iso8601-utc 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --smallfiles 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --oplogSize 200 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --keyFile /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --setParameter enableLocalhostAuthBypass=0 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --setParameter oplogFetcherSteadyStateMaxFetcherRestarts=0 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --replSet F50190FF-36F7-486B-B34F-FDE64B4665E9 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --bind_ip=0.0.0.0 (all ipv4 addresses) 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --sslMode requireSSL 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --sslAllowInvalidHostnames 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --sslPEMKeyFile /opt/splunk/etc/auth/server.pem 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --sslPEMKeyPasswod ***** 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --tlsDisabledProtocols noTLS1_0,noTLS1_1 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --sslCipherConfig ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --nounixsocket 04-10-2023 11:42:33.804 +0000 INFO MongodRunner [45127 KVStoreConfigurationThread] - Using mongod command line --noscripting 04-10-2023 11:42:33.828 +0000 ERROR MongodRunner [45128 MongodLogThread] - mongod exited abnormally (exit code 2, status: PID 45129 exited with code 2) - look at mongod.log to investigate. 04-10-2023 11:42:33.858 +0000 ERROR KVStoreBulletinBoardManager [32487 MainThread] - Failed to migrate KV Store storage engine to WiredTiger. This prevented running the latest version of KV Store. Migrate KV Store storage engine to WiredTiger and then upgrade KV Store. The Mmapv1 engine is deprecated. 04-10-2023 11:42:33.858 +0000 ERROR KVStoreBulletinBoardManager [45128 MongodLogThread] - KV Store process terminated abnormally (exit code 2, status PID 45129 exited with code 2). See mongod.log and splunkd.log for details. 04-10-2023 11:42:33.858 +0000 WARN KVStoreConfigurationProvider [45128 MongodLogThread] - Action scheduled, but event loop is not ready yet 04-10-2023 11:42:33.858 +0000 ERROR KVStoreBulletinBoardManager [45128 MongodLogThread] - KV Store changed status to failed. KVStore process terminated.. 04-10-2023 11:42:33.858 +0000 WARN KVStoreConfigurationProvider [32487 MainThread] - Action scheduled, but event loop is not ready yet 04-10-2023 11:42:33.864 +0000 WARN KVStoreConfigurationProvider [32487 MainThread] - Action scheduled, but event loop is not ready yet 04-10-2023 11:42:33.864 +0000 INFO loader [32487 MainThread] - SAML cert db registration with KVStore successful 04-10-2023 11:42:33.864 +0000 INFO CertStorageProvider [32487 MainThread] - Updating status from unknown to failed
Need a Hand? Hurricane Labs is Here to Help?
The splunk show kvstore-status command showed the following output:
[splunk@splunkserver kvstore_upgrade]$ /opt/splunk/bin/splunk show kvstore-status This member: backupRestoreStatus : Ready disabled : 0 guid : F50190FF-36F7-486B-B34F-FDE64B4665E9 port : 8191 standalone : 1 status : failed storageEngine : mmapv1
[splunk@splunkserver kvstore_upgrade]$ /opt/splunk/bin/splunk show kvstore-status This member: backupRestoreStatus : Ready disabled : 0 guid : F50190FF-36F7-486B-B34F-FDE64B4665E9 port : 8191 standalone : 1 status : failed storageEngine : mmapv1
[splunk@splunkserver kvstore_upgrade]$ /opt/splunk/bin/splunk show kvstore-status This member: backupRestoreStatus : Ready disabled : 0 guid : F50190FF-36F7-486B-B34F-FDE64B4665E9 port : 8191 standalone : 1 status : failed storageEngine : mmapv1
Now, we needed to figure out why the KV Store status was showing as failed (and more importantly) how to fix it.
Researching the solution
Reviewing logs on multiple Splunk environments led us to a clue in the migrate.log file. KV Store upgrades looked to have these types of entries recorded:
[App Key Value Store migration] Starting migrate-kvstore. Created version file path=/opt/splunk/var/run/splunk/kvstore_upgrade/versionFile36 Created version file path=/opt/splunk/var/run/splunk/kvstore_upgrade/versionFile40
[App Key Value Store migration] Starting migrate-kvstore. Created version file path=/opt/splunk/var/run/splunk/kvstore_upgrade/versionFile36 Created version file path=/opt/splunk/var/run/splunk/kvstore_upgrade/versionFile40
[App Key Value Store migration] Starting migrate-kvstore. Created version file path=/opt/splunk/var/run/splunk/kvstore_upgrade/versionFile36 Created version file path=/opt/splunk/var/run/splunk/kvstore_upgrade/versionFile40
and
[App Key Value Store migration] Starting migrate-kvstore. Created version file path=/opt/splunk/var/run/splunk/kvstore_upgrade/versionFile42
[App Key Value Store migration] Starting migrate-kvstore. Created version file path=/opt/splunk/var/run/splunk/kvstore_upgrade/versionFile42
[App Key Value Store migration] Starting migrate-kvstore. Created version file path=/opt/splunk/var/run/splunk/kvstore_upgrade/versionFile42
The purpose of these files is not documented, and they contain no content:
splunk@splunksearch02:/opt/splunk/var/run/splunk/kvstore_upgrade$ file versionFile42 versionFile42: empty
splunk@splunksearch02:/opt/splunk/var/run/splunk/kvstore_upgrade$ file versionFile42 versionFile42: empty
splunk@splunksearch02:/opt/splunk/var/run/splunk/kvstore_upgrade$ file versionFile42 versionFile42: empty
Our best guess is that the presence of this file tells Splunk what version of the KV Store engine to use. We decided to try removing the versionFile40 and versionFile42 files, and creating a versionFile36 in its place to correspond to a version that used the old mmapv1 storage engine.
At this point, we crossed our fingers and restarted Splunk. To our relief, Splunk restarted and KV Store successfully came up this time too!
[splunk@splunkserver kvstore_upgrade]$ /opt/splunk/bin/splunk show kvstore-status This member: backupRestoreStatus : Ready date : Mon Apr 10 12:22:12 2023 dateSec : 1681129332.136 disabled : 0 guid : F50190FF-36F7-486B-B34F-FDE64B4665E9 oplogEndTimestamp : Mon Apr 10 12:22:12 2023 oplogEndTimestampSec : 1681129332 oplogStartTimestamp : Mon Apr 10 00:17:05 2023 oplogStartTimestampSec : 1679617025 port : 8191 replicaSet : F50190FF-36F7-486B-B34F-FDE64B4665E9 replicationStatus : KV store captain standalone : 1 status : ready storageEngine : mmapv1 KV store members: 127.0.0.1:8191 configVersion : 1 electionDate : Mon Apr 10 12:21:49 2023 electionDateSec : 1681129309 hostAndPort : 127.0.0.1:8191 optimeDate : Mon Apr 10 12:22:12 2023 optimeDateSec : 1681129332 replicationStatus : KV store captain uptime : 24
[splunk@splunkserver kvstore_upgrade]$ /opt/splunk/bin/splunk show kvstore-status This member: backupRestoreStatus : Ready date : Mon Apr 10 12:22:12 2023 dateSec : 1681129332.136 disabled : 0 guid : F50190FF-36F7-486B-B34F-FDE64B4665E9 oplogEndTimestamp : Mon Apr 10 12:22:12 2023 oplogEndTimestampSec : 1681129332 oplogStartTimestamp : Mon Apr 10 00:17:05 2023 oplogStartTimestampSec : 1679617025 port : 8191 replicaSet : F50190FF-36F7-486B-B34F-FDE64B4665E9 replicationStatus : KV store captain standalone : 1 status : ready storageEngine : mmapv1 KV store members: 127.0.0.1:8191 configVersion : 1 electionDate : Mon Apr 10 12:21:49 2023 electionDateSec : 1681129309 hostAndPort : 127.0.0.1:8191 optimeDate : Mon Apr 10 12:22:12 2023 optimeDateSec : 1681129332 replicationStatus : KV store captain uptime : 24
[splunk@splunkserver kvstore_upgrade]$ /opt/splunk/bin/splunk show kvstore-status This member: backupRestoreStatus : Ready date : Mon Apr 10 12:22:12 2023 dateSec : 1681129332.136 disabled : 0 guid : F50190FF-36F7-486B-B34F-FDE64B4665E9 oplogEndTimestamp : Mon Apr 10 12:22:12 2023 oplogEndTimestampSec : 1681129332 oplogStartTimestamp : Mon Apr 10 00:17:05 2023 oplogStartTimestampSec : 1679617025 port : 8191 replicaSet : F50190FF-36F7-486B-B34F-FDE64B4665E9 replicationStatus : KV store captain standalone : 1 status : ready storageEngine : mmapv1 KV store members: 127.0.0.1:8191 configVersion : 1 electionDate : Mon Apr 10 12:21:49 2023 electionDateSec : 1681129309 hostAndPort : 127.0.0.1:8191 optimeDate : Mon Apr 10 12:22:12 2023 optimeDateSec : 1681129332 replicationStatus : KV store captain uptime : 24
At this point, we needed to do a storage migration process to get the engine upgraded to WiredTiger on serverVersion 3.6.17:
[splunk@splunkserver kvstore_upgrade]$ /opt/splunk/bin/splunk migrate kvstore-storage-engine --target-engine wiredTiger --enable-compression Starting KV Store storage engine upgrade: Phase 1 (dump) of 2: ........................................................................................................................................................................................................ Phase 2 (restore) of 2: .........................................................................................................................................................................................................................................................................................................................................................................................Data sanity check completed successfully for all configured KV Store collections Successfully migrated to storage engine wiredTiger
[splunk@splunkserver kvstore_upgrade]$ /opt/splunk/bin/splunk migrate kvstore-storage-engine --target-engine wiredTiger --enable-compression Starting KV Store storage engine upgrade: Phase 1 (dump) of 2: ........................................................................................................................................................................................................ Phase 2 (restore) of 2: .........................................................................................................................................................................................................................................................................................................................................................................................Data sanity check completed successfully for all configured KV Store collections Successfully migrated to storage engine wiredTiger
[splunk@splunkserver kvstore_upgrade]$ /opt/splunk/bin/splunk migrate kvstore-storage-engine --target-engine wiredTiger --enable-compression Starting KV Store storage engine upgrade: Phase 1 (dump) of 2: ........................................................................................................................................................................................................ Phase 2 (restore) of 2: .........................................................................................................................................................................................................................................................................................................................................................................................Data sanity check completed successfully for all configured KV Store collections Successfully migrated to storage engine wiredTiger
After this conversion, our kvstore-status showed that we were running on WiredTiger on server version 3.6:
[splunk@splunkserver kvstore_upgrade]$ /opt/splunk/bin/splunk show kvstore-status --verbose WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details. This member: backupRestoreStatus : Ready date : Mon Apr 10 12:30:54 2023 dateSec : 1681129854.993 disabled : 0 featureCompatibilityVersion : 3.6 guid : F50190FF-36F7-486B-B34F-FDE64B4665E9 oplogEndTimestamp : Mon Apr 10 12:30:54 2023 oplogEndTimestampSec : 1681129854 oplogStartTimestamp : Mon Apr 10 12:29:07 2023 oplogStartTimestampSec : 1681129747 port : 8191 replicaSet : F50190FF-36F7-486B-B34F-FDE64B4665E9 replicationStatus : KV store captain standalone : 1 status : ready storageEngine : wiredTiger KV store members: 127.0.0.1:8191 configVersion : 1 electionDate : Mon Apr 10 12:29:07 2023 electionDateSec : 1681129747 hostAndPort : 127.0.0.1:8191 optimeDate : Mon Apr 10 12:30:54 2023 optimeDateSec : 1681129854 replicationStatus : KV store captain serverVersion : 3.6.17 uptime : 110
[splunk@splunkserver kvstore_upgrade]$ /opt/splunk/bin/splunk show kvstore-status --verbose WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details. This member: backupRestoreStatus : Ready date : Mon Apr 10 12:30:54 2023 dateSec : 1681129854.993 disabled : 0 featureCompatibilityVersion : 3.6 guid : F50190FF-36F7-486B-B34F-FDE64B4665E9 oplogEndTimestamp : Mon Apr 10 12:30:54 2023 oplogEndTimestampSec : 1681129854 oplogStartTimestamp : Mon Apr 10 12:29:07 2023 oplogStartTimestampSec : 1681129747 port : 8191 replicaSet : F50190FF-36F7-486B-B34F-FDE64B4665E9 replicationStatus : KV store captain standalone : 1 status : ready storageEngine : wiredTiger KV store members: 127.0.0.1:8191 configVersion : 1 electionDate : Mon Apr 10 12:29:07 2023 electionDateSec : 1681129747 hostAndPort : 127.0.0.1:8191 optimeDate : Mon Apr 10 12:30:54 2023 optimeDateSec : 1681129854 replicationStatus : KV store captain serverVersion : 3.6.17 uptime : 110
[splunk@splunkserver kvstore_upgrade]$ /opt/splunk/bin/splunk show kvstore-status --verbose WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details. This member: backupRestoreStatus : Ready date : Mon Apr 10 12:30:54 2023 dateSec : 1681129854.993 disabled : 0 featureCompatibilityVersion : 3.6 guid : F50190FF-36F7-486B-B34F-FDE64B4665E9 oplogEndTimestamp : Mon Apr 10 12:30:54 2023 oplogEndTimestampSec : 1681129854 oplogStartTimestamp : Mon Apr 10 12:29:07 2023 oplogStartTimestampSec : 1681129747 port : 8191 replicaSet : F50190FF-36F7-486B-B34F-FDE64B4665E9 replicationStatus : KV store captain standalone : 1 status : ready storageEngine : wiredTiger KV store members: 127.0.0.1:8191 configVersion : 1 electionDate : Mon Apr 10 12:29:07 2023 electionDateSec : 1681129747 hostAndPort : 127.0.0.1:8191 optimeDate : Mon Apr 10 12:30:54 2023 optimeDateSec : 1681129854 replicationStatus : KV store captain serverVersion : 3.6.17 uptime : 110
Next, we performed another KV Store migration to get the server version up to 4.2.17:
[splunk@splunkserver kvstore_upgrade]$ /opt/splunk/bin/splunk migrate migrate-kvstore [App Key Value Store migration] Starting migrate-kvstore. Started standalone KVStore update, start_time="2023-04-10 12:32:47". [App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600seconds. [App Key Value Store migration] Migration is not required. Created version file path=/opt/splunk/var/run/splunk/kvstore_upgrade/versionFile40
[splunk@splunkserver kvstore_upgrade]$ /opt/splunk/bin/splunk migrate migrate-kvstore [App Key Value Store migration] Starting migrate-kvstore. Started standalone KVStore update, start_time="2023-04-10 12:32:47". [App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600seconds. [App Key Value Store migration] Migration is not required. Created version file path=/opt/splunk/var/run/splunk/kvstore_upgrade/versionFile40
[splunk@splunkserver kvstore_upgrade]$ /opt/splunk/bin/splunk migrate migrate-kvstore [App Key Value Store migration] Starting migrate-kvstore. Started standalone KVStore update, start_time="2023-04-10 12:32:47". [App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600seconds. [App Key Value Store migration] Migration is not required. Created version file path=/opt/splunk/var/run/splunk/kvstore_upgrade/versionFile40
At this point, the server version was showing 4.2:
[splunk@splunkserver kvstore_upgrade]$ /opt/splunk/bin/splunk show kvstore-status --verbose WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details. This member: backupRestoreStatus : Ready date : Mon Apr 10 12:38:00 2023 dateSec : 1681130280.039 disabled : 0 featureCompatibilityVersion : 4.2 guid : F50190FF-36F7-486B-B34F-FDE64B4665E9 oplogEndTimestamp : Mon Apr 10 12:37:59 2023 oplogEndTimestampSec : 1681130279 oplogStartTimestamp : Mon Apr 10 12:29:07 2023 oplogStartTimestampSec : 1681129747 port : 8191 replicaSet : F50190FF-36F7-486B-B34F-FDE64B4665E9 replicationStatus : KV store captain standalone : 1 status : ready storageEngine : wiredTiger KV store members: 127.0.0.1:8191 configVersion : 1 electionDate : Mon Apr 10 12:36:29 2023 electionDateSec : 1681130189 hostAndPort : 127.0.0.1:8191 optimeDate : Mon Apr 10 12:37:59 2023 optimeDateSec : 1681130279 replicationStatus : KV store captain serverVersion : 4.2.17
[splunk@splunkserver kvstore_upgrade]$ /opt/splunk/bin/splunk show kvstore-status --verbose WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details. This member: backupRestoreStatus : Ready date : Mon Apr 10 12:38:00 2023 dateSec : 1681130280.039 disabled : 0 featureCompatibilityVersion : 4.2 guid : F50190FF-36F7-486B-B34F-FDE64B4665E9 oplogEndTimestamp : Mon Apr 10 12:37:59 2023 oplogEndTimestampSec : 1681130279 oplogStartTimestamp : Mon Apr 10 12:29:07 2023 oplogStartTimestampSec : 1681129747 port : 8191 replicaSet : F50190FF-36F7-486B-B34F-FDE64B4665E9 replicationStatus : KV store captain standalone : 1 status : ready storageEngine : wiredTiger KV store members: 127.0.0.1:8191 configVersion : 1 electionDate : Mon Apr 10 12:36:29 2023 electionDateSec : 1681130189 hostAndPort : 127.0.0.1:8191 optimeDate : Mon Apr 10 12:37:59 2023 optimeDateSec : 1681130279 replicationStatus : KV store captain serverVersion : 4.2.17
[splunk@splunkserver kvstore_upgrade]$ /opt/splunk/bin/splunk show kvstore-status --verbose WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details. This member: backupRestoreStatus : Ready date : Mon Apr 10 12:38:00 2023 dateSec : 1681130280.039 disabled : 0 featureCompatibilityVersion : 4.2 guid : F50190FF-36F7-486B-B34F-FDE64B4665E9 oplogEndTimestamp : Mon Apr 10 12:37:59 2023 oplogEndTimestampSec : 1681130279 oplogStartTimestamp : Mon Apr 10 12:29:07 2023 oplogStartTimestampSec : 1681129747 port : 8191 replicaSet : F50190FF-36F7-486B-B34F-FDE64B4665E9 replicationStatus : KV store captain standalone : 1 status : ready storageEngine : wiredTiger KV store members: 127.0.0.1:8191 configVersion : 1 electionDate : Mon Apr 10 12:36:29 2023 electionDateSec : 1681130189 hostAndPort : 127.0.0.1:8191 optimeDate : Mon Apr 10 12:37:59 2023 optimeDateSec : 1681130279 replicationStatus : KV store captain serverVersion : 4.2.17
Now KV Store is running correctly and on the current version. We fixed the problem!
Conclusion
Do I expect that you’ll ever be in a situation where you will find this information useful? I hope not. Did I write this so that I can have some notes in case I ever run into a similar problem in the future? Absolutely.
This is a great example of running into a problem where you have to make some educated guesses on a possible solution with limited information to go on. I’m glad we were able to figure this one out and hope these notes might help you if you ever see this problem in your Splunk environment. If not, hello to my future self who is reading this months or years from now and again fighting with a broken KV Store somewhere.
