- Ian Gillespie
- Sep 07, 2016
- Tested on Splunk Version: N/A
In this tutorial, we’re going to cover how to create a KV Store both through the Splunk user interface, as well as by modifying collections.conf and transforms.conf. KV Stores in Splunk are nothing more than Mongo Databases, so they allow us to easily apply CRUD (Create / Read / Update / Delete) to our data.
KV Stores, unlike regular Splunk Lookup files, contain a unique ID field called _key. It’s a much more stable way to update your data than doing so through a regular Lookup file. KV Stores are also much more performant than lookup files. Lookup files must be replicated across indexers, which can cause performance problems if they become too large. KV Stores on the other hand, do not need to be replicated across indexers. This is also built on MongoDB, so it may make more sense to those experienced with working with databases versus dealing with a lookup file, which is more of a flat file that doesn't contain any sort of unique attribute in it (_key).
Prerequisites: You should know how to navigate through Splunk’s UI (user interface). You should also have a basic understanding of Splunk's query language, as well as how lookups work.
Note: We have both blog and screencast pieces of this tutorial available. So, if you would prefer a more audiovisual option feel free to view the screencasts.
We’re going to cover how to create a KV Store both through the UI, as well as by modifying collections.conf and transforms.conf. KV Stores in splunk are nothing more than Mongo Databases, so they allow us to easily apply CRUD (Create / Read / Update / Delete) to our data.
We will also cover editing a KV Store directly using the Splunk Search Language.
I highly recommend using the Splunk Lookup Editor to create and edit your lookup files and KV Store collections. I will first show you how to create the KV Store collection using the old-fashioned way of creating a collections.conf file. I will then show you how to create it using the Splunk Lookup Editor, as well as through curl on the command line.
KV Store Creation Using collections.conf
Go into the Search app’s local directory and create a collections.conf file. Inside of collections.conf add the following:
[task_collection] field.Task_Name = string field.Task_Description = string field.Estimated_Completion_Date = string field.Notes = string field.Status = string
Save the file.
Lookup Definition Creation Using transforms.conf
Navigate to the Search app’s local folder and create a transforms.conf file, if one doesn’t already exist.
In the transforms.conf file add the following:
[task_lookup] external_type = kvstore collection = task_collection fields_list = _key,Task_Name, Task_Description,Estimated_Completion_Date,Notes,Status
Now, there are a couple of ways we can try to refresh transforms.conf. The documentation says we can try running | extract reload=t in the search window. As an admin we can also try running the following in the browser:
If all else fails, restart Splunk.
KV Store Collection Creation Through the UI
If you’ve installed the Splunk Lookup Editor, navigate to that app and create a new KV Store:
Name the KV Store spp_collection and put it in the Search app. Add the necessary fields:
Click ‘Save Lookup’.
While we could make Estimated_Completion_Date a time field, for the sake of simplicity we will leave it as a string.
Once we save it we can easily add new values to our KV store using the Lookup Editor interface:
Above, you will notice in the right most dropdown it says, "Showing entries for: nobody". We can view the data in our KV Store, "for: nobody" meaning it's globally accessible or for our particular user, which we cannot access through the Splunk Query Language.
Lookup Definition Creation Through the UI
Go to Settings < Lookup then click, and then next to Lookup Definitions click ‘Add New’.
Under Supported Fields, we will add all the fields from the KV Store. We want to also make sure to add the _key field, which will act as a unique identifier for our fields.
Select the destination app. In our case, it is our Search app.
We don’t need to worry about advanced options. Click on ‘Save’.
If you're looking for something different than the typical "one-size-fits-all" security mentality, you've come to the right place.