Splunk Tutorial: CRUDing a KV Store in Splunk Using Python

In a previous blog series, I covered how to create and CRUD a KV Store using Splunk’s SPL (Search Processing Language). In this blog post I am going to cover how to do the same thing using Python.



Introduction

In a previous blog series, I covered how to create and CRUD a KV Store using Splunk’s SPL (Search Processing Language). Feel free to check out the various pieces and parts of that tutorial here: Creating and CRUDing a KV Store in Splunk: Part 1 and Part 2.

In this blog post I'm going to cover how to do the same thing using Python. The blog post is available, as well as the subsequent screencasts. So, feel free to read, watch or do both.

In order to follow along with this you should download the Splunk Python SDK.

Export the PYTHONPATH

You can place the Splunk SDK folder where you want, but you will need to add the folder to your Python path in order to run the examples:

export PYTHONPATH=~/splunk-sdk-python

Using .splunkrc

In this example, I’m going to use a .splunkrc file to store my credentials. The .splunkrc file is a handy way for us to store our credentials when we connect to Splunk through our Python script.

You don’t have to use a .splunkrc file, but its easier than having to write this every time we want to execute a file, see below:

<python_script>.py --username=”<username>” --password=”<password>”

Example .splunkrc

# Splunk host (default: localhost)
host=localhost
# Splunk admin port (default: 8089)
port=8089
# Splunk username
username=admin
# Splunk password
password=changeme
# Access scheme (default: https)
scheme=https
# Your version of Splunk (default: 5.0)
version=6.4.2 #only needed for the JavaScript SDK

Where to put .splunkrc

The location of the .splunkrc file will depend on whether or not you’re following along using Windows.

On Windows, you will put the .splunkrc in C:\Users\currenusername\.splunkrc

If you are on Linux or OSX place it in ~/.splunkrc

How does this work?

When we run our Python file, Splunk is going to check and see if user credentials have been passed into the command line. If not, it will then check if a .splunkrc file exists.

Inside of your Splunk SDK folder, there is an examples folder. This is where we will add our task_collection.py and add the following:

import sys, json
from splunklib.client import connect

Importing connect will allow us to actually connect to Splunk. We will pass our credentials to it shortly. Next, we will try to import parse, which will be used to pull out our credentials from command line arguments or the .splunkrc file. If it cannot be imported that means you did not successfully export the SDK to your PYTHONPATH, as described above.

try:
    from utils import parse
except ImportError:
    raise Exception("Add the SDK repository to your PYTHONPATH to run the examples "
"(e.g., export PYTHONPATH=~/splunk-sdk-python.")

We then define the main() function where we will set up an opts variable from which we will pull out our user credentials. We will also set the owner to nobody and the app context to search. We will then connect to Splunk using our opts.kwargs.

def main():
    #in this example, i’m using a .splunkrc file to pull credentials
    opts = parse(sys.argv[1:], {}, ".splunkrc")
    opts.kwargs["owner"] = "nobody"
    opts.kwargs["app"] = "search"
    service = connect(**opts.kwargs)

Next, in the main() function, we will set up our collection name and set the collection using service.kvstore.

    collection_name = "task_collection"
    collection = service.kvstore[collection_name]

We will also check if the collection exists and if it does not, then we will create it:

    #if the collection is found, print it out
    #if not, then create the collection
    if collection_name in service.kvstore:
        print "Collection %s found!" % collection_name
    else:
        service.kvstore.create(collection_name)

You can then read the data from KV Store collection, using the query() function:

    #print out the data from the collection
    print "Collection data: %s" % json.dumps(collection.data.query(), indent=1)

Finally, you will want to run the following, in order to execute the main() function, when you run the Python script from the command line:

if __name__ == "__main__":
    main()

Once you’ve added everything, save the file.

Create

To create, or insert, new data in your collection add the following to the Python script inside of the main() function above where we are printing out the collection data using the query() function:

collection.data.insert(json.dumps({"Task_Name":"Python 
Task","Task_Description":"This task was created in python.","Status":"In 
Progress","Estimated_Completion_Date":"October 20th",
"Notes":"No notes at this time."}))

All we are doing is inserting JSON to add a new value to KV Store.

Save the script and then run it on the command line. You should get back something like this:

Collection task_collection found!
Collection data: [
 {
  "Status": "In Progress",
  "_key": "57a6a49067174ac5cf6ec013",
  "Notes": "No notes at this time.",
  "Task_Description": "This task was created in python.",
  "Task_Name": "Python Task",
  "Estimated_Completion_Date": "October 20th",
  "_user": "nobody"
 }
]

Read

As described above, we can use query() to read the data from our KV Store. In our example we are specifically running this (see below) to print out the results:

print "Collection data: %s" % json.dumps(collection.data.query(), indent=1)

Update

Currently, we don’t have an update in our file. Go ahead and comment out the collection.data.insert that currently exists and then add:

collection.data.update(str(“<_key_id>”), json.dumps({"Task_Name":"Python 
Task 2.0","Task_Description":"This task was updated in 
Python.","Status":"Delayed","Estimated_Completion_Date":"November 
2nd","Notes":"This project has been somewhat delayed...whoops."}))

The key to updating is to first provide a string version of the _key you want to update and then as the second parameter include all the fields you want to update from your KV Store.

Save your file and rerun it, and you should see the updated values in the output.

Delete

We can either delete a specific row based on the _key:

collection.data.delete(json.dumps({"_key":"<_key_id>"}))

Or we can delete the entire collection:

collection.data.delete()

So, what can we do with this information? We could CRUD our KV Store from outside of Splunk using the Python SDK, or we could create a custom REST endpoint, modify our data and the CRUD the collection in some way.

In Closing

With all of this information, you should have a pretty solid understand of how to create a new KV store collection, as well as a lookup definition that allows us to communicate with our collection through the Splunk query language. We also covered how to CRUD our KV store collection through the Splunk query language as well as doing the same through Python. If you have any questions, feel free to leave them below in the comments section.




Close off Canvas Menu