Learn How To Build a Real-Time Filtering Table in Splunk: Part 1

It’s pretty simple to create a table in Splunk. By default, Splunk needs to refetch the data in order to filter it down. However, what if you had a set of data and you wanted to easily filter that table in real-time? This tutorial will take you through step-by-step on how to do this. The first portion will cover the basics of setting up an app through the Splunk Web Framework, which will result in the creation of a custom input field and table.



It’s pretty simple to create a table in Splunk. By default, Splunk needs to refetch the data in order to filter it down. However, what if you had a set of data and you wanted to easily filter that table in real-time?

Let's say you have a predefined list of subnets in a lookup. You shouldn't have to refetch the data to find a match, if you're searching for something specific, especially since the data isn't changing frequently enough. In this case, having something that filters in real-time would be much more effective.

I am going to take you through step-by-step how to do just that. Due to the amount of content we will be covering, this tutorial will be split into two separate posts. The first portion will cover the basics of setting up an app through the Splunk Web Framework, which will result in the creation of a custom input field and table. The second will cover how to add the filtering functionality to what we have built in the first.

Oh, and if you enjoy a more visual route, there are related screencasts split across three videos.

Already familiar with the Splunk Web Framework? You will probably be alright skimming through this first part.

Caution: There's some heavy coding ahead, specifically in regards to JavaScript. I will do my best to guide you through each step.

Part 1: The Necessities

Download the zipped db_exploits.csv file. This contains a list of database exploits (http://www.exploit-db.com/) and we will use this data to populate our lookup. Once you have this downloaded, go into Splunk and create a new lookup table from this .csv file. We’ll be referencing it in our search as| inputlookup db_exploits.csv

Feel free to also download working examples of the app:

By the end of Part 1, you should have something like this: Mynewapp Part1 By the end of Part 2, you should have something like this: Mynewapp Part2

Part 2: Create Your App

Since we are using the built-in Splunk Web Framework, we are going to create our app from the command line at $SPLUNK_HOME/etc/apps/framework and run: ./splunkdj createapp <appname> #name whatever you like

It will then ask for your username and password and then prompt you with: The <appname> app was created at '$SPLUNK_HOME/etc/apps/<appname>'. Please restart Splunk.

Once you restart, go to http://localhost:8000/dj/en-us/<appname>/home/ and you should see something like this:

Default App View
Default App view

Note: If you don’t restart Splunk you will get a 404 error.

In the console go back to$SPLUNK_HOME/etc/apps/<appname>/ and you will see a directory structure like this:

Appname Directory
App name directory

Everything we will be doing will be happening inside the Django directory. Go to django/<appname>/ and you will see a directory structure like this:

Django Directory
Django directory

The three important directories we will be dealing with are:

  • static - this is where we keep our .css and .js files
  • templates - where Django templates are kept
  • templatetags - here we will define two new Django template tags filtertable and filterinput to be used in our Django template

First, let’s take a look at the default Django template inside of the templates directory called home.html:

{% extends "splunkdj:base_with_account_bar.html" %}
{% load splunkmvc %}
{% block title %}{{app_name}} Home Page{% endblock title %}

{% block css %}
{% endblock css %}

{% block content %}

Template message: {{message}}

    You should also look in the JavaScript console...

{% endblock content%}
{% block js %}

{% endblock js %}

Here's an outline of what each section in this template is for:

  • {% extends "splunkdj:base_with_account_bar.html" %} loads the the top Splunk bar.
  • {% load splunkmvc %} loads the default Splunk mvc javascript file
  • {% block title %}{{app_name}} Home Page{% endblock title %} provides the title of the page, which would end up being ‘mynewapp Home Page’ after being rendered
  • {% block css %} is where we put either inline CSS or link to external CSS files
  • {% block content %} is where the content goes - our custom template tags that we will create shortly will go here
  • {% block js %} is where we will keep our JavaScript template

Part 3: Add the Search Manager

Before continuing, we're going to add in a new block called {% block managers %}. This is where we will be keeping our search manager, which will call our Database Exploit lookup and populate our table.

Right after the {% endblock content %} add:

{% block managers %}
{% searchmanager id="dbe"
search="| inputlookup db_exploits.csv | rename date as Date, description as Description, file as File, platform as Platform, port as Port, type as Type | table Date, Description, File, Platform, Port, Type | sort -Date | head 500" preview=True cache=True %}
{% endblock managers %}

We provide an id of 'dbe' in order to reference this search later, when we add our table template tag. This is so it knows where to pull it’s data from. We are using a lookup called db_exploits.csv to populate our search and the search itself is pretty straightforward. I’m also limiting this to 500 results, because if we try to filter a ridiculous amount it could return 10,000 results and be way too performance heavy on the browser.

Part 4: Add the JavaScript template

Let's first create our input field JavaScript template. This will be a simple input field, so there's no need to use one of Splunk’s built in form elements. Also, because I want the input field to pass it’s value to the table, I will be using a Backbone View that will utilize this template. If you’ve never used Backbone before, don’t worry, it should all make sense once you see how it fits together. Just know that Splunk Web Framework's JavaScript components use Backbone as their core, so it makes sense for us to do the same. For now, we will need a template to reference, which will be added into our{% block js %}. This template will be referenced inside our filterinput.js file, and will allow us to attach the functionality we define there to the JavaScript template we define in our home.html file.

Go ahead and remove the default tags inside the {% block js %} inside the Django template located at appname>/django/<appname>/templates/ and add the following and save the file:

<script type="text/template" id="filterFieldTemplate">
    <input type="text"
               class="form-control"
               id="filterField"
               placeholder="Filter table" />
</script>

Part 5: Create Custom Template Tags

At this point, we need to create custom template tags for our table and input. Go to <appname>/django/<appname>/templatetags/ and create two new files called filterinput.py and filtertable.py.

Go into filtertable.py and add the following:

from django import template
from splunkdj.templatetags.tagutils import component_context
register = template.Library()
@register.inclusion_tag('splunkdj:components/component.html', takes_context=True)
def filtertable(context, id, *args, **kwargs): # The template tag
    return component_context(
        context,
        "filtertable", # The custom view's CSS class name
        id,
        "view",
        "<appname>/filtertable", # Path to the JavaScript class/file for the view
        kwargs
    )

Here we are creating our filtertable tag. The name itself is derived from the method name. If we called this method foobizbaz then in our .html template we would define it as

{% foobizbaz %}. The path to the javascript file links to the location of <appname>/django/<appname>/static, which can be confusing since it's just <appname>/filtertable. Keep in mind that directory is where the file actually exists.

Add the following to the filterinput.py file:

from django import template
from splunkdj.templatetags.tagutils import component_context
register = template.Library()
@register.inclusion_tag('splunkdj:components/component.html', takes_context=True)
def filterinput(context, id, *args, **kwargs): # The template tag
    return component_context(
        context,
        "filterinput", # The custom view's CSS class name
        id,
        "view",
        "mynewapp/filterinput", # Path to the JavaScript class/file for the view
        kwargs
    )

Part 6: Add your custom template tags into the template

Go back to the home.html file located in <appname>/django/<appname>/templates/and first remove the following from the{% block content %} :

   <div>
       <div>
           <p>Template message: {{message}}</p>
           <p>You should also look in the JavaScript console...</p>
       </div>
   </div>

then add inside the {% block content %}:

<div id="dashboard">
   <div>
       <h2>Database Exploit Filter Table</h2>
   </div>

   <!-- filterinput custom template tag -->
   {% filterinput id="filterinput" %}

   <!-- filtertable custom template tag -->
   {% filtertable id="filtertable" managerid="dbe" %}    
</div>

The managerid in the filtertable tag references the search manager we added earlier, so it knows where to pull in the data from. At this point, it’s not going to know how to render these tags and, if we were to visit our page at http://localhost:8000/dj/en-us/mynewapp/home/, we would get a Django template error. This is because Django has no idea how to render these tags and this is where the JavaScript will come into play. When we use one of our custom tags, JavaScript will handle how that data should be rendered.

Part 7: JavaScript all the things

Go to <appname>/django/<appname>/static/<appname>/ and create two new .js files called filterinput.js and filtertable.js.

The basic structure of filtertable.js is as follows:

define(function(require, exports, module) {
    var _ = require('underscore');
    var mvc = require('splunkjs/mvc');
    var SimpleSplunkView = require('splunkjs/mvc/simplesplunkview');
    var FilterTable = SimpleSplunkView.extend({
    //class to provide this $el
        /*
        * the $el is this specific dom element, which in this case will be a table
        */
        className: "filtertable",
        options: {
            data: "results"
        },
        formatData: function(data) {
            //this is where formatting occurs - i.e. a table
            return data;
        },
       //creates the view
       createView: function() {
           return this;
       },
       // Override this method to put the Splunk data into the view
       updateView: function(viz, data) {
           //returns back the first row of data
           var myResults = data[0];
           //appends data to dom
           this.$el.html(myResults);
       }
    });

    return FilterTable;

});

Looking through filtertable.js, we are first using the define() method to define a new module. Then, we load in the necessary files including Underscoresplunkjs mvc and the SimpleSplunkView.

The filter table extends the SimpleSplunkView inheriting all of its properties and providing us an easy way to handle the data that Splunk gives us from our search.

The options() method tells Splunk to return "results," instead of the other option of a "preview." The formatData() method is where we will eventually be formatting the data into a table format. createView() creates the view, we won't be doing anything else with this method. updateView() updates when the table needs to be re-rendered by Splunk. Finally, we return FilterTable at the end.

As for filterinput.js, we are going to add the following:

define(function(require, exports, module) {
    var _ = require('underscore');
    var Backbone = require("backbone");
    var FilterTable = require('./filtertable');
    var SimpleSplunkView = require('splunkjs/mvc/simplesplunkview');
    /* Create a simple backbone view that will be used for our filter field */
    var FilterInput = Backbone.View.extend({
        el: '#filterinput',
        initialize: function() {
            //define the template we want to use for this --
            //this is defined in the 'block js' of home.html
            this.template = _.template($('#filterFieldTemplate').html());
            this.render();
        },
        //render the input
        render: function() {
            this.$el.html(this.template());
            return this;
        }
    });

    return FilterInput;

});

There are some differences between filterinput.js file and filtertable.js file. As we won't need to be handling data sent by Splunk, it would be unnecessary to extend from SimpleSplunkView. Instead, I am using a Backbone View, which is what SimpleSplunkView also extends from. However, we don't need all the added Splunk specific functionality. Due to the fact we are using a Backbone View we need to load in Backbone directly up top. We are also adding a reference to our filtertable.js, since we will be connecting the two.

First, el: '#filterinput' defined a pre-exisitng element in our html to attach this element to. If you remember #filterinput is defined in our template in the content block as {% filterinput id="filterinput" %}. In theinitialize() method I am defining the template for this input, which is the JavaScript template that was added to our home.html file earlier. It then calls the render() method and attaches the template to the DOM with .

Now, before we can view this in the browser, a reference needs to be added to the JavaScript files in our Django template.

Part 8: Add a reference to your JavaScript files

Go back to your home.html template in >span class="theme:twilight font:consolas font-size:16 line-height:25 lang:default highlight:0 decode:true crayon-inline"><appname>/django/<appname>/templates/ and right below {% load splunkmvc %} in home.html you will want to add the following:

<!-- load in filtertable.js file -->
{% load filtertable %}

<!-- load in filterinput.js file -->
{% load filterinput %}

If you go to view the page in the browser, you should see one line of data output below the input field:

Filter Table One Line Output 1024X156
Filter table

This is because in the updateView() method inside of filtertable.js we have var myResults = data[0] effectively pulling out the first line. What we need to do next is loop through our data, output all of the rows, and format them into an actual table.

Add the following to the formatData() method in filtertable.js method (be sure to remove the original return data):

var myDataString = "";
//format each row -- this uses the underscore method _.each to loop through the results
_.each(data, function(row, index) {
&nbsp;&nbsp;&nbsp;myDataString = myDataString + '<tr class=“body”><td>' + row[0] + '</td><td>' + row[1] + ' ' + &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp;row[2] &nbsp;+ '</td><td>' + row[3] + '</td><td>' + row[4] &nbsp;+ '</td></tr>';
});

//wrap the string in a <table> tag and give it some headers
myDataString = "<table class='table table-striped' id='dbetable'><thead><tr><th>Updated</th><th>Description</th><th>Category</th><th>Port
</th></tr></thead><tbody>" + myDataString + "</tbody></table";

return myDataString;

Above, we first define a new empty string myDataString, followed by the Underscore method _.each() to loop through each row of data and wrap table rows and columns around them. At the end, we take the rows and wrap a <table /> tag around them so it formats it nicely.

Also, in the updateView() method, replace what is currently there with since the data has already been formatted in the >span class="theme:twilight font:consolas font-size:16 line-height:25 lang:default decode:true crayon-inline ">formatData() method, there is nothing else we need to do here.

In the end, your filtertable.js file should look like this:

define(function(require, exports, module) {
    var _ = require('underscore');
    var mvc = require('splunkjs/mvc');
    var SimpleSplunkView = require('splunkjs/mvc/simplesplunkview');

    var FilterTable = SimpleSplunkView.extend({
        //class to provide this $el
        /*
        * the $el is this specific dom element, which in this case will be a table
        */
        className: "filtertable",
        options: {
            data: "results"
        },
        formatData: function(data) {
            //this is where formatting occurs - i.e. a table
            var myDataString = "";
            //format each row
            _.each(data, function(row, index) {
                myDataString = myDataString + '<tr><td>' + row[0] + 
                '</td><td>' + row[1] + ' ' + row[2] + '</td><td>' + row[3] + 
                '</td><td>' + row[4] + '</td></tr>';
            });

            //wrap the string
            myDataString = "<table class='table table-striped' id='dbetable'><thead><tr>" +
            "<th>Updated</th><th>Description</th><th>Category</th><th>Port</th></tr>" + 
            "</thead><tbody>" + myDataString + "</tbody></table";
            return myDataString;
        },
        //creates the view
        createView: function() {
            return this;
        },
        // Override this method to put the Splunk data into the view
        updateView: function(viz, data) {
            this.$el.html(data);
        }
    });

    return FilterTable;

});

Now, save the file. If you go back to the browser and view the page (http://localhost:8000/dj/en-us/<appname>/home/), you should see the input field and the table. At this point, if you type anything into the input field, it won’t filter the table. This will be handled in the next part as we connect the input field to the table so it does its job and filters as we type.




Close off Canvas Menu