Seasonality, which states predictable variations in data will occur over specific time periods, is one the most important concepts in statistical analysis of time series data in Splunk. For example, it's expected that you'd see more data logged during business hours, and less during off-hour times. These variations can throw a wrench into typical anomaly detection techniques–as outlined in part 1–if not taken into account.
This article will offer an explanation of seasonality as well as techniques for taking it into account in your searches; we will also provide you with a practical example of how to account for this type of behavior in your anomaly detection searches.
Real world example
To help explain seasonality, we’ll work through a real world example in detecting unexpected dips in indexed data. Many sources of machine data generate more logs during normal business hours (when they’re being actively used), so this is a situation where taking seasonality into account is appropriate.
Before we start, here is the full example that we’ll break down:
index=_internal source=*license_usage.log* type="Usage"
| bin _time span=15m
| stats sum(b) AS byte_sum by idx, _time
| xyseries utctime, idx, byte_sum
| makecontinuous _time span=15m
| fillnull value=0
| untable _time idx byte_sum
| eval date_wday=strftime(_time,"%w")
| eval date_hour=strftime(_time,"%H")
| eval weekday_weekend=if(((date_wday > 0) AND (date_wday < 6)), "weekday", "weekend")
| eval biz_hours=if((((date_hour < 8) OR (date_hour > 18)) OR (weekend_weekday=="weekend")), "no", "yes")
| stats avg(byte_sum) as average, stdev(byte_sum) as std by idx, weekday_weekend, biz_hours
| outputlookup avg_index_bytes_15m.csv
index=_internal source=*license_usage.log* type="Usage"
| bin _time span=15m
| stats sum(b) AS byte_sum by idx, _time
| xyseries utctime, idx, byte_sum
| makecontinuous _time span=15m
| fillnull value=0
| untable _time idx byte_sum
| eval date_wday=strftime(_time,"%w")
| eval date_hour=strftime(_time,"%H")
| eval weekday_weekend=if(((date_wday > 0) AND (date_wday < 6)), "weekday", "weekend")
| eval biz_hours=if((((date_hour < 8) OR (date_hour > 18)) OR (weekend_weekday=="weekend")), "no", "yes")
| stats avg(byte_sum) as average, stdev(byte_sum) as std by idx, weekday_weekend, biz_hours
| outputlookup avg_index_bytes_15m.csv
index=_internal source=*license_usage.log* type="Usage"
| bin _time span=15m
| stats sum(b) AS byte_sum by idx, _time
| xyseries utctime, idx, byte_sum
| makecontinuous _time span=15m
| fillnull value=0
| untable _time idx byte_sum
| eval date_wday=strftime(_time,"%w")
| eval date_hour=strftime(_time,"%H")
| eval weekday_weekend=if(((date_wday > 0) AND (date_wday < 6)), "weekday", "weekend")
| eval biz_hours=if((((date_hour < 8) OR (date_hour > 18)) OR (weekend_weekday=="weekend")), "no", "yes")
| stats avg(byte_sum) as average, stdev(byte_sum) as std by idx, weekday_weekend, biz_hours
| outputlookup avg_index_bytes_15m.csv
Once you get the stats generated in the lookup–and have a search that populates it every so often–you can implement the following:
index=_internal host=*.splunkcloud.com source=*license_usage.log* type="Usage"
| bin _time span=15m
| eval date_wday=strftime(_time,"%w")
| eval date_hour=strftime(_time,"%H")
| eval weekday_weekend=if(((date_wday > 0) AND (date_wday < 6)), "weekday", "weekend")
| eval biz_hours=if((((date_hour < 8) OR (date_hour > 18)) OR (weekend_weekday=="weekend")), "no", "yes")
| stats sum(b) AS byte_sum by idx, weekday_weekend, biz_hours, _time
| join type=inner idx,weekday_weekend, biz_hours [| inputlookup avg_index_bytes_15m.csv]
| eval devs=(byte_sum - average)/std
| eval currentGB=round((byte_sum/1024/1024/1024), 3)
| eval averageGB=round((average/1024/1024/1024), 3)
| eval stdGB=round((std/1024/1024/1024), 3)
| table _time, idx, currentGB, averageGB, devs, stdGB
| where devs < -3 AND averageGB > 0.75
| sort 0 - devs
| rename idx AS Index, currentGB AS "GB Indexed over Past 15 Minutes", averageGB AS "Average GB Indexed per 15 Minutes", stdGB AS "Standard Deviation", devs AS "Z-Score"
index=_internal host=*.splunkcloud.com source=*license_usage.log* type="Usage"
| bin _time span=15m
| eval date_wday=strftime(_time,"%w")
| eval date_hour=strftime(_time,"%H")
| eval weekday_weekend=if(((date_wday > 0) AND (date_wday < 6)), "weekday", "weekend")
| eval biz_hours=if((((date_hour < 8) OR (date_hour > 18)) OR (weekend_weekday=="weekend")), "no", "yes")
| stats sum(b) AS byte_sum by idx, weekday_weekend, biz_hours, _time
| join type=inner idx,weekday_weekend, biz_hours [| inputlookup avg_index_bytes_15m.csv]
| eval devs=(byte_sum - average)/std
| eval currentGB=round((byte_sum/1024/1024/1024), 3)
| eval averageGB=round((average/1024/1024/1024), 3)
| eval stdGB=round((std/1024/1024/1024), 3)
| table _time, idx, currentGB, averageGB, devs, stdGB
| where devs < -3 AND averageGB > 0.75
| sort 0 - devs
| rename idx AS Index, currentGB AS "GB Indexed over Past 15 Minutes", averageGB AS "Average GB Indexed per 15 Minutes", stdGB AS "Standard Deviation", devs AS "Z-Score"
index=_internal host=*.splunkcloud.com source=*license_usage.log* type="Usage"
| bin _time span=15m
| eval date_wday=strftime(_time,"%w")
| eval date_hour=strftime(_time,"%H")
| eval weekday_weekend=if(((date_wday > 0) AND (date_wday < 6)), "weekday", "weekend")
| eval biz_hours=if((((date_hour < 8) OR (date_hour > 18)) OR (weekend_weekday=="weekend")), "no", "yes")
| stats sum(b) AS byte_sum by idx, weekday_weekend, biz_hours, _time
| join type=inner idx,weekday_weekend, biz_hours [| inputlookup avg_index_bytes_15m.csv]
| eval devs=(byte_sum - average)/std
| eval currentGB=round((byte_sum/1024/1024/1024), 3)
| eval averageGB=round((average/1024/1024/1024), 3)
| eval stdGB=round((std/1024/1024/1024), 3)
| table _time, idx, currentGB, averageGB, devs, stdGB
| where devs < -3 AND averageGB > 0.75
| sort 0 - devs
| rename idx AS Index, currentGB AS "GB Indexed over Past 15 Minutes", averageGB AS "Average GB Indexed per 15 Minutes", stdGB AS "Standard Deviation", devs AS "Z-Score"
There are two spans that ensure data is accurate. The first takes into account the fact that data may not come in during certain time periods. The following will fill in data during those spans where no logs are generated.
| xyseries _time, idx, byte_sum
| makecontinuous _time span=15m
| fillnull value=0
| untable _time idx byte_sum
| xyseries _time, idx, byte_sum
| makecontinuous _time span=15m
| fillnull value=0
| untable _time idx byte_sum
| xyseries _time, idx, byte_sum
| makecontinuous _time span=15m
| fillnull value=0
| untable _time idx byte_sum
This packs data into a specific format, makes it continuous, fills in null values with a value, and then unpacks the data. Note that the xyseries command takes exactly three arguments. If you have more than three you’ll need to do something like the following:
| stats list(field1) as field1, list(field2) as field2, list(field3) as field3 by idx, byte_sum, _time
| eval zipped=mvzip(mvzip(field1,field2,"!!!!!field2="),field3,"!!!!!field3=")
| mvexpand zipped
| fields _time, idx, byte_sum, zipped
| mvexpand zipped
| rex field=zipped "^(?<field1>.*)!!!!!field2=(?<field2>.*)!!!!!field3=(?<field3>.*)$"
| stats list(field1) as field1, list(field2) as field2, list(field3) as field3 by idx, byte_sum, _time
| eval zipped=mvzip(mvzip(field1,field2,"!!!!!field2="),field3,"!!!!!field3=")
| mvexpand zipped
| fields _time, idx, byte_sum, zipped
| mvexpand zipped
| rex field=zipped "^(?<field1>.*)!!!!!field2=(?<field2>.*)!!!!!field3=(?<field3>.*)$"
| stats list(field1) as field1, list(field2) as field2, list(field3) as field3 by idx, byte_sum, _time
| eval zipped=mvzip(mvzip(field1,field2,"!!!!!field2="),field3,"!!!!!field3=")
| mvexpand zipped
| fields _time, idx, byte_sum, zipped
| mvexpand zipped
| rex field=zipped "^(?<field1>.*)!!!!!field2=(?<field2>.*)!!!!!field3=(?<field3>.*)$"
Using !!!!! is arbitrary; you only need to have a separator string that won’t appear in your data normally.
The second seasonality piece is the following:
| eval date_wday=strftime(_time,"%w")
| eval date_hour=strftime(_time,"%H")
| eval weekday_weekend=if(((date_wday > 0) AND (date_wday < 6)), "weekday", "weekend")
| eval biz_hours=if((((date_hour < 8) OR (date_hour > 18)) OR (weekend_weekday=="weekend")), "no", "yes")
| eval date_wday=strftime(_time,"%w")
| eval date_hour=strftime(_time,"%H")
| eval weekday_weekend=if(((date_wday > 0) AND (date_wday < 6)), "weekday", "weekend")
| eval biz_hours=if((((date_hour < 8) OR (date_hour > 18)) OR (weekend_weekday=="weekend")), "no", "yes")
| eval date_wday=strftime(_time,"%w")
| eval date_hour=strftime(_time,"%H")
| eval weekday_weekend=if(((date_wday > 0) AND (date_wday < 6)), "weekday", "weekend")
| eval biz_hours=if((((date_hour < 8) OR (date_hour > 18)) OR (weekend_weekday=="weekend")), "no", "yes")
This should be more self explanatory. We get the day of the week and the hour from the timestamp, and we evaluate when it occurs. From this, we can calculate statistics based on which category the event fits into.
Conclusion
With these techniques, you can now incorporate seasonality in your searches. It is a powerful technique which can really help you cut down on the noise in your alerts. Be sure to keep an eye out for Part 3 of this series–I’ll be taking a look at some less commonly-used commands and how they may (or may not) be useful in your investigations.
