Splunk for SharePoint

Background

Splunk is a superb tool for mining your existing data, be it in the form of log files, databases or pretty much anything else. I’ve implemented it at two clients and used it to fulfill distinct requirements. At one it was used at an adhoc fault finding tool which indexed web traffic logs, performance counters and an assortment of  application logs. At the other client it was used as a web analytics tool and to provide data to a web application via a REST API. I’ll talk about these uses later but none of that will be helpful if you can’t get it to index your data in the first instance.

Lets be clear: while I think Splunk is an amazing tool, it is first and foremost aimed at the *nix world (their sales guys will try to convince you otherwise but I assure you this is true.) This first became apparent to me when I started indexing iis logs. If you install Splunk’s indexer on the same server that the logs exist on (or map to it via UNC) then it’ll detect the log files correctly, but this kind of implementation will not work on a larger environment where you’ll need to install forwarders which collect the logs files and send the data to an indexer. In this setup, Splunk fails to detect the log file type and so doesn’t categorise the data within them. The work around follows below (I’ve even shown you how to index ULS logs because I’m nice like that):

Other info

This config was applied to Splunk v4.2.4

Config

• inputs.conf on the forwarder. This tells Splunk to collect log files and point them to the appropriate stanza in props.conf:

[default]
host = YourIndexServer

[monitor://c:\inetpub\logs\LogFiles\]
sourcetype=iis

[monitor://c:\ULSLogs\*]
sourcetype=uls

• props.conf on the forwarder. This tells Splunk not to bother with field extraction. In theory it shouldn’t be needed but I couldn’t get it to work without it:
  

[iis]

CHECK_FOR_HEADER = False

[uls]
CHECK_FOR_HEADER = False

• props.conf on the indexer. This tells Splunk not to perform any field extraction but defines the time stamp type. It also points to the appropriate stanza in transforms.conf :

[iis]
CHECK_FOR_HEADER = False
TIME_PREFIX = :\s
MAX_TIMESTAMP_LOOKAHEAD = 128
TIME_FORMAT = %Y-%m-%d %H:%M:%S
REPORT-iis = iis
TZ = GMT

[uls]
CHECK_FOR_HEADER = False
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 32
TIME_FORMAT = %m/%d/%Y %H:%M:%S.%3N
REPORT-uls = uls
TZ = GMT
SHOULD_LINEMERGE = False
TRANSFORMS-uls-level-null = uls-level-null

• transforms.conf on the indexer. This is where the magic happens. The first two stanzas extract the various fields from the logs (you’ll need to adjust these to your environment)  while the last stanza throws away any uls events which don’t match the regex statement. This allowed us to shrink our indexing requirement from close to 10gb a day to roughly 3gb, and means that we can tweak ULS logging without fear of breaching our license :

[iis]
FIELDS=”date”, “time”, “s-ip”, “cs-method”, “cs-uri-stem”, “cs-uri-query”, “s-port”, “cs-username”, “c-ip”, “cs(User-Agent)”, “cs(Referer)”, “sc-status”, “sc-substatus”, “sc-win32-status”, “time-taken”
DELIMS = ” ”

[uls]
FIELDS=”Timestamp”, “Process”, “TID”, “Area”, “Category”, “EventID”, “Level”, “Message”, “Correlation”
DELIMS = “\t”

[uls-level-null]
REGEX = ^(?:[^\t]+\t){6}(?!Critical|High|Unexpected|Exception)
FORMAT = nullQueue
DEST_KEY = queue