Have you ever found yourself wondering what ports your Linux-based Splunk installation was listening on, but weren’t sure how to go about finding out?
Common advice for solving this problem is to use either the netstat or ss command with the ‘-p’ flag to show program names and grep for the name of the process.
splunk@hdf-template:~$ ss -ltnup | grep 'splunk'
tcp LISTEN 0 128 0.0.0.0:8089 0.0.0.0:* users:(("splunkd",pid=1327,fd=4))
tcp LISTEN 0 128 0.0.0.0:8000 0.0.0.0:* users:(("splunkd",pid=1327,fd=143))
splunk@hdf-template:~$splunk@hdf-template:~$ ss -ltnup | grep 'splunk'
tcp LISTEN 0 128 0.0.0.0:8089 0.0.0.0:* users:(("splunkd",pid=1327,fd=4))
tcp LISTEN 0 128 0.0.0.0:8000 0.0.0.0:* users:(("splunkd",pid=1327,fd=143))
splunk@hdf-template:~$splunk@hdf-template:~$ ss -ltnup | grep 'splunk'
tcp LISTEN 0 128 0.0.0.0:8089 0.0.0.0:* users:(("splunkd",pid=1327,fd=4))
tcp LISTEN 0 128 0.0.0.0:8000 0.0.0.0:* users:(("splunkd",pid=1327,fd=143))
splunk@hdf-template:~$splunk@hdf-template:~$ ss -ltnup | grep 'splunk'
tcp LISTEN 0 128 0.0.0.0:8089 0.0.0.0:* users:(("splunkd",pid=1327,fd=4))
tcp LISTEN 0 128 0.0.0.0:8000 0.0.0.0:* users:(("splunkd",pid=1327,fd=143))
splunk@hdf-template:~$Need a Hand? Hurricane Labs is Here to Help?
However, Splunk runs as multiple processes rather than a single one, and not all of those processes have “splunk” in the name. In addition to the processes built into Splunk, apps can also be installed that may open their own listening port, which grepping for “splunk” will also not find.
So, what can we do?
On a modern Linux system, the lsof command will list all of the open files that various processes have open. Because Linux treats sockets as files just like it does everything else, we can use specific flags to the lsof command to list all of the listening TCP and UDP ports, then filter out only those that belong to the “splunk” user using the common awk utility.
splunk@hdf-template:~$ ( lsof -nP -i -sTCP:LISTEN ; lsof -nP -iUDP ) | awk 'NR==1 || $3 == "splunk" { print $0 }'
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
splunkd 1292 splunk 4u IPv4 38867 0t0 TCP *:8089 (LISTEN)
splunkd 1292 splunk 149u IPv4 42133 0t0 TCP *:8000 (LISTEN)
mongod 1380 splunk 10u IPv4 40825 0t0 TCP *:8191 (LISTEN)
python3.7 1467 splunk 7u IPv4 43288 0t0 TCP 127.0.0.1:8065 (LISTEN)splunk@hdf-template:~$ ( lsof -nP -i -sTCP:LISTEN ; lsof -nP -iUDP ) | awk 'NR==1 || $3 == "splunk" { print $0 }'
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
splunkd 1292 splunk 4u IPv4 38867 0t0 TCP *:8089 (LISTEN)
splunkd 1292 splunk 149u IPv4 42133 0t0 TCP *:8000 (LISTEN)
mongod 1380 splunk 10u IPv4 40825 0t0 TCP *:8191 (LISTEN)
python3.7 1467 splunk 7u IPv4 43288 0t0 TCP 127.0.0.1:8065 (LISTEN)splunk@hdf-template:~$ ( lsof -nP -i -sTCP:LISTEN ; lsof -nP -iUDP ) | awk 'NR==1 || $3 == "splunk" { print $0 }'
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
splunkd 1292 splunk 4u IPv4 38867 0t0 TCP *:8089 (LISTEN)
splunkd 1292 splunk 149u IPv4 42133 0t0 TCP *:8000 (LISTEN)
mongod 1380 splunk 10u IPv4 40825 0t0 TCP *:8191 (LISTEN)
python3.7 1467 splunk 7u IPv4 43288 0t0 TCP 127.0.0.1:8065 (LISTEN)splunk@hdf-template:~$ ( lsof -nP -i -sTCP:LISTEN ; lsof -nP -iUDP ) | awk 'NR==1 || $3 == "splunk" { print $0 }'
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
splunkd 1292 splunk 4u IPv4 38867 0t0 TCP *:8089 (LISTEN)
splunkd 1292 splunk 149u IPv4 42133 0t0 TCP *:8000 (LISTEN)
mongod 1380 splunk 10u IPv4 40825 0t0 TCP *:8191 (LISTEN)
python3.7 1467 splunk 7u IPv4 43288 0t0 TCP 127.0.0.1:8065 (LISTEN)Conclusion
Using this method, you can be more confident about catching every possible process running as the dedicated service account for Splunk. You may even find that a previous Splunk administrator has set up other listening processes that are unrelated to Splunk but still run as the same user that you may not have known about!
Happy Splunking!
