- Adam Army
- Apr 15, 2019
- Tested on Splunk Version: All versions
Many of the administrative tasks surrounding Splunk certificates are confusing and lack helpful step-by-step instructions. We’ve been using this guide internally at Hurricane Labs for some time now, and felt it would be an awesome resource for the Splunk community. With this ‘Master Guide’, we hope to provide a map on your quest to conquer all of your Splunk certificate needs.
Brian Glenn, Technical Review of Original Documentation
Tom Kopchak, Public Tutorial Adaptation
One of the most common yet frequently overlooked Splunk administrative tasks revolves around certificates. By default, Splunk uses a self-signed certificate for the ports that it uses for its web interface, API, and data receiving. However, it is best practice to configure Splunk to use certificates issued and trusted by your organization.
This ‘Master Guide’ should serve as a resource that offers you clarity along with instructions to help you overcome some of these Splunk certificates challenges.
Before proceeding with your certificate changes there are a few things we would like to cover, including the various types of Splunk certificates, Hurricane Labs’ recommended standards, and a couple more quick pointers.
Note: Always make backups and test your settings! Failure to configure certificates correctly can result in splunkweb and/or Splunk not starting correctly.
This is for browser communication, commonly encountered when using splunkweb (web ports such as 443, 8443, or 8000).
This is for internal Splunk communication (typically port 8089).
This is for sending data from forwarders to indexers or other forwarders securely.
Use the common name (CN, hostname) when naming items.
Use the same certificate provided for splunkweb for splunkd as well! Both should be replaced! (Note: This should be observed for any Splunk system within your environment, not just search heads).
Any file containing the private key (<hostname>.key and <hostname>.pem, for example) should have the following permissions set:
If you don’t have the need to use a specific Certificate Authority, you may be able to leverage a free service known as ‘Let’s Encrypt’ to generate certificates to use instead of the self-signed ones that Splunk comes with. Splunk has also published a quick tutorial on using ‘Let’s Encrypt’ with Splunk that may be helpful for you to check out.
Now you’re ready to get started!
The CSR can be generated on the Splunk system itself.
Although best security practices would recommend generating a new key and CSR each time, it is not uncommon to take advantage of an existing key, or CSR if replacing an expired/expiring certificate.
You should confirm the following before re-using this material:
openssl rsa -in <hostname>.key -text -noout | grep "Private-Key"
Note: This should be at least 2048, ideally 4096.
openssl req -in <hostname>.csr -noout -text | grep -A 1 "X509v3 Subject"
This will show the settings that were applied from /etc/ssl/openssl.cnf when the CSR was generated.
These values should make sense for the system, in other words ensure the DNS names and IP address are correct. If this is a clustered system, this part requires special attention. The preferred method is to use a separate certificate for each member where the subject name is the server and the Subject Alternative Name includes the load-balancer, VIP, or whatever is used to access the clustered environment. This will vary depending on your specific case.
The example below will help guide you through how to generate a new CSR.
First, determine if an existing CSR is available and if it can be used.
To create a new RSA key:
openssl genrsa -out `hostname`.key 4096
Before generating the CSR, we will need to know all the hostnames the system may be accessed by and all the IP addresses it may use as well.
The following information may also be helpful, if appropriate in your environment:
You may need to adjust the OpenSSL configuration, double check the following in /etc/ssl/openssl.cnf:
openssl req -new -key `hostname`.key -out `hostname`.csr
root@example:/opt/splunk/etc/auth/mycerts# openssl req -new -key example.key -out example.csr You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [AU]:US State or Province Name (full name) [Some-State]:New York Locality Name (eg, city) :New York Organization Name (eg, company) [Internet Widgits Pty Ltd]:The Example Magical Products Company Organizational Unit Name (eg, section) :MPC Common Name (e.g. server FQDN or YOUR name) :example Email Address :firstname.lastname@example.org Please enter the following 'extra' attributes to be sent with your certificate request A challenge password : An optional company name :
Prior to sending the CSR to a certificate signing authority, verify it! To verify the CSR:
openssl req -in <hostname>.csr -noout -text
This CSR can then be copied to your local system and provided to your certificate signing authority to generate the certificate. What you will receive back will likely differ between each certificate provider!
1. You will want to assemble a <hostname>.crt file that contains the server certificate at the top, and the chain below it all the way to the root (at the bottom).
2. Do not be confused if your certificate provider sends you back a file named <hostname>.crt, this does not mean it contains everything you need. You may need to rename this (or other files) so that you can use the name to these standards.
3. A full certificate chain (from the server cert to the CA) should be used for every certificate that is deployed.
4. The certificate you receive from your provider may vary. You might receive one of the following:
5. You should be able to cat the certificate file to tell if it contains more than one certificate. A certificate file with only the server certificate will look like this:
-----BEGIN CERTIFICATE----- … -----END CERTIFICATE-----
Whereas a certificate file with the server certificate and certificate chain will look like this:
-----BEGIN CERTIFICATE----- … -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- … -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- … -----END CERTIFICATE-----
6. You will need to combine these in the right order in a completed <hostname>.crt file.
Note: If you receive certificate files that are in a format other than X509, you will need to convert them. An example of converting a .pfx file, which is Windows PKCS12:
The provider gave you a .pfx file, which is Windows PKCS12. This file contains our certificates and key, and we need to extract them. Since this is a password protected file, you will need the password used.
echo -n "<provided_password>" | /usr/bin/openssl pkcs12 -in <pfxfile> -passin stdin -nocerts -passout pass:password -out /opt/splunk/etc/auth/splunkweb/<hostname>.key.new
Since we’re dealing with a key file, OpenSSL requires us to encrypt the extracted key file, which is what we’re doing above. However, we’ll unencrypt that key using the password we set in the command above. Note: in the command above it uses password, it doesn’t need to be secure since we’re just using it to decrypt, and this is different than the provided password.
/usr/bin/openssl rsa -in /opt/splunk/etc/auth/splunkweb/<hostname>.key.new -passin pass:password -out /opt/splunk/etc/auth/splunkweb/<hostname>.key
Now we have our unencrypted key file, <hostname>.key, we can throw away the encrypted version that we extracted from the original .pfx file:
rm -f /opt/splunk/etc/auth/splunkweb/<hostname>.key.new
We have the key file, now we need to get the certificate file. This is similar, but since it doesn’t involve a key we don’t need to worry about encryption:
echo -n "<customer_provided_password>" | /usr/bin/openssl pkcs12 -in "pfxfile" -passin stdin -nokeys -out /opt/splunk/etc/auth/splunkweb/<hostname>.crt
In a certificate, the subject is the host the certificate is for and the issuer is the host that signed it. A certificate chain should then start with a subject of the server you are generating the certificate for, then a certificate for whatever host signed that certificate, and so on until the subject and issuer are the same. That would be the root Certificate Authority (CA).
Subject: CN=example Issuer: CN=cert3 Subject: CN=cert3 Issuer: CN=cert2 Subject: CN=cert2 Issuer: CN=cert1 Subject: CN=cert1 Issuer: CN=cert1
You can use the OpenSSL command to view the subject and issuer of a certificate file, along with other information, such as:
openssl x509 -in /opt/splunk/etc/auth/splunkweb/<hostname>.crt -text -noout
Combining the cat command with the redirection >> into the certificate file (<hostname>.crt) should append the contents of the intermediate certificate to the certificate file (<hostname>.crt):
cat /opt/splunk/etc/auth/splunkweb/examplechain.crt >> /opt/splunk/etc/auth/splunkweb/example.crt
Likewise, you can manually copy and paste these in a text editor to compile the full certificate chain in <hostname>.crt.
This is the step where order matters - ensure that you are appending the parts of the certificate chain in order, beginning with the server certificate, followed by any intermediate certificates (in the right order), followed by the root certificate.
Place the complete <hostname>.crt (containing server cert and full chain) on the Splunk server in $SPLUNK_HOME/etc/auth/splunkweb
At this point, you should have the following certificate materials in $SPLUNK_HOME/etc/auth/splunkweb:
Once we have these materials, we should verify that they match. Since you may have handed the CSR off to a third-party, you must rely on them to sign it correctly. Unfortunately, this isn’t always handled appropriately.
We can confirm to some extent that this was done correctly by comparing the modulus (think of it like a thumbprint) of the RSA key we generated the CSR with against the modulus of the certificate they provided (they should match).
To obtain the modulus of the certificate:
openssl x509 -in /opt/splunk/etc/auth/splunkweb/example.crt -noout -modulus
Example output (some text omitted):
To obtain the modulus of the RSA key:
openssl rsa -in /opt/splunk/etc/auth/splunkweb/example.key -noout -modulus
Example output (some text omitted):
Note: If these do not match for some reason, it means there was an issue with the certificate signing, commonly that the CSR you generated was not used. You will want to have the certificate re-signed using the original CSR at this point and try again.
To complete the PEM file needed for the splunkd certificate, we need to set a password to the RSA key and add that password to our PEM file:
Start by confirming that an old backup doesn’t exist:
Look for anything .pem or .pem.old
Move the current PEM splunkd certificate file to a backup:
mv /opt/splunk/etc/auth/`hostname`.pem /opt/splunk/etc/auth/`hostname`.pem.old
Make a copy of your new PEM splunkd certificate file, before making further changes:
cp /opt/splunk/etc/auth/splunkweb/`hostname`.crt /opt/splunk/etc/auth/`hostname`.pem
Use the following command to set the password on the RSA private key and then append that password to our new PEM splunkd certificate file:
echo -n "password" | /usr/bin/openssl rsa -in /opt/splunk/etc/auth/splunkweb/`hostname`.key -aes256 -passout stdin >> /opt/splunk/etc/auth/`hostname`.pem
At this point, we should now have a PEM file at /opt/splunk/etc/auth/<hostname>.pem that contains everything splunkd needs:
And we should also have a PEM file at /opt/splunk/etc/auth/splunkweb/<hostname>.crt that has the full certificate chain only that Splunkweb needs.
The final step is to confirm/configure Splunk to use these files. If we’re replacing expired certificates (and we followed standard naming practices), it’s likely that the configuration files are already configured and will just use our new files. If this is the first time changing the certificates from the default self-signed ones that come with Splunk, we’ll need to create local versions of web.conf (for the Splunkweb certificate) and server.conf (for the splunkd certificate).
Note: In most cases, the local configuration for web.conf and server.conf exist in /opt/splunk/etc/system/local, but they could be deployed in app as well. Review your environment to be sure! Remember, btool is your friend here.
Edit (or create) $SPLUNK_HOME/etc/system/local/server.conf:
[sslConfig] serverCert = <hostname>.pem
Edit (or create) $SPLUNK_HOME/etc/system/local/web.conf:
[settings] serverCert = etc/auth/splunkweb/<hostname>.crt privKeyPath = etc/auth/splunkweb/<hostname>.key
Much of the work above will not actually go “into effect” until Splunk (or Splunkweb) is restarted. Therefore if you make these changes and for some reason do not test them, someone else may restart Splunk for an unrelated reason and find themselves with a mess to figure out. DO NOT leave this for someone else to find! Confirm things are working or roll back if needed.
Verify Splunkweb from web browser:
Also, you’ll want to designate port 8089 in the web address to test the splunkd certificate (otherwise, this is the same verification as Splunkweb).
In most cases Splunk will continue to function, but symptoms of having a bad certificate will show up, such as untrusted errors when browsing to the site. In some newer browsers, it may require some difficult workarounds to continue browsing to a site with an expired certificate. There is also a setting (strict cert checking) in Splunk that can be enabled to cause this to break, but this is rarely implemented.
The Splunk command line command splunk create ssl can be used to generate a new self-signed certificate. Note that Splunk will also typically generate new self-signed certificates when upgraded. Be prepared to provide specific information such as server name.
Technically, no, however it is Hurricane Labs’ recommendation to put a certificate on everything. Note that we would want a web certificate on anything that has Splunkweb enabled, and the splunkd certificate anywhere that TCP port 8089 takes place. Self-signed and customer-signed certificates can be used in the same environment without any technical issue, though it is against our standard and recommendation.
If you're looking for something different than the typical "one-size-fits-all" security mentality, you've come to the right place.