Splunk Certificates: Master Guide

Contributing Authors:
Brian Glenn, Technical Review of Original Documentation
Tom Kopchak, Public Tutorial Adaptation

One of the most common yet frequently overlooked Splunk administrative tasks revolves around certificates. By default, Splunk uses a self-signed certificate for the ports that it uses for its web interface, API, and data receiving. However, it is best practice to configure Splunk to use certificates issued and trusted by your organization.

This ‘Master Guide’ should serve as a resource that offers you clarity along with instructions to help you overcome some of these Splunk certificates challenges.

Things to know before you begin.

Before proceeding with your certificate changes there are a few things we would like to cover, including the various types of Splunk certificates, Hurricane Labs’ recommended standards, and a couple more quick pointers.

Note: Always make backups and test your settings! Failure to configure certificates correctly can result in splunkweb and/or Splunk not starting correctly.

There are three types of Splunk certificates:

web

This is for browser communication, commonly encountered when using splunkweb (web ports such as 443, 8443, or 8000).

• While splunkweb does not default to using HTTPS, it is recommended that HTTPS always be used. Failure to configure HTTPS would result in credentials being available in plaintext and not provide any protection in transit for data being viewed by a Splunk user.
• Web communication that uses SSL (HTTPS) requires a certificate to avoid the “Your connection is not private” warning message when browsing.

splunkd

This is for internal Splunk communication (typically port 8089).

• splunkd is the Splunk API port – this is used for many functions within Splunk, such as monitoring, internal communication, deployment server functionality, and interaction with the environment using other tools or systems.
• You can test this by specifying port 8089 in a web browser – splunkd runs as HTTPS by default.

s2s

This is for sending data from forwarders to indexers or other forwarders securely.

• Out of the three, this is the least likely to see deployed in the field.
• The most commonly used port for s2s is tcp/9997. Compression is common, encryption is significantly less common.
• We will not be covering this type of certificate in this tutorial.

General

• The same certificate material can (and should) be used for web and splunkd certificates. It is not recommended that the instance’s local certificate be used for anything production. In most cases, not only is there no reason to use different certificates but using the same allows for ease of management. Note that there may be extraneous circumstances that present a valid need to separate certificates, but these tend to be rare. More customized work needs to be done with the customer for the s2s option.
• Certificate material should be kept in $SPLUNK_HOME/etc/auth. Note that additional directories may be created and used.

Hurricane Labs’ Recommended Standards

Use the common name (CN, hostname) when naming items.

Use the same certificate provided for splunkweb for splunkd as well! Both should be replaced! (Note: This should be observed for any Splunk system within your environment, not just search heads).

Any file containing the private key (<hostname>.key and <hostname>.pem, for example) should have the following permissions set:

• NEVER have “everyone” read/write/execute permissions (the last digit should be 0), or on Windows “Everyone” should not be included in permissions.
• Group read permissions may make sense, but in most cases it will not. Group members probably should not be able to read/execute either.
• When in doubt, chmod 400 as a starting point.
• For Windows users, similar permissions can be achieved via Properties > Security > Advanced on the key file, allowing only the key owner access:

A quick note on ‘Let’s Encrypt’.

If you don’t have the need to use a specific Certificate Authority, you may be able to leverage a free service known as ‘Let’s Encrypt’ to generate certificates to use instead of the self-signed ones that Splunk comes with. Splunk has also published a quick tutorial on using ‘Let’s Encrypt’ with Splunk that may be helpful for you to check out.

Now you’re ready to get started!

Begin by creating a CSR (Certificate Signing Request).

The CSR can be generated on the Splunk system itself.

Although best security practices would recommend generating a new key and CSR each time, it is not uncommon to take advantage of an existing key, or CSR if replacing an expired/expiring certificate.

You should confirm the following before re-using this material:

Key Size

Note: This should be at least 2048, ideally 4096.

X509v3 Subject Alternative Names

This will show the settings that were applied from /etc/ssl/openssl.cnf when the CSR was generated.

These values should make sense for the system, in other words ensure the DNS names and IP address are correct. If this is a clustered system, this part requires special attention. The preferred method is to use a separate certificate for each member where the subject name is the server and the Subject Alternative Name includes the load-balancer, VIP, or whatever is used to access the clustered environment. This will vary depending on your specific case.

To create a new CSR use the following procedure.

The example below will help guide you through how to generate a new CSR.

Step 1

First, determine if an existing CSR is available and if it can be used.

• As mentioned before, you may be able to use an existing CSR, or existing key if one exists. A system built to the Hurricane Labs standard would store these at $SPLUNK_HOME/etc/auth/splunkweb/, but you may have to look around to see if they exist in an alternative location. (Note: Be sure to verify the key length and Subject Alternative Names as noted above).
• If an existing CSR is not available you will need to create a new RSA key and CSR with it.

Step 2

To create a new RSA key:

Before generating the CSR, we will need to know all the hostnames the system may be accessed by and all the IP addresses it may use as well.

The following information may also be helpful, if appropriate in your environment:

• Email address
• Organizational unit

Step 3

You may need to adjust the OpenSSL configuration, double check the following in /etc/ssl/openssl.cnf:

1. Uncomment this line: # req_extensions = v3_req # The extensions to add to a certificate request
2. Under [ v3_req ] add the following line:
   subjectAltName = DNS:<FQDN>, DNS:<hostname>, IP:<ip_address>

Example output:

Step 4

Prior to sending the CSR to a certificate signing authority, verify it! To verify the CSR:

What’s next?

This CSR can then be copied to your local system and provided to your certificate signing authority to generate the certificate. What you will receive back will likely differ between each certificate provider!

1. You will want to assemble a <hostname>.crt file that contains the server certificate at the top, and the chain below it all the way to the root (at the bottom).

2. Do not be confused if your certificate provider sends you back a file named <hostname>.crt, this does not mean it contains everything you need. You may need to rename this (or other files) so that you can use the name to these standards.

3. A full certificate chain (from the server cert to the CA) should be used for every certificate that is deployed.

4. The certificate you receive from your provider may vary. You might receive one of the following:

• Separate certificates (server certificate, intermediate certificate(s), root certificate)
• The server certificate on its own and the rest of the certificate chain in one file
• Any combination of the above

5. You should be able to cat the certificate file to tell if it contains more than one certificate. A certificate file with only the server certificate will look like this:

Whereas a certificate file with the server certificate and certificate chain will look like this:

6. You will need to combine these in the right order in a completed <hostname>.crt file.

Note: If you receive certificate files that are in a format other than X509, you will need to convert them. An example of converting a .pfx file, which is Windows PKCS12:

Example: Convert PKCS12 to X509

The provider gave you a .pfx file, which is Windows PKCS12. This file contains our certificates and key, and we need to extract them. Since this is a password protected file, you will need the password used.

Since we’re dealing with a key file, OpenSSL requires us to encrypt the extracted key file, which is what we’re doing above. However, we’ll unencrypt that key using the password we set in the command above. Note: in the command above it uses password, it doesn’t need to be secure since we’re just using it to decrypt, and this is different than the provided password.

Now we have our unencrypted key file, <hostname>.key, we can throw away the encrypted version that we extracted from the original .pfx file:

We have the key file, now we need to get the certificate file. This is similar, but since it doesn’t involve a key we don’t need to worry about encryption:

A quick note on investigating certificate chains.

In a certificate, the subject is the host the certificate is for and the issuer is the host that signed it. A certificate chain should then start with a subject of the server you are generating the certificate for, then a certificate for whatever host signed that certificate, and so on until the subject and issuer are the same. That would be the root Certificate Authority (CA).

For example:

You can use the OpenSSL command to view the subject and issuer of a certificate file, along with other information, such as:

Example: Compile the certificate chain into the server certificate file.

Combining the cat command with the redirection >> into the certificate file (<hostname>.crt) should append the contents of the intermediate certificate to the certificate file (<hostname>.crt):

Likewise, you can manually copy and paste these in a text editor to compile the full certificate chain in <hostname>.crt.

This is the step where order matters – ensure that you are appending the parts of the certificate chain in order, beginning with the server certificate, followed by any intermediate certificates (in the right order), followed by the root certificate.

Place the complete <hostname>.crt (containing server cert and full chain) on the Splunk server in $SPLUNK_HOME/etc/auth/splunkweb

At this point, you should have the following certificate materials in $SPLUNK_HOME/etc/auth/splunkweb:

1. Server certificate, <hostname>.crt
2. RSA key, <hostname>.key
   a.  We created this above when we generated the CSR
3. CSR, <hostname>.csr
   a.  This was also created above

Once we have these materials, we should verify that they match. Since you may have handed the CSR off to a third-party, you must rely on them to sign it correctly. Unfortunately, this isn’t always handled appropriately.

We can confirm to some extent that this was done correctly by comparing the modulus (think of it like a thumbprint) of the RSA key we generated the CSR with against the modulus of the certificate they provided (they should match).

Example: Compare modulus of RSA key against customer-provided certificate to ensure they match.

To obtain the modulus of the certificate:

Example output (some text omitted):

To obtain the modulus of the RSA key:

Example output (some text omitted):

Note: If these do not match for some reason, it means there was an issue with the certificate signing, commonly that the CSR you generated was not used. You will want to have the certificate re-signed using the original CSR at this point and try again.

To complete the PEM file needed for the splunkd certificate, we need to set a password to the RSA key and add that password to our PEM file:

Example: Set password on RSA private key and add it to splunkd PEM file.

Start by confirming that an old backup doesn’t exist:

Look for anything .pem or .pem.old

• If a .old file exists, it is likely an older backup, and you should decide if you want to keep it as we will be overwriting it in the next step.

Move the current PEM splunkd certificate file to a backup:

Make a copy of your new PEM splunkd certificate file, before making further changes:

Use the following command to set the password on the RSA private key and then append that password to our new PEM splunkd certificate file:

At this point, we should now have a PEM file at /opt/splunk/etc/auth/<hostname>.pem that contains everything splunkd needs:

• Server certificate
• Certificate chain
• RSA private key (with password set)

And we should also have a PEM file at /opt/splunk/etc/auth/splunkweb/<hostname>.crt that has the full certificate chain only that Splunkweb needs.

Configuring Splunk to use the new certificate.

The final step is to confirm/configure Splunk to use these files. If we’re replacing expired certificates (and we followed standard naming practices), it’s likely that the configuration files are already configured and will just use our new files. If this is the first time changing the certificates from the default self-signed ones that come with Splunk, we’ll need to create local versions of web.conf (for the Splunkweb certificate) and server.conf (for the splunkd certificate).

Example: Configuring Splunk to use customer-signed certificates

Note: In most cases, the local configuration for web.conf and server.conf exist in /opt/splunk/etc/system/local, but they could be deployed in app as well. Review your environment to be sure! Remember, btool is your friend here.

Edit (or create) $SPLUNK_HOME/etc/system/local/server.conf:

• Note that the serverCert assumes /opt/splunk/etc/auth as the filepath here, if for some reason it is in a different place (it really shouldn’t be though) you can designate the full path.

Edit (or create) $SPLUNK_HOME/etc/system/local/web.conf:

• Note that Splunk should know what SPLUNK_HOME is (such as /opt/splunk) so we don’t need to include it here:

Be sure to test and verify!

Much of the work above will not actually go “into effect” until Splunk (or Splunkweb) is restarted. Therefore if you make these changes and for some reason do not test them, someone else may restart Splunk for an unrelated reason and find themselves with a mess to figure out. DO NOT leave this for someone else to find! Confirm things are working or roll back if needed.

Verify Splunkweb from web browser:

• Command+option+i (In Google Chrome, on a Mac)
• Control+shift+i (In Google Chrome, on Windows)
• Go to Security, then view the cert, make sure it has the new info on it

Also, you’ll want to designate port 8089 in the web address to test the splunkd certificate (otherwise, this is the same verification as Splunkweb).

FAQ

Q: What happens when a certificate expires? Does Splunk continue to function?

In most cases Splunk will continue to function, but symptoms of having a bad certificate will show up, such as untrusted errors when browsing to the site. In some newer browsers, it may require some difficult workarounds to continue browsing to a site with an expired certificate. There is also a setting (strict cert checking) in Splunk that can be enabled to cause this to break, but this is rarely implemented.

Q: Although our recommendation is to use customer-signed certificates, how would one renew a Splunk self-signed certificate if it expired?

The Splunk command line command splunk create ssl can be used to generate a new self-signed certificate. Note that Splunk will also typically generate new self-signed certificates when upgraded. Be prepared to provide specific information such as server name.

Q: Do all Splunk infrastructure pieces need a customer-signed certificate?

Technically, no, however it is Hurricane Labs’ recommendation to put a certificate on everything. Note that we would want a web certificate on anything that has Splunkweb enabled, and the splunkd certificate anywhere that TCP port 8089 takes place. Self-signed and customer-signed certificates can be used in the same environment without any technical issue, though it is against our standard and recommendation.