Configuring HAProxy & Splunk With REST API & SDK Compatibility

As a customer of Splunk I used HAProxy as a software load balancer to distribute users amongst my search heads. I was using the old search head pooling technology at the time, but the same principal holds true for our search head clustering feature; both require a load balancer to distribute users to your search heads. At the time, I couldn’t quite get HAProxy configured to allow use of the REST API. I now believe that was a function of the fact that I was on the 1.4.x branch which didn’t support SSL proxying.

Late last year I had a customer who used our professional services to help with a project. It revolved around using our SDK’s and REST API to surface some data to their customers. The customer was also my previous employer and they were still using the HA Proxy server that I had originally configured. Since the project required proxying of REST API and SDK calls, an upgrade and re-config of HAProxy was in order. Luckily, HAProxy had released the 1.5.x branch which supports SSL proxying.

Here are the steps for configuring HAProxy to proxy users to the default web port 8000 and SSL REST API/SDK requests to the default management port 8089.

1. Downloaded HAProxy 1.5.12 (latest version as of this post)

2. Verify that gcc, openssl-dev, zlib-dev and pcre-dev are installed with yum/dpkg or whatever your Linux distro package manager is.

3. Build and install HAProxy

$ tar -zxf haproxy-1.5.12.tar.gz
$ cd haproxy-1.5.12
$ sudo make install
$ ln -s /usr/local/sbin/haproxy /usr/sbin/

4. Install attached init script to /etc/init.d/haproxy (RHEL/CentOS compatible)

5. Create directory (if it doesn’t already exist) and create blank config file.

# mkdir /etc/haproxy
# touch /etc/haproxy/haproxy.cfg

6. Copy your Splunk SSL certificate to the HAProxy config dir. If you’re using the default certificate that ships with splunk then you’d copy $SPLUNK_HOME/etc/auth/server.pem and place that on your HAProxy server in /etc/haproxy/cert.pem. I’d highly encourage you to generate your own SSL certificate and use this in place of the default certificate. Here are the docs for configuring Splunk to use your own SSL certicicate.

7. Modify the following config by substituting your Splunk search head server names for the highlighted servers. Paste the contents in /etc/haproxy/haproxy.cfg. Feel free to tune any of the timeouts or maxconn in the global and defaults section. I’m by no means an HAProxy expert. To add more servers update the frontend and backend sections adding your search heads and giving them a unique name; e.g. ‘server s3’ for the third search head.

    maxconn 2048
    tune.ssl.default-dh-param 2048

    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

frontend http-in
    bind *:8000
    reqadd X-Forwarded-Proto:\ http
    default_backend backend-splunk-http

frontend https-in
    bind *:8089 ssl crt ./cert.pem
    reqadd X-Forwarded-Proto:\ https
    default_backend backend-splunk-https

backend backend-splunk-https
    # Use load balancer session cookie persistence
    balance roundrobin
    cookie SERVERID insert indirect nocache
    server s1 splunk-server-1:8089 ssl verify none check cookie s1
    server s2 splunk-server-2:8089 ssl verify none check cookie s2

backend backend-splunk-http
    # Use load balancer session cookie persistence
    balance roundrobin
    cookie SERVERID insert indirect nocache
    server s1 splunk-server-1:8000 check cookie s1
    server s2 splunk-server-2:8000 check cookie s2

8. Start HAProxy and supply the certificate password of ‘password’ if you are using the default SSL certificate that ships with Splunk.

# /etc/init.d/haproxy start

9. Test your setup. Substitute your HAProxy server in the highlighted section




curl -k -u admin:changeme https://haproxy-server:8089/services/licenser/slaves

You should now have a working HAProxy software load balancer that is compatible with the REST API and SDK’s.

In splunklib 1.2.3 cookie handling is broken, so the second request will not send cookies returned from the first which makes this not work. In 1.5.0, it works. I’m not sure exactly at what release in between it was fixed.

This problem will present as “splunklib.binding.AuthenticationError: Request failed: Session is not logged in.” in your Python application.

Alex Davies
October 10, 2015

Hello to all, how is everything, I think every one is getting more from this site, and your views
are pleasant for new visitors.

October 31, 2015