How to solve "reverse proxy broken" error

June 24, 2020 6 minutes read

When you decide to install Jenkins, unless you plan to use it for testing or training purposes, usually it is not accessed directly. If it is going to be used for more than a couple of users, thus teams or corporate environments, you may want to put a load balancer or Web server in front of it.

By default, Jenkins listens on port 8080 and it is pretty common to expose it through a more secure HTTPS layer.

If you set up this kind of configuration for the first time, chances are you will get an error in the Jenkins UI saying that the reverse proxy is broken

In this post I will show you how to fix this error by helping you understand what does it mean and why it occurs.

Setting up a reverse proxy

You are probably more used to deal with the typical corporate proxy (also called forward proxy) that companies usually require you to setup for accessing the Internet or external resources.

While a forward proxy is used by clients like web browsers, reverse proxies are used by the other end of the communication, for example, Web servers.

In this context, a reverse proxy is the Web server used as the public entry point for accessing an internal Jenkins instance not publicly accesible.

There are many types of reverse proxies, both hardware and software. For this example we will be using nginx, one of the most popular software solutions out there.

The idea is to expose Jenkins to the users by setting up a nginx server that will enable an HTTPS communication with the user an will redirect the traffic to the underlying Jenkins instance.

The minimal nginx configuration for redirecting HTTPS traffic to Jenkins is the following

server {
    listen          80;
    server_name     jenkins.sergiosanchez.com;   (1)

    return          301 https://$host$request_uri;     (2)
}

server {
    listen                  443 ssl;
    server_name             jenkins.sergiosanchez.com;    (3)

    ssl_certificate         /etc/nginx/certs/nginx-selfsigned.crt;
    ssl_certificate_key     /etc/nginx/certs/nginx-selfsigned.key;     (4)

    location / {
        proxy_pass          http://172.18.55.20:8083;     (5)
    }
}

1	HTTP listening on http://jenkins.sergiosanchez.com
2	Redirect HTTP to HTTPS for better security
3	HTTPS listening on https://jenkins.sergiosanchez.com
4	Self signed certificate required for enabling https. Detailed instructions on how to generate them here
5	Rule for forwarding requests to the Jenkins instance host and port

After configuring nginx, you need to let Jenkins know the URL that users will use to access it. This is set in the Jenkins Location configuration section, located in Manage Jenkins → Global Configuration

Jenkins will use this parameter for example to compose the job URLs included in notification emails.

Once this has been setup, if you access to the Global Configuration page (through Manage Jenkins menu link) you will spot the error commented at the beginning of the post.

Why is this happening?

Clicking on the More Info button next to the error message Jenkins forwards you to a detailed page explaining the problem.

Basically it is caused because the embedded Tomcat Web server in which Jenkins is running is unable to know by itself the protocol, host and port of the entry point for the Jenkins instance (nginx in this case).

To troubleshoot this problem, Jenkins provides a test endpoint that you can use to verify the configuration and get some hints on why it is not working.

You can see below an example of invoking this endpoint using cURL

curl \
    -u <user>:<password> \  (1)
    -k \                    (2)
    -i \                    (3)
    -L \                    (4)
    -e https://jenkins.sergiosanchez.com/manage https://jenkins.sergiosanchez.com/ \  (5)
    administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/test              (6)

1	Only required if anonymous access is disabled in Jenkins security configuration
2	Ignore SSL errors (caused in this case due to self-signed certificates)
3	Include HTTP response headers (for debugging purposes)
4	Follow URL provided in Location HTTP header
5	Simulates the source URL from which the test endpoint is being invoked (referer). In this case I’m using the context root "/" but "/jenkins" is also a typical configuration.

The test endpoint invoked by the cURL command just returns the HTTP header Location along with a redirect 302 HTTP status code.

Since with have provided -L parameter, cURL will automatically detect the 302 redirect code and will invoke the URL set in Location HTTP header

The Location header is built based on the Jenkins Location URL set in Global Configuration and will point to the testForReverseProxySetup endpoint including as path parameter the URL set in the Referrer parameter (-e) provided in the cURL command.

Test endpoint URL pattern

<Jenkins Location URL>/testForReverseProxySetup/<Encoded Referer Parameter>

The goal of this endpoint it to compare the parameter (Referer URL) with the URL generated by Jenkins with the information it has in order to verify if it matches.

Since we are providing neither protocol nor host nor port to Jenkins, the URL built by it will contain Jenkins own protocol, host and port thus http://<jenkins_host>:<jenkins_port>;

Since both URLs don’t match, an 404 error is returned, confirming there is an error in the proxy configuration. In this case you will found something like this in cURL command output:

HTTP/1.1 404 http://192.168.144.5:8083/manage vs. https://jenkins.sergiosanchez.com/manage
Server: nginx/1.17.10
Date: Sat, 23 May 2020 16:46:59 GMT
Content-Type: text/html;charset=iso-8859-1
Content-Length: 598
Connection: keep-alive
X-Content-Type-Options: nosniff
Cache-Control: must-revalidate,no-cache,no-store

How to fix it

Therefore, the solution is to provide nginx configuration information from the reverse proxy (nginx) to Jenkins in the form of HTTP headers.

In the case of nginx the solution is pretty simple. We basically have to include the following information when redirecting requests to Jenkins by means of HTTP standard headers:

proxy_set_header        Host $host:$server_port;     (1)
proxy_set_header        X-Forwarded-Proto $scheme;   (2)

1	Host header containing nginx host and port
2	X-Forwarded-Proto contaning protocol used by the user (https)

Another way of achieving the same effect -and the approach recommended in Jenkins documentation- is setting all three required values in separate X-Forwarded-* headers:

proxy_set_header        X-Forwarded-Host $host;
proxy_set_header        X-Forwarded-Port $server_port;
proxy_set_header        X-Forwarded-Proto $scheme;

So the final nginx.conf file content file would be something like:

server {
    listen       80;
    server_name  jenkins.sergiosanchez.com;

    return       301 https://$host$request_uri;
}

server {
    listen       443 ssl;
    server_name  jenkins.sergiosanchez.com;

    ssl_certificate     /etc/nginx/certs/nginx-selfsigned.crt;
    ssl_certificate_key /etc/nginx/certs/nginx-selfsigned.key;

    location / {
        proxy_pass          http://172.18.55.20:8083;
        proxy_set_header    Host $host:$server_port;
        proxy_set_header    X-Forwarded-Proto $scheme;
    }
}

After making this changes and restarting nginx, if we execute the cURL command again we will get an 200 HTTP status code, confirming the configuration is working OK.

HTTP/1.1 200 OK

Is you access to Manage Jenkins page in the Jenkins console, no broken proxy message should appear now.

Testing the solution with Docker

An easy way of validating the solution locally is using Docker. I will show you how to setup a basic configuration using official Nginx and Jenkins Docker images.

To simplify configuration, we will use Docker Compose tool to build the infrastructure. However the following steps are required:

Create self-signed certificate for HTTPS access. You can follow these instructions to generate them. Bear in mind that the browser will show the page as insecure because the certificate is not signed by a valid CA
Clone the git repository containing the Docker Compose configuration file (docker-compose.yml) and basic nginx configuration
Add an entry in hosts file pointing to localhost for the name jenkins.sergiosanchez.com

You will find more instructions in the repository

After starting the docker infrastructure, if you access to Manage Jenkins → Global configuration in the Jenkins console and set the Jenkins Location URL, you will see no reverse proxy error message.

In addition, if you check the configuration using Jenkins test endpoint, you will get a successful configuration message (HTTP Error code 200)

Demo tested on a WSL2/Windows 10 environment

As usual, you can find this code in GitHub