How to restrict access with basic authentication

Protect an Apache site on TurboStack with HTTP basic authentication while letting trusted IP addresses through without a login, including the Varnish case.

In this article we tackle the problem of how to decide whether a visitor should or should not log in on a server with basic authentication enabled, based on their IP address.

So what is the result we want to achieve? We want to implement an .htpasswd file so visitors need a valid login, except when the request comes from a whitelisted IP address. In that case no login is asked and the visitor is taken straight to the site. Like a VIP that would skip the waiting queue for a club.

This method is used for Apache (apache2).

Method 1: Server without Varnish enabled

There is a difference when a server has or does not have Varnish enabled. For now we keep it simple and assume there is no interruption from any service like Varnish. In that case we use the following setup.

For best practice, put this code at the top of your .htaccess file:

AuthType Basic
AuthName "Restricted content"
AuthUserFile /var/www/prod/apache2/.htpasswd

# Whitelisted IPs are granted access without a login prompt
Require ip 203.0.113.10
# Only a person with valid credentials is let in
Require valid-user

Apache treats the two Require lines as "either one is enough": a request from 203.0.113.10 is allowed without a prompt, and every other request must supply a valid login.

Method 2: Server with Varnish enabled

For a server with Varnish enabled, a different approach is needed. All requests that go through Varnish pass the X-Forwarded-For header, but it may contain some tampered information about the visitor's IP. Because of this, the request for immediate access is denied and the visitor is asked to log in. To make sure this does not happen, we add a variable for the header that contains the whitelisted IP address.

The code below does the trick (the IP should be written between quotes):

AuthType Basic
AuthName "Restricted content"
AuthUserFile /var/www/prod/apache2/.htpasswd

# Only a person with valid credentials is let in
Require valid-user
# Create the variable for the header (the IP should be written between quotes)
SetEnvIf X-Forwarded-For 203.0.113.10 AllowIP
# Include the env variable
Require env AllowIP

A request whose X-Forwarded-For header contains 203.0.113.10 is let through; everyone else is asked to log in.

Method 3: Ask for the login in front of Varnish

Methods 1 and 2 put the rules in .htaccess, which Apache reads from the document root. With Varnish enabled that document root belongs to the backend: Apache serves a second, plain virtual host that Varnish requests when it has nothing cached, while the virtual host on port 443 only passes traffic to Varnish and has no document root of its own.

Two things follow from that. The front virtual host never reads .htaccess, and a page Varnish already holds is returned without asking Apache. The login prompt is only enforced on a cache miss.

To ask for the login before the cache, put the rules in ~/apache2/outside/50vhost_post.conf instead:

<Location "/">
    AuthType Basic
    AuthName "Restricted content"
    AuthUserFile /var/www/prod/apache2/.htpasswd

    # Whitelisted IPs are granted access without a login prompt
    Require ip 203.0.113.10
    # Only a person with valid credentials is let in
    Require valid-user
</Location>

That file is read as part of the front virtual host, so the directives need a <Location> container - unlike .htaccess, which is already scoped to the directory it sits in. The front virtual host also sees the visitor's own address, so Require ip takes that address directly and the SetEnvIf from Method 2 is not needed here.

Apply the change:

tscli apache reload

Block the infamous Bytespider bot

Sometimes a server can go high in load due to the infamous Bytespider bot. This one can be excluded by adding this piece of code inside the .htaccess:

<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} Bytespider [NC]
    RewriteRule .* - [F]
</IfModule>

For broader protection against abusive traffic, prefer the Firewall over per-site rules.