Protecting apps from unnecessary static files handling

Recently I came across a Twitter/X post from Povilas Korop where he shows the benefits of nightwatch and how it shows excessive bot traffic.

Most of these evil bots are trying to access static files like plugin.xml, wp-login.php, phpMyAdmin-2.6.0 and so on. The default nginx config for Laravel tries first to serve static files, and if not found, it passes the request to the app. This means that every time a bot tries to access a non-existing static file, the app handles the request, which is unnecessary load, as seen in the post from Povilas above.

To protect my apps from most of these unnecessary requests, I want to make some modifications to my Varnish Cache config (read further for necessary nginx config changes) with these rules:

Treat all requests with a ”.” (dot) in the URI as static files
Treat all access to some specific directories like static files (like /assets, /build, /storage or /vendor)
Don’t allow access to static files that start with a dot (like .env, .htaccess, etc.)
Make an exception to allow access to .well-known files (for Let’s Encrypt and other services)
Allow for some other exceptions like dynamic sitemap.xml file.

This works well for almost all of my apps since I know all routes and can tell that almost none of them should have a dot in the URI, with some exceptions.

Varnish Config Modifications

So here is the Varnish Cache config I’ve come up with, with some additional comments for better understanding:

sub vcl_recv {
  # check for static files access
  call custom_handle_static_files;

  # at this point the request is routed to the app
  return (pass);
}

sub custom_handle_static_files {
  # remove query string for path checking,
  # because otherwise a request with an email address
  # or another url in it would be treated as a static file
  set req.http.x-path = regsuball(req.url, "(\?|&).*$", "");

  # If there is a dot in the path or
  # if the request is for some specific directories,
  # treat it as a static file request, set the nginx host
  # to staticfiles and remove unnecessary headers for better caching
  if (req.http.x-path ~ "\." || req.url ~ "^/(assets|build|media|storage|vendor)/") {
    set req.http.host = "staticfiles";
    unset req.http.https;
    unset req.http.x-forwarded-proto;
    unset req.http.cookie;
  }

  # Allow for some exceptions to be passed right away to nginx
  if (req.url ~ "^/\.well-known/") {
    return (pass);
  }

  # Access to files starting with a dot
  # is forbidden, as well as some other extensions
  # The reason for the 429 status code (Too Many Requests)
  # is to let evil bots think they are being rate limited,
  # so they might stop trying. Most of the evil bots stop
  # right there to save resources
  if (req.url ~ "(?i)(/\.|\.php|\.htm|\.phtm|\.cgi|\.asp|wp-)") {
    return (synth(429));
  }

  # Large static files are delivered
  # directly to the end-user without caching
  if (req.http.x-path ~ "\.(7z|avi|bz2|flac|flv|gz|mka|mkv|mov|mp3|mp4|mpeg|mpg|ogg|ogm|opus|rar|tar|tgz|tbz|txz|wav|webm|xz|zip|pdf)$" || req.url ~ "^/(media|storage)/") {
    return (pass);
  }

  # At this point we know for sure it's a possible static file,
  # so we can cache it, wether the file exists or not
  if (req.http.x-path ~ "\.") {
    return (hash);
  }
}

Nginx Config Modifications

To make this work, I had to modify my nginx config a bit, to add a new server block for the staticfiles host:

server {
    server_name staticfiles;

    root /var/www/html/public;

    access_log  off;
    error_log /dev/null;
    autoindex off;

    charset utf-8;
    index index.html;

    # Block access to miscellaneous protected files
    location ~* /.*\.(?:bak|co?nf|cfg|php|ya?ml|ts|dist|fla|in[ci]|log|sh|sql|sqlite)$ {
        deny all;
    }

    error_page 404 /404.html;
    location = /404.html {
        root /var/www/html;
        internal;
    }

    location / {
        try_files $uri $uri/ =404;
    }
}

In this config I do the following:

Don’t log anything since I don’t need to know about requests for static files
Block access to some more protected files that might be present in the app root
Serve a custom 404 page if the static file is not found
Try to serve the static file directly, or a directory with an index.html file, otherwise return a 404 error

This has reduced the load on my apps significantly and also reduced the error logs in my apps and my monitoring tools, which makes it easier to spot real issues.

No Varnish Cache, just Nginx?

I’ve found a blog post about doing almost all of this work in just nginx, but I prefer to do it in Varnish Cache since I use it in front of all my apps: https://www.getpagespeed.com/server-setup/nginx-try_files-is-evil-too.