Dangerous Paths: block reconnaissance scanners

• Back to Artica reverse-proxy main section

Every public web service is continuously scanned by automated tools looking for files that should never have been published: /.env files holding database passwords, /.git/ directories exposing the whole source tree, forgotten /backup.sql dumps, cloud credentials in /.aws/credentials, CI tokens, editor swap files.
These probes cost the attacker nothing, they are launched by the million, and a single hit is enough to compromise an application that was otherwise perfectly maintained.

Dangerous Paths watches the requests your reverse proxy is already serving, recognises those probes against a curated database of known dangerous paths, scores the source, and bans it at the firewall before it finds anything.

Concept

The engine works on the access log stream the reverse proxy produces anyway. There is no extra parsing daemon, no additional listening socket, and nothing is added to the request path — a request is examined after it has been answered, so the feature cannot slow your web sites down.

Three ideas make up the whole design:

1. A curated database of dangerous paths. 
   Artica publishes a list of paths that have no legitimate reason to be requested on a production web site — secrets, source control, backups, cloud credentials, CI configuration, administration consoles. Each entry carries a category and a score reflecting how damaging that particular file would be if it were exposed. 
   The appliance refreshes this database on its own.
2. Scoring instead of a single trigger. 
   One request for /.env is already worth 100 points, because a hit on that file is game over. 
   A request for /.gitignore is worth far less on its own — but a source asking for /.gitignore, then /.gitattributes, then /.env is unmistakably enumerating, and the accumulated score crosses the threshold. 
   Scores add up per source, inside a sliding time window.
3. A ban at the packet level. 
   When the threshold is reached, the source address is added to a dedicated firewall group, for a duration you choose. 
   The scanner stops receiving answers altogether — it does not simply get 403s, it gets nothing.

What is examined, and what is not

The status code of the answer decides what happens, and this distinction is the heart of the feature:

Benefits

• It blocks the reconnaissance phase, not the exploitation phase. 
  A Web Application Firewall inspects a request and decides whether it is malicious. 
  This engine watches a sequence of requests and decides whether the source is hostile. 
  The scanner is removed before it reaches the URL that would actually have worked.
• It costs nothing on the request path. 
  Detection happens on the log stream, after the answer has been sent. 
  Enabling it on a busy site does not add latency.
• It tells you when you are genuinely exposed. 
  The day a dangerous path answers 200, you are told — by the very traffic that was trying to find it. Few tools give you that.
• It drops packets rather than answering them. 
  Because the ban is applied by the firewall, a banned scanner consumes no worker, no backend connection and no bandwidth for the duration of the ban.
• It complements the WAF instead of duplicating it. 
  WAF inspects content; this engine judges behaviour. 
  A probe for /.env is a perfectly well-formed request that no WAF rule has any reason to reject.
• It can run without blocking anything. 
  An observation mode lets you measure exactly what would have been banned, on real traffic, before you let it act.

Use cases

• A public site under constant background scanning
  Any web site exposed to the Internet receives thousands of probes a day from botnets enumerating well-known paths. 
  Enable the feature with the default settings: a source that asks for one critical file, or accumulates several minor ones within a minute, is banned for an hour. 
  Your logs get quieter, your backends stop being woken up for nothing.
• Auditing before enforcing
  You have a legacy application and you are not sure whether legitimate clients request unusual paths. 
  Turn the detection on but leave Ban Offenders At The Firewall off. 
  The engine scores and alerts exactly as it would in blocking mode, and writes nothing to the firewall. 
  After a few days, the Alert Only Events counter and the logs tell you precisely what enforcement would have done. Then turn it on.
• Discovering an accidental exposure
  A deployment mistake copies .env next to the application root. 
  The next scanner that asks for it receives it, and the Served Dangerous Paths counter increases. 
  Nobody had to audit the file tree — the attackers did the audit and the appliance reported it.
• Protecting a site you do not control
  The reverse proxy fronts an application maintained by a third party, whose release cycle you cannot influence. 
  You cannot patch the application, but you can stop the enumeration that precedes the exploit, from the outside, without touching a line of their code.
• Coexisting with a contracted security audit
  A pentest is scheduled and the auditor's scanner must not be banned in the first thirty seconds. 
  Add the auditor's address to the whitelist for the duration of the engagement, and remove it afterwards. 
  The rest of the Internet stays protected throughout.

Where to find it

The feature is configured per web service. 
Open Web services, select your site, and look for Dangerous Paths in the Security section of the service parameters.

Status

The first tab reports what the engine is actually doing. 
It is the page to open when you want to know whether protection is real, not merely enabled.

Settings

Detection is disabled by default and is turned on per site.

Out-of-range values are refused, never silently adjusted. 
If you enter a scoring window of 5 seconds, the appliance tells you so rather than quietly applying 10 — you always know what is running.

Observation mode in practice

Turning Ban Offenders At The Firewall off is the recommended way to introduce the feature on a sensitive production site. 
The engine behaves exactly as it would in blocking mode, and every threshold crossing is written to the log in the following form:

DangerousPaths: 203.0.113.42 signalé SANS bannissement (site=78 règle=/.env catégorie=env-secrets score=100) — EnableDangerousPathsFw=0

Compare it with what a real ban looks like:

DangerousPaths: 203.0.113.42 banni 3600s (site=78 règle=/.env catégorie=env-secrets score=100)

Both lines carry the source, the site, the rule that matched, its category and the accumulated score — so a week in observation mode gives you the complete list of what enforcement would have blocked, with the reason for each decision.

One point deserves attention. 
The firewall is shared by every site on the appliance: a single set of banned addresses, a single chain, one common set of protected ports. 
Turning this setting off stops this site from arming bans; it does not exempt it from bans armed by another site sharing the same port. 
This is a deliberate design decision, not a limitation to work around — a banned scanner is hostile to every site it can reach.

Whitelist

Sources listed here are never scored and never banned, whatever they request.

Each entry combines a source address and a path:

• An empty field, or *, means all — * as the path exempts the source from everything.
• A path may end with * to cover a subtree, for example /.well-known/*.
• The source must be a single, exact IP address (or *). Ranges in CIDR notation such as 192.168.90.0/24 are not accepted: an exemption is granted to a machine you know, not to a subnet whose future occupants you do not.

Typical entries: a monitoring probe that checks unusual URLs, an ACME validation agent, a partner scanner, or a contracted penetration tester for the duration of the engagement.

Blocked IPs

The sources banned at this instant, across every site, with the time each one has left. 
The padlock on the right releases an address immediately.

Releasing an address removes it from the firewall, but the engine keeps its memory of that source until the ban duration has elapsed: 
it will not be banned twice for the same burst. If you want a released address to be permanently exempt, add it to the whitelist.

IPv6 sources are banned by their /64 prefix, not by their exact address. 
A single machine is routinely given billions of addresses inside its prefix and changes them at no cost, so banning one address would make IPv6 support purely decorative.

Database

The last tab lets you browse the path database the engine matches against. Search by path, filter by category, or both.

Each entry shows its path, its category and its score. 
This is where you check why a given source was banned, or confirm that a path you consider sensitive is covered. 
Results are capped, and the page tells you when a search returned more entries than it displays — narrow the search rather than assume you have seen everything.

How the ban is applied

Bans are enforced by a dedicated firewall chain that touches nothing else on the appliance:

• Banned addresses live in dedicated ipset sets, one for IPv4 (exact addresses) and one for IPv6 (/64 prefixes), with the expiry handled by the set itself.
• Traffic from those sets is sent to a dedicated NGINX_DANGEROUS_FS chain that drops it. 
  No rule belonging to another Artica feature is ever modified.
• Only the ports your enabled web services actually listen on are protected. 
  Administration ports — SSH, the Artica console, the reverse-proxy API — are explicitly excluded, so a misconfiguration can never lock you out of your own appliance.
• A watchdog verifies every minute that the rules are still in place, and restores them if another component flushed the firewall. 
  Restoration preserves the bans in progress.
• Disabling a site, or switching every site to observation mode, stops new rules from being installed but never destroys anything: 
  bans already in progress run to their natural expiry.

Good practice

1. Start in observation mode on production sites, for a few days, and read the log lines it produces.
2. Whitelist your own monitoring before enabling enforcement, especially probes that deliberately request unusual URLs.
3. Watch the Served Dangerous Paths counter. It is the only counter that reports a problem with your site rather than with a visitor.
4. Keep the default threshold unless you have a measured reason to change it. It is calibrated so that one critical file is enough, while a single harmless minor path is not.
5. Raise the ban duration rather than lowering the threshold if scanners come back: a longer ban costs an attacker far more than a hair-trigger that risks catching a legitimate client.