Well-known files and robots.txt

• Back to Artica reverse-proxy main section

Modern browsers, mobile operating systems, password managers, calendar clients, mail servers and security researchers all expect to find small standardised files at fixed URLs on your domain: /robots.txt, /.well-known/security.txt, /.well-known/assetlinks.json, /.well-known/change-password and a dozen more. 
They are defined by RFC 8615 and its companions.

Publishing them normally means editing files on the backend web server 
A server you may not own, may not be allowed to touch, or which is a black-box appliance. 
Well-known files & robots.txt lets the reverse proxy answer those URLs itself, per website, from the Artica web console.

Concept

The feature is an editor of standard responses, not a file manager. For each website rule you enable one or more entries from a fixed catalogue of 15 resources. Artica then renders the content to disk, generates one exact per enabled entry, validates the whole configuration, and only then publishes it.

Three properties are worth understanding before you enable anything.

• The catalogue is closed. 
  You choose a resource from a list; you never type a URL. 
  It is therefore impossible to create an arbitrary path in your configuration from the console, by accident or otherwise.
• Your backend always keeps the last word. 
  A resource you leave disabled is still handed to your backend exactly as before. 
  Artica only substitutes an empty 200 when the backend answers 404 
  See No more 404 on discovery URLs below. 
  Enabling the feature never hides an endpoint your application already handles.
• Nothing runs per request. 
  Once the configuration is published, reverse-proxy serves a static file or returns a redirect. 
  There is no Lua, no PHP, no database lookup and no measurable latency on the request path.

No more 404 on discovery URLs

Crawlers, mobile operating systems and security scanners request these URLs constantly, whether or not you publish them. 
On a site that does not, every one of those requests becomes a 404 in your logs — noise that hides the 404s you actually care about.

Artica removes that noise without taking anything away from your application. 
For each resource you leave disabled, it emits a location that still proxies to your backend, and replaces only its 404 with an empty 200

• Only 404 is replaced. 
  A 502 or 504 passes through untouched: 
  Turning a backend outage into an empty 200 would hide the incident instead of reporting it. 

For the same reason the three redirect entries (change-password, caldav, carddav) get no fallback.
A client asking for them expects a Location header, and an empty 200 would tell it the resource exists and is empty.

The fallback applies to reverse-proxy websites that have a real backend. 
A static HTML site or a blackhole rule has nothing to defer to, so nothing is emitted for them.

Benefits

• Publish without touching the backend. 
  The typical case: the application is a vendor appliance, a legacy server, or a container image you rebuild from a pipeline. 
  Adding a security.txt to it is a change request; adding it here is two minutes.
   
• One place for every site. 
  Each website rule carries its own set. Nothing is shared, nothing leaks between sites, and deleting a site removes its files and its configuration.
   
• Content is validated before it is published. 
  JSON must parse, and must be an array or an object depending on the resource. 
  XML must be well formed. security.txt must carry a Contact: field. An MTA-STS policy must declare a valid version, mode and max_age. 
  Invalid content is refused at save time, with the parser's own message.
   
• A broken entry can never break the site. 
  If a generated file is missing, or a stored value looks tampered with, the entry is silently skipped — the request falls through to the backend and the rest of the configuration is untouched.
   The configuration is tested before it reaches production, so a failure leaves the previous configuration in place.
• Clean logs. 
  Discovery URLs stop producing 404s, without a single one of your application's own responses being replaced.
• A real robots.txt engine.
  Not a text box: user-agent groups, ordered directives, several sitemaps, and a preview rendered by the exact same code that writes the published file.
• Correct Content-Type, which is harder than it looks.
   /.well-known/host-meta.json must be served as application/jrd+json, not application/json.
 /.well-known/apple-app-site-association has no file extension at all. 
  Artica forces the right type for each entry.

Use cases

• Security disclosure contact (security.txt)
  Bug-bounty programmes, penetration testers and CERTs look for /.well-known/security.txt before they try to reach you any other way. 
  Publishing it takes one entry and does not involve the application team.
• Mobile application association
  Android App Links need /.well-known/assetlinks.json and Apple Universal Links need /.well-known/apple-app-site-association, 
  both served from the domain with the exact right content type. 
  These are typically requested by a mobile team that has no access to the web backend.
• Password manager integration
  Enable /.well-known/change-password as a redirect to your account page.
  Browsers and password managers will take the user straight there when they offer to rotate a leaked password.
• Calendar and address-book discovery
  CalDAV and CardDAV clients probe /.well-known/caldav and /.well-known/carddav on the domain and expect a redirect to the real service path. 
  Two redirect entries replace a backend change on a groupware appliance.
• Crawler control on an application you cannot modify
  A staging copy, an admin interface or a paginated search that crawlers should not walk: build the rules here rather than shipping a robots.txt inside the application image.
• Identity provider metadata
  Publish OpenID Connect or OAuth metadata (/.well-known/openid-configuration, /.well-known/oauth-authorization-server) 
  for a service that does not expose them itself — or override them while you migrate an issuer.

Where to find it

• Open Web services, select your website, and look for Well-known files & robots.txt in the site parameters. 
  The line shows how many resources are currently enabled.

The resource list

• Clicking the line opens the catalogue. 
  The 15 resources are grouped by purpose, each with a one-line explanation of what it does and its current state.

• ACTIVE means the reverse proxy answers that URL itself.
• INACTIVE means the request goes to your backend as before — the resource is not blocked, it is simply not handled here.

Editing a static resource

• Click any entry to edit it. 
  The URI and the Content-Type are read-only: 
  They come from the catalogue and cannot be changed, which is what guarantees the response is standards-compliant. 
  The content is edited in a syntax-highlighting editor.

Two buttons help before you commit:

• Validate 
  Checks the stored content and shows the simulated HTTP response — status, headers and body. 
  Nothing is fetched from the network.
• Reset 
  Disables the resource and deletes its stored configuration.

Cache-Control is optional. 
Left empty, no cache header is added. Do not put a long lifetime on security.txt or on OAuth metadata: you want to be able to correct them quickly.

Enabling a resource takes the URL away from your application. 
The modal says so on every entry. It matters most for openid-configuration, webfinger, assetlinks.json and robots.txt, which a modern application may already generate dynamically. 
When in doubt, request the URL first and see whether your backend already answers it.

Editing a redirect resource

Three entries are redirects rather than documents: change-password, caldav and carddav. 
They ask for a target and an HTTP status instead of content.

The target may be a local path (/account/change-password) or an absolute http:// or https:// URL. 
Anything else is refused — including any value containing a control character, which is how header-injection attempts are blocked. 
Allowed statuses are 301, 302, 303, 307 and 308; 302 is the sensible default.

The robots.txt engine

robots.txt opens its own editor with four tabs.

• Settings
  Enable the file, choose the editor mode and set an optional Cache-Control.

• Structured (default) 
  Builds the file from the groups and directives you define below.
• Raw 
  Serves exactly the text you store. 
  Your structured rules are kept, so you can switch back at any time without losing them.

User-agent groups
Each group targets one user-agent and holds its own directives. 
The Rules column shows how many it contains; the arrows change the order in which groups are rendered.

Inside a group you may use Allow, Disallow and Crawl-delay. 

Note that Crawl-delay is not universally supported — several major crawlers ignore it. 
Global directives (Sitemap, and the non-standard Host) live in their own tab and are rendered after the groups.
Several Sitemap entries are allowed.

Preview
The preview is produced by the same renderer that writes the published file, so what you read here is exactly what crawlers will receive.

The engine is deliberately neutral. 
It never adds a directive you did not configure, and never emits Disallow: / on its own. 
A group with no active rule is omitted rather than rendered as an empty User-agent block, which some parsers attach to the next group.

Good practice

• Check what your backend already serves before enabling an entry. 
  If the application generates /.well-known/openid-configuration itself, leaving the entry disabled is the right choice.
• WebFinger is served statically in this version. 
  If your application must return different answers depending on the resource or rel query parameters, leave that entry disabled.
• MTA-STS needs DNS too. 
  Publishing /.well-known/mta-sts.txt is only half the job: 
  The policy is normally fetched from the mta-sts. subdomain and requires the _mta-sts DNS record. 
  Artica does not change DNS.
• Set an Expires: date in security.txt. 
  RFC 9116 requires it for a compliant file.
  The console warns you when it is missing but still lets you save.
• Keep robots.txt honest. 
  It is a request, not an access control. 
  Never rely on Disallow to protect a private path — use authentication, or the reverse proxy URL filters.