Modern browsers, mobile operating systems, password managers, calendar clients, mail servers and security researchers all expect to find small standardised files at fixed URLs on your domain:
/robots.txt,/.well-known/security.txt,/.well-known/assetlinks.json,/.well-known/change-passwordand a dozen more.
They are defined by RFC 8615 and its companions.Publishing them normally means editing files on the backend web server
A server you may not own, may not be allowed to touch, or which is a black-box appliance.
Well-known files & robots.txt lets the reverse proxy answer those URLs itself, per website, from the Artica web console.
¶ Concept
The feature is an editor of standard responses, not a file manager. For each website rule you enable one or more entries from a fixed catalogue of 15 resources. Artica then renders the content to disk, generates one exact per enabled entry, validates the whole configuration, and only then publishes it.
Three properties are worth understanding before you enable anything.
- The catalogue is closed.
You choose a resource from a list; you never type a URL.
It is therefore impossible to create an arbitrary path in your configuration from the console, by accident or otherwise. - Your backend always keeps the last word.
A resource you leave disabled is still handed to your backend exactly as before.
Artica only substitutes an empty200when the backend answers404
See No more 404 on discovery URLs below.
Enabling the feature never hides an endpoint your application already handles. - Nothing runs per request.
Once the configuration is published, reverse-proxy serves a static file or returns a redirect.
There is no Lua, no PHP, no database lookup and no measurable latency on the request path.
¶ No more 404 on discovery URLs
Crawlers, mobile operating systems and security scanners request these URLs constantly, whether or not you publish them.
On a site that does not, every one of those requests becomes a 404 in your logs — noise that hides the 404s you actually care about.
Artica removes that noise without taking anything away from your application.
For each resource you leave disabled, it emits a location that still proxies to your backend, and replaces only its 404 with an empty 200
| Situation | What the client receives |
|---|---|
| The resource is enabled in Artica | The document you configured |
| Disabled, and your backend serves the URL | Your backend's own answer, untouched |
| Disabled, and your backend answers 404 | 200 with an empty body |
| Your backend is down (502, 504) | The error, unchanged |
- Only
404is replaced.
A502or504passes through untouched:
Turning a backend outage into an empty200would hide the incident instead of reporting it.
For the same reason the three redirect entries (change-password, caldav, carddav) get no fallback.
A client asking for them expects a Location header, and an empty 200 would tell it the resource exists and is empty.
The fallback applies to reverse-proxy websites that have a real backend.
A static HTML site or a blackhole rule has nothing to defer to, so nothing is emitted for them.
¶ Benefits
- Publish without touching the backend.
The typical case: the application is a vendor appliance, a legacy server, or a container image you rebuild from a pipeline.
Adding asecurity.txtto it is a change request; adding it here is two minutes.
- One place for every site.
Each website rule carries its own set. Nothing is shared, nothing leaks between sites, and deleting a site removes its files and its configuration.
- Content is validated before it is published.
JSON must parse, and must be an array or an object depending on the resource.
XML must be well formed.security.txtmust carry aContact:field. An MTA-STS policy must declare a validversion,modeandmax_age.
Invalid content is refused at save time, with the parser's own message.
- A broken entry can never break the site.
If a generated file is missing, or a stored value looks tampered with, the entry is silently skipped — the request falls through to the backend and the rest of the configuration is untouched.
The configuration is tested before it reaches production, so a failure leaves the previous configuration in place. - Clean logs.
Discovery URLs stop producing 404s, without a single one of your application's own responses being replaced. - A real robots.txt engine.
Not a text box: user-agent groups, ordered directives, several sitemaps, and a preview rendered by the exact same code that writes the published file. - Correct
Content-Type, which is harder than it looks.
/.well-known/host-meta.jsonmust be served asapplication/jrd+json, notapplication/json.
/.well-known/apple-app-site-associationhas no file extension at all.
Artica forces the right type for each entry.
¶ Use cases
- Security disclosure contact (security.txt)
Bug-bounty programmes, penetration testers and CERTs look for/.well-known/security.txtbefore they try to reach you any other way.
Publishing it takes one entry and does not involve the application team. - Mobile application association
Android App Links need/.well-known/assetlinks.jsonand Apple Universal Links need/.well-known/apple-app-site-association,
both served from the domain with the exact right content type.
These are typically requested by a mobile team that has no access to the web backend. - Password manager integration
Enable/.well-known/change-passwordas a redirect to your account page.
Browsers and password managers will take the user straight there when they offer to rotate a leaked password. - Calendar and address-book discovery
CalDAV and CardDAV clients probe/.well-known/caldavand/.well-known/carddavon the domain and expect a redirect to the real service path.
Two redirect entries replace a backend change on a groupware appliance. - Crawler control on an application you cannot modify
A staging copy, an admin interface or a paginated search that crawlers should not walk: build the rules here rather than shipping arobots.txtinside the application image. - Identity provider metadata
Publish OpenID Connect or OAuth metadata (/.well-known/openid-configuration,/.well-known/oauth-authorization-server)
for a service that does not expose them itself — or override them while you migrate an issuer.
¶ Where to find it
- Open Web services, select your website, and look for Well-known files & robots.txt in the site parameters.
The line shows how many resources are currently enabled.
¶ The resource list
- Clicking the line opens the catalogue.
The 15 resources are grouped by purpose, each with a one-line explanation of what it does and its current state.
- ACTIVE means the reverse proxy answers that URL itself.
- INACTIVE means the request goes to your backend as before — the resource is not blocked, it is simply not handled here.
¶ Editing a static resource
- Click any entry to edit it.
The URI and theContent-Typeare read-only:
They come from the catalogue and cannot be changed, which is what guarantees the response is standards-compliant.
The content is edited in a syntax-highlighting editor.
Two buttons help before you commit:
- Validate
Checks the stored content and shows the simulated HTTP response — status, headers and body.
Nothing is fetched from the network. - Reset
Disables the resource and deletes its stored configuration.
Cache-Controlis optional.
Left empty, no cache header is added. Do not put a long lifetime onsecurity.txtor on OAuth metadata: you want to be able to correct them quickly.
Enabling a resource takes the URL away from your application.
The modal says so on every entry. It matters most for openid-configuration, webfinger, assetlinks.json and robots.txt, which a modern application may already generate dynamically.
When in doubt, request the URL first and see whether your backend already answers it.
¶ Editing a redirect resource
Three entries are redirects rather than documents: change-password, caldav and carddav.
They ask for a target and an HTTP status instead of content.
The target may be a local path (/account/change-password) or an absolute http:// or https:// URL.
Anything else is refused — including any value containing a control character, which is how header-injection attempts are blocked.
Allowed statuses are 301, 302, 303, 307 and 308; 302 is the sensible default.
¶ The robots.txt engine
robots.txt opens its own editor with four tabs.
- Settings
Enable the file, choose the editor mode and set an optionalCache-Control.
- Structured (default)
Builds the file from the groups and directives you define below. - Raw
Serves exactly the text you store.
Your structured rules are kept, so you can switch back at any time without losing them.
User-agent groups
Each group targets one user-agent and holds its own directives.
The Rules column shows how many it contains; the arrows change the order in which groups are rendered.
Inside a group you may use Allow, Disallow and Crawl-delay.
Note that
Crawl-delayis not universally supported — several major crawlers ignore it.
Global directives (Sitemap, and the non-standardHost) live in their own tab and are rendered after the groups.
SeveralSitemapentries are allowed.
Preview
The preview is produced by the same renderer that writes the published file, so what you read here is exactly what crawlers will receive.
The engine is deliberately neutral.
It never adds a directive you did not configure, and never emits Disallow: / on its own.
A group with no active rule is omitted rather than rendered as an empty User-agent block, which some parsers attach to the next group.
¶ Good practice
- Check what your backend already serves before enabling an entry.
If the application generates/.well-known/openid-configurationitself, leaving the entry disabled is the right choice. - WebFinger is served statically in this version.
If your application must return different answers depending on theresourceorrelquery parameters, leave that entry disabled. - MTA-STS needs DNS too.
Publishing/.well-known/mta-sts.txtis only half the job:
The policy is normally fetched from themta-sts.subdomain and requires the_mta-stsDNS record.
Artica does not change DNS. - Set an
Expires:date in security.txt.
RFC 9116 requires it for a compliant file.
The console warns you when it is missing but still lets you save. - Keep robots.txt honest.
It is a request, not an access control.
Never rely onDisallowto protect a private path — use authentication, or the reverse proxy URL filters.