The XML Sitemap feature generates and serves a standards-compliant
sitemap.xmlfor any website published behind the Artica reverse-proxy, without touching the backend application.
Artica crawls the site directly on its backend servers, keeps only the canonical URLs that are really indexable, writes one or more Sitemap XML files, and serves them itself from the reverse-proxy.
The sitemap is regenerated on a schedule, announced inrobots.txt, and every decision the crawler took is visible in the console.The feature works per published website and is disabled by default.
It is available for reverse-proxy websites that forward to at least one backend server.
¶ Concept
A sitemap is the list of URLs a site owner wants search engines to crawl and index.
Applications such as WordPress, PrestaShop or Magento generate their own from their database.
Static sites, in-house applications and legacy sites usually have none, and their administrators have no simple way to produce one.
Artica fills that gap from the one place that already sees every published site: the reverse-proxy.
How a generation works:
- Crawl the backend, not the public site.
The crawler connects to the backend servers defined for the website and sends the site's public host name in theHostheader.
It never goes through the public listener, so it is not blocked by the Web Application Firewall, the JavaScript challenge, EdgeGuard or IP reputation rules, and it does not pollute the access logs or the statistics of the site.
No URL is ever sent to the crawler from the console:
It only receives a website identifier, so it cannot be turned into a proxy towards arbitrary hosts.
Redirections to any host other than the site's own host names are refused. - Two independent verdicts per URL.
Each page receives a follow verdict (should its links be explored?) and a publish verdict (should it be listed in the sitemap?).
A pagination page such as/page/2or anoindex,followpage is followed but not published:
This is how the crawler reaches deep articles and products without listing the pagination itself.
Merging the two verdicts, which is the classic mistake of this kind of tool, silently drops half of an e-commerce catalogue. - Canonical URLs and deduplication.
URLs are normalised (host and scheme in lowercase, fragment removed, default ports removed, tracking parameters such asutm_*,gclidandfbclidremoved) andrel=canonicalis honoured.
Two paths serving the same content are published once, under their canonical URL.
The source of an internal redirection is recorded as a duplicate of its target. - Classification.
Every published URL is classified as a page, an article or a product, from JSON-LD structured data first (Article,BlogPosting,Product…), then OpenGraph, then HTML structure.
Images embedded in a page are collected and attached to that page. - Real modification dates.
Thelastmodof each URL is taken from the most reliable source available, in this order:
JSON-LDdateModified,article:modified_time, the HTTPLast-Modifiedheader, a<time datetime>element, the date known from the previous crawl, and finally the date of the last successful scan of that URL.
The generation timestamp is never used as a universallastmod: search engines require this field to reflect an actual content change. - Generation and atomic publication.
The inventory is turned into one file per family (pages,articles,products,images) under asitemap.xmlindex.
Each run writes into its own directory; the files are validated, and only then is thecurrentlink switched to the new generation.
The previous generation keeps being served until that very instant, and stays on disk for rollback. - Served by the reverse-proxy service.
The reverse-proxy serves/sitemap.xmland the family files itself, from local disk.
The backend is never asked for them.
When the feature is disabled, the locations are removed and the backend receives/sitemap.xmlexactly as before. - Announced in robots.txt.
When the Well-known files & robots.txt feature manages the site'srobots.txt, aSitemap:
line pointing to the generated sitemap is added at rendering time, deduplicated against any line you already wrote, and removed automatically when the sitemap is disabled. - Scheduled.
A scheduler wakes up every minute, compares the end of the last successful run with the configured interval and queues the sites that are due, with a ±10 % spread so that all sites configured on the same interval do not start at the same second.
At most two crawls run at the same time on an appliance, whatever the number of sites.
¶ Benefits
- A sitemap for every site, including those that cannot produce one
Static sites, in-house applications, legacy applications, sites whose CMS plugin is broken or missing. - Zero change on the backend.
Nothing is installed on the application servers.
The site's code is not modified; the sitemap is served by the reverse-proxy from local disk, so the backend does not even receive the requests of search engines for it. - Only indexable URLs.
noindex,robots.txtdisallows, error pages, soft 404s, internal search results, pagination, duplicates and external canonicals are filtered out, so the sitemap does not send crawlers to pages that hurt the site's ranking. - Meaningful
lastmoddates,
taken from the content itself rather than from the generation time, which is what search engines actually use to prioritise re-crawls. - Nothing is ever broken by a backend incident.
A crawl that fails, is interrupted, or finds zero URL because the backend was down never replaces a healthy sitemap.
The previous generation keeps being served. - Full transparency.
Every discovered URL is listed with its verdict and, when excluded, the reason.
An administrator can always answer the question "why is this page not in my sitemap?". - Polite and bounded.
Concurrency, depth, page count, request timeout and page size are configurable per site.
The crawler backs off on429and503answers and never retries aggressively.
¶ Use cases
- Static or hand-made websites.
(HTML, documentation portals, showcase sites) that have no engine capable of generating a sitemap. - In-house web applications
whose development team has no time to add SEO features:
The reverse-proxy administrator can publish a sitemap without a single deployment on the application side. - Legacy applications
that nobody can modify any more but which still carry public content that must stay indexed. - E-commerce catalogues
where products and articles are reachable only through paginated listings:
The crawler follows pagination without publishing it, classifies products and articles separately, and produces an image sitemap for product pictures. - Sites migrated behind Artica
where the historical sitemap plugin was lost or references the old host name:
Artica generates a fresh one on the canonical host. - SEO audits.
Even without publishing the sitemap, the URLs tab gives an inventory of what a crawler really sees on the backend, with HTTP status, content type and exclusion reasons, which is a cheap way to spotnoindexmistakes, redirect chains and duplicate content. - Replacing a poor application sitemap.
When the application does publish a sitemap but a bad one (stale, incomplete, wrong host), the administrator can knowingly take over with the Override existing sitemap option.
¶ Where to find it
Open Web services, select the website, and locate the XML Sitemap row in the site's parameters.
The row shows whether the feature is active and, when it is, the number of indexable URLs found by the last successful run.
Click the edit button to open the management window.
The management window has five tabs: Overview, Settings, URLs, Exclusions and History.
¶ Enabling the feature and settings
Open the Settings tab, switch Enable XML Sitemap on, adjust the parameters if needed and click Apply.
| Setting | Default | Range | Meaning |
|---|---|---|---|
| Enable XML Sitemap | Off | Activates the crawler, the scheduler and the reverse locations for this site. | |
| Override Existing Sitemap | Off | One-shot confirmation used when the backend already serves its own /sitemap.xml, or when Artica could not check it (see below). It is not stored. |
|
| Generation Interval (Hours) | 12 | 1 – 720 | Time between two scheduled generations, counted from the end of the last successful run. |
| Maximum Pages | 10 000 | 10 – 1 000 000 | Maximum number of pages fetched in one run. When reached, the run ends with the statussuccess_with_limits and the URLs that were discovered but never fetched are countedunder the crawl_limit reason. |
| Maximum Crawl Depth | 15 | 1 – 100 | Maximum number of links between the home page and a page. Deeper URLs are counted under crawl_depth. |
| Concurrency | 4 | 1 – 32 | Number of simultaneous requests sent to the backend. |
| Request Timeout (Seconds) | 15 | 2 – 120 | Timeout of each backend request. |
| Maximum Page Size (Bytes) | 8 388 608 | 262 144 – 33 554 432 | Pages larger than this are not parsed and are excluded with the body_too_large reason. |
| Respect Robots.Txt | On | Reads the site's robots.txt from the backend and excludes disallowed pathsfrom both the crawl and the sitemap ( robots_disallow). |
|
| Include Images | On | Collects the images of each published page and produces an image sitemap. | |
| Include Articles | On | Publishes the URLs classified as articles, in their own file. | |
| Include Products | On | Publishes the URLs classified as products, in their own file. | |
| Exclude Query Strings | Off | When on, any URL carrying a query string is excluded (query_string).Leave it off on sites that use parameters as legitimate page identifiers. |
Values outside the allowed range are refused with an explicit error, never silently clamped.
¶ What happens when you enable it
Enabling the feature triggers two checks before anything is written:
- The backend is probed for an existing sitemap.
Artica requests/sitemap.xmlon the backend.
Three outcomes are possible:- the backend answers
404or anything that is not an XML sitemap: activation proceeds; - the backend already serves an XML sitemap:
activation is refused with the message the backend already publishes its own sitemap.xml.
Taking over would silently hide an application sitemap that is often richer than a crawl.
Switch Override Existing Sitemap on and apply again to take over deliberately; - the backend could not be reached within 10 seconds:
activation is refused with a distinct message, could not verify.
A check that did not happen is never treated as "no sitemap found".
The same override option lets you proceed.
- the backend answers
- The website configuration is rebuilt and validated synchronously.
The reverse-proxy configuration of the site is regenerated with the sitemap locations, tested, and deployed.
If the test fails, the previous configuration stays in place.
Activation is also refused, without any override, when the site type does not support it, when its configuration cannot be read, or when/sitemap.xmlis already served by a custom content file of the same site.
After a successful activation, the first generation starts at the next scheduler tick (within a minute).
You can also start it immediately with Generate now in the Overview tab.
Disabling the feature removes the everse-proxy locations, so the backend receives /sitemap.xml again as before.
The generated files stay on disk for the history but are no longer served.
¶ Overview tab
The Overview shows the state of the feature for this site and the result of the last successful run.
| Field | Meaning |
|---|---|
| State | Idle, or Running while a crawl is in progress for this site. |
| Last Run | Start time of the last successful run. Failed or interrupted runs are listed in the History tab but never shown here. |
| Indexable URLs / Excluded URLs | Number of distinct URLs of the last successful run that were published, and number that were excluded. Together they equal the size of the inventory shown in the URLs tab. |
| Pages Fetched | Number of HTTP requests actually sent to the backend during the run. |
| Errors | Number of URLs excluded because of a network error. |
| Next Run | Nominal date of the next scheduled generation (last successful run plus the interval). The real start is spread by up to 10 % to avoid all sites starting together. The field is absent until a first successful run exists. |
| Published URLs | Total number of entries in the published files. A page that carries images appears both in its family file and in the image sitemap, so this number can exceed the number of indexable URLs. |
| Top 5 exclusion reasons | The most frequent reasons of the last run. See the URLs tab for the full list. |
Generate now starts a manual run immediately.
The console then polls the daemon every three seconds and refreshes the tab when the run ends.
A second click while a run is in progress is refused (A generation is already running for this site), as is a manual run when the appliance-wide limit of two simultaneous crawls is reached.
¶ URLs tab
The URLs tab is the inventory built by the crawler:
every URL discovered on the site, with its content type, HTTP status, verdict and, when excluded, the reason.
It is paginated on the server and can be filtered by path, verdict, content type, exclusion reason and HTTP status.
The exclusion reasons form a closed list:
| Reason | The URL was excluded because… |
|---|---|
noindex |
the page carries noindex in a robots or googlebot meta tag or in the X-Robots-Tag header. Its links are still followed unless nofollow is present. |
robots_disallow |
the path is disallowed by the site's robots.txt (when Respect Robots.Txt is on). |
non_200 |
the backend answered something other than 200 (404, 403, 5xx…). |
redirect_external |
the page redirects to a host that is not one of the site's host names. The redirection is not followed. |
canonical_external |
its rel=canonical points outside the site. |
duplicate |
another URL is the canonical form of the same content: explicit canonical, source of an internal redirection, or identical title, H1 and visible text. |
search |
it is an internal search results page. |
pagination |
it is a pagination page (/page/2, ?page=2…). It is followed, so that deep content is discovered, but not published. |
low_value |
it is a filter, sort, cart, checkout or similar variant with no indexing value. |
unsupported_content_type |
the response is not HTML (PDF, JSON, image, archive…). |
body_too_large |
the page exceeds Maximum Page Size. |
soft_404 |
the page answers 200 but is recognised as a "not found" page. |
query_string |
it carries a query string and Exclude Query Strings is on. |
custom_exclusion |
it matches one of the rules of the Exclusions tab. |
invalid_url |
the URL could not be parsed or normalised. |
network_error |
the backend request failed (timeout, connection error). |
crawl_depth |
it was discovered deeper than Maximum Crawl Depth and was never fetched. Counted in the run, but absent from the table since it has no inventory line. |
crawl_limit |
it was discovered after Maximum Pages was reached and was never fetched. Same remark as above. |
Weak heuristics never exclude a page on their own:
an article whose slug contains the word search is not treated as a search results page.
Detection combines the exact path, its segments, the query parameters and the HTML signals.
¶ Exclusions tab
Exclusion rules let you keep URLs out of the sitemap even though the crawler would publish them:
a private area, a staging section, a family of parameters.
A rule only affects the publish verdict; the crawler still follows the links of an excluded page, so excluding a listing page does not hide what it links to.
Click Add exclusion to create a rule, or the pencil icon to edit one.
| Match type | Matches when… | Example |
|---|---|---|
exact |
the URL path is exactly the pattern. | /private-page excludes /private-page only, not /private-pages. |
prefix |
the URL path starts with the pattern. | /prive/ excludes the whole /prive/ tree. |
contains |
the URL path contains the pattern. | /preview excludes any path containing that word. |
query_param |
the URL carries a query parameter with that name. | sessionid excludes every URL with a sessionid= parameter. |
Rules are matched on the decoded path and parameters, never on the raw URL string.
A rule can be disabled without being deleted.
Rules take effect at the next run, scheduled or started with Generate now;
the excluded URLs then appear in the URLs tab with the custom_exclusion reason.
¶ History tab
The History tab lists the last runs of this site, most recent first, with their trigger (manual or scheduled), status, counters, duration, and whether the run was published.
| Status | Meaning | Published |
|---|---|---|
success |
The crawl completed normally. | Yes |
success_with_limits |
The crawl completed but hit Maximum Pages, or met isolated timeouts or 5xx answers. The sitemap is published; review the crawl_limit and network_error counters. |
Yes |
failed |
The home page did not answer, no backend was reachable, the generated XML did not validate, or a file-system error occurred. |
No — the previous generation stays served |
interrupted |
The run was cancelled or the daemon was stopped while it was running. | No |
The last two generations are kept on disk; older ones are deleted after each successful publication.
¶ What is published
The reverse-proxy serves the following URLs for the site, from local disk, with the security modules that could block a search engine crawler disabled on these paths:
/sitemap.xml— a sitemap index when several families or several files exist, otherwise a single URL set;/sitemap-pages-NNNN.xml,/sitemap-articles-NNNN.xml,/sitemap-products-NNNN.xml,/sitemap-images-NNNN.xml— the family files, numbered from0001, split when a file would exceed the protocol limits.
Example of the index generated for the site shown above:
<?xml version="1.0" encoding="UTF-8"?><sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <sitemap><loc>http://sitemap-test.articatech.unix/sitemap-pages-0001.xml</loc><lastmod>2026-09-02T13:57:45Z</lastmod></sitemap> <sitemap><loc>http://sitemap-test.articatech.unix/sitemap-articles-0001.xml</loc><lastmod>2026-09-02T13:57:45Z</lastmod></sitemap> <sitemap><loc>http://sitemap-test.articatech.unix/sitemap-products-0001.xml</loc><lastmod>2026-09-02T13:57:45Z</lastmod></sitemap> <sitemap><loc>http://sitemap-test.articatech.unix/sitemap-images-0001.xml</loc><lastmod>2026-09-02T13:57:45Z</lastmod></sitemap></sitemapindex>
A family file, with the lastmod taken from the page content:
<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url><loc>http://sitemap-test.articatech.unix/</loc><lastmod>2026-08-25T08:00:00Z</lastmod></url> <url><loc>http://sitemap-test.articatech.unix/product-1</loc><lastmod>2026-09-02T15:52:48+02:00</lastmod></url></urlset>
In the image sitemap, the <loc> of each entry is the page, and its images are listed inside it, as the protocol requires:
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1"> <url> <loc>http://sitemap-test.articatech.unix/product-1</loc> <image:image><image:loc>http://sitemap-test.articatech.unix/img/product-1.jpg</image:loc></image:image> </url></urlset>
When robots.txt is managed by Artica for this site, it ends with the announcement of the generated sitemap:
User-agent: *Disallow: /prive/Sitemap: http://sitemap-test.articatech.unix/sitemap.xml
URLs are built on the site's canonical host name and public scheme (HTTPS when the site has an SSL listener), never on the backend address.
¶ Safety rules worth knowing
- An empty or failed crawl never empties a healthy sitemap.
A backend that is down for a few hours produces a run with zero URL;
publishing it would remove every URL of the site from the search engines' index.
Such a run is not published and the previous generation keeps being served.
A genuinely empty site, whose home page answers and whose crawl completes normally, is published as is. - Publication is atomic.
The new generation is written next to the old one and the served link is switched in one operation,
after the XML has been validated.
Search engines never see a half-written file. - The crawler sees the backend before the reverse-proxy rules.
URL filters, rewrites, custom locations and caches configured on the site are not applied to the crawl.
In rare cases the sitemap can list a URL that the public site rewrites or refuses; add an exclusion rule for it.
Crawling the public listener instead would expose the crawler to the site's own WAF and anti-bot challenges. - The crawler is polite.
It identifies itself with its own User-Agent, respectsrobots.txtwhen asked to, limits concurrency per site,
slows down on429and503and honoursRetry-After. - The appliance limits itself.
No more than two crawls run at the same time, all sites included;
the others wait for the next scheduler tick. - Stopping the daemon cancels running crawls without publishing anything; they appear as
interruptedin the History.
¶ REST API
The feature is driven by the reverse-proxy daemon's REST API on the local Unix socket /run/reverse-proxy.sock. All routes are scoped by the website identifier.
| Route | Purpose |
|---|---|
GET /sitemap-xml/{serviceid}/settings |
The twelve settings as resolved for this site. |
POST /sitemap-xml/{serviceid}/settings |
Writes settings. Enabling probes the backend; the answer carries rebuild_ok and rebuild_error. Pass "force_replace": true (a JSON boolean) to override a conflict. |
POST /sitemap-xml/{serviceid}/generate |
Starts a manual run and returns its run_id immediately; 409 already_running or 503 no_slots_available otherwise. |
POST /sitemap-xml/{serviceid}/cancel |
Cooperative cancellation of the running crawl. |
GET /sitemap-xml/{serviceid}/status |
scheduler_running (the module started on this appliance), enabled (this site), running (a crawl is in flight), last successful run, next_run_at, published manifest. |
GET /sitemap-xml/{serviceid}/runs |
Run history, most recent first. |
GET /sitemap-xml/{serviceid}/runs/{runid} |
One run with its exclusion counters. |
GET /sitemap-xml/{serviceid}/urls |
Filterable, paginated inventory (q, indexable, content_type, reason, status, offset, limit up to 500). |
GET / POST /sitemap-xml/{serviceid}/exclusions |
List, create or update exclusion rules. |
POST /sitemap-xml/{serviceid}/exclusions/{id}/delete |
Removes a rule. |
Example from the appliance shell:
curl -s --unix-socket /run/reverse-proxy.sock http://x/sitemap-xml/90/statuscurl -s -X POST --unix-socket /run/reverse-proxy.sock http://x/sitemap-xml/90/generate
¶ Troubleshooting
| Symptom | Cause and action |
|---|---|
| Activation refused: the backend already publishes its own sitemap.xml | The application serves a sitemap. Keep it if it is good; otherwise enable Override Existing Sitemap and apply again. |
| Activation refused: could not verify | The backend did not answer the probe within 10 seconds. Check the backend's health (Backend SLA, backend diagnostic) before overriding. |
| Activation refused: URI already served | /sitemap.xml is defined as a custom content file in the site's directories. Remove that entry first. |
| Enabled, but no run ever appears | Check scheduler_running in the status route: when the module could not open its database at daemon start, nothing is generated for any site. Restart the reverse-proxy daemon and check its log. |
Runs end in failed |
The home page did not answer 200 on the backend, or no backend was reachable. Test the backend from the appliance with the site's host name in the Host header. |
| Zero or very few indexable URLs | Look at the Top 5 exclusion reasons and filter the URLs tab by reason. Typical causes: a site-wide noindex on a staging backend, a robots.txt that disallows everything, every page redirecting to another host (redirect_external). |
success_with_limits on every run |
Raise Maximum Pages or Maximum Crawl Depth, or add exclusion rules for the parameter variants that inflate the crawl. |
/sitemap.xml returns 404 on the public site |
No generation has been published yet (first run still to come, or every run failed). The location exists but the file does not. |