XML Sitemap generator

• Back to Artica reverse-proxy main section

The XML Sitemap feature generates and serves a standards-compliant sitemap.xml for any website published behind the Artica reverse-proxy, without touching the backend application. 
Artica crawls the site directly on its backend servers, keeps only the canonical URLs that are really indexable, writes one or more Sitemap XML files, and serves them itself from the reverse-proxy. 
The sitemap is regenerated on a schedule, announced in robots.txt, and every decision the crawler took is visible in the console.

The feature works per published website and is disabled by default. 
It is available for reverse-proxy websites that forward to at least one backend server.

Concept

A sitemap is the list of URLs a site owner wants search engines to crawl and index. 
Applications such as WordPress, PrestaShop or Magento generate their own from their database.
Static sites, in-house applications and legacy sites usually have none, and their administrators have no simple way to produce one. 
Artica fills that gap from the one place that already sees every published site: the reverse-proxy.

How a generation works:

1. Crawl the backend, not the public site. 
   The crawler connects to the backend servers defined for the website and sends the site's public host name in the Host header. 
   It never goes through the public listener, so it is not blocked by the Web Application Firewall, the JavaScript challenge, EdgeGuard or IP reputation rules, and it does not pollute the access logs or the statistics of the site. 
   No URL is ever sent to the crawler from the console: 
   It only receives a website identifier, so it cannot be turned into a proxy towards arbitrary hosts. 
   Redirections to any host other than the site's own host names are refused.
2. Two independent verdicts per URL. 
   Each page receives a follow verdict (should its links be explored?) and a publish verdict (should it be listed in the sitemap?). 
   A pagination page such as /page/2 or a noindex,follow page is followed but not published: 
   This is how the crawler reaches deep articles and products without listing the pagination itself. 
   Merging the two verdicts, which is the classic mistake of this kind of tool, silently drops half of an e-commerce catalogue.
3. Canonical URLs and deduplication. 
   URLs are normalised (host and scheme in lowercase, fragment removed, default ports removed, tracking parameters such as utm_*, gclid and fbclid removed) and rel=canonical is honoured.
    Two paths serving the same content are published once, under their canonical URL.
   The source of an internal redirection is recorded as a duplicate of its target.
4. Classification. 
   Every published URL is classified as a page, an article or a product, from JSON-LD structured data first (Article, BlogPosting, Product…), then OpenGraph, then HTML structure. 
   Images embedded in a page are collected and attached to that page.
5. Real modification dates. 
   The lastmod of each URL is taken from the most reliable source available, in this order: 
   JSON-LD dateModified, article:modified_time, the HTTP Last-Modified header, a <time datetime> element, the date known from the previous crawl, and finally the date of the last successful scan of that URL. 
   The generation timestamp is never used as a universal lastmod: search engines require this field to reflect an actual content change.
6. Generation and atomic publication. 
   The inventory is turned into one file per family (pages, articles, products, images) under a sitemap.xml index. 
   Each run writes into its own directory; the files are validated, and only then is the current link switched to the new generation. 
   The previous generation keeps being served until that very instant, and stays on disk for rollback.
7. Served by the reverse-proxy service. 
   The reverse-proxy serves /sitemap.xml and the family files itself, from local disk. 
   The backend is never asked for them. 
   When the feature is disabled, the locations are removed and the backend receives /sitemap.xml exactly as before.
8. Announced in robots.txt. 
   When the Well-known files & robots.txt feature manages the site's robots.txt, a Sitemap: 
   line pointing to the generated sitemap is added at rendering time, deduplicated against any line you already wrote, and removed automatically when the sitemap is disabled.
9. Scheduled. 
   A scheduler wakes up every minute, compares the end of the last successful run with the configured interval and queues the sites that are due, with a ±10 % spread so that all sites configured on the same interval do not start at the same second. 
   At most two crawls run at the same time on an appliance, whatever the number of sites.

Benefits

• A sitemap for every site, including those that cannot produce one 
  Static sites, in-house applications, legacy applications, sites whose CMS plugin is broken or missing.
• Zero change on the backend. 
  Nothing is installed on the application servers.
  The site's code is not modified; the sitemap is served by the reverse-proxy from local disk, so the backend does not even receive the requests of search engines for it.
• Only indexable URLs.
   noindex, robots.txt disallows, error pages, soft 404s, internal search results, pagination, duplicates and external canonicals are filtered out, so the sitemap does not send crawlers to pages that hurt the site's ranking.
• Meaningful lastmod dates, 
  taken from the content itself rather than from the generation time, which is what search engines actually use to prioritise re-crawls.
• Nothing is ever broken by a backend incident. 
  A crawl that fails, is interrupted, or finds zero URL because the backend was down never replaces a healthy sitemap.
   The previous generation keeps being served.
• Full transparency. 
  Every discovered URL is listed with its verdict and, when excluded, the reason. 
  An administrator can always answer the question "why is this page not in my sitemap?".
• Polite and bounded. 
  Concurrency, depth, page count, request timeout and page size are configurable per site.
  The crawler backs off on 429 and 503 answers and never retries aggressively.

Use cases

• Static or hand-made websites.
   (HTML, documentation portals, showcase sites) that have no engine capable of generating a sitemap.
• In-house web applications
   whose development team has no time to add SEO features:
  The reverse-proxy administrator can publish a sitemap without a single deployment on the application side.
• Legacy applications 
  that nobody can modify any more but which still carry public content that must stay indexed.
• E-commerce catalogues 
  where products and articles are reachable only through paginated listings: 
  The crawler follows pagination without publishing it, classifies products and articles separately, and produces an image sitemap for product pictures.
• Sites migrated behind Artica 
  where the historical sitemap plugin was lost or references the old host name:
   Artica generates a fresh one on the canonical host.
• SEO audits. 
  Even without publishing the sitemap, the URLs tab gives an inventory of what a crawler really sees on the backend, with HTTP status, content type and exclusion reasons, which is a cheap way to spot noindex mistakes, redirect chains and duplicate content.
• Replacing a poor application sitemap.
   When the application does publish a sitemap but a bad one (stale, incomplete, wrong host), the administrator can knowingly take over with the Override existing sitemap option.

Where to find it

Open Web services, select the website, and locate the XML Sitemap row in the site's parameters. 
The row shows whether the feature is active and, when it is, the number of indexable URLs found by the last successful run. 
Click the edit button to open the management window.

The management window has five tabs: Overview, Settings, URLs, Exclusions and History.

Enabling the feature and settings

Open the Settings tab, switch Enable XML Sitemap on, adjust the parameters if needed and click Apply.

Values outside the allowed range are refused with an explicit error, never silently clamped.

What happens when you enable it

Enabling the feature triggers two checks before anything is written:

1. The backend is probed for an existing sitemap. 
   Artica requests /sitemap.xml on the backend. 
   Three outcomes are possible:
   • the backend answers 404 or anything that is not an XML sitemap: activation proceeds;
   • the backend already serves an XML sitemap: 
     activation is refused with the message the backend already publishes its own sitemap.xml. 
     Taking over would silently hide an application sitemap that is often richer than a crawl. 
     Switch Override Existing Sitemap on and apply again to take over deliberately;
   • the backend could not be reached within 10 seconds: 
     activation is refused with a distinct message, could not verify. 
     A check that did not happen is never treated as "no sitemap found". 
     The same override option lets you proceed.
2. The website configuration is rebuilt and validated synchronously. 
   The reverse-proxy configuration of the site is regenerated with the sitemap locations, tested, and deployed.
    If the test fails, the previous configuration stays in place. 
   Activation is also refused, without any override, when the site type does not support it, when its configuration cannot be read, or when /sitemap.xml is already served by a custom content file of the same site.

After a successful activation, the first generation starts at the next scheduler tick (within a minute). 
You can also start it immediately with Generate now in the Overview tab.

Disabling the feature removes the everse-proxy locations, so the backend receives /sitemap.xml again as before.
 The generated files stay on disk for the history but are no longer served.

Overview tab

The Overview shows the state of the feature for this site and the result of the last successful run.

Generate now starts a manual run immediately. 
The console then polls the daemon every three seconds and refreshes the tab when the run ends. 
A second click while a run is in progress is refused (A generation is already running for this site), as is a manual run when the appliance-wide limit of two simultaneous crawls is reached.

URLs tab

The URLs tab is the inventory built by the crawler: 
every URL discovered on the site, with its content type, HTTP status, verdict and, when excluded, the reason. 
It is paginated on the server and can be filtered by path, verdict, content type, exclusion reason and HTTP status.

The exclusion reasons form a closed list:

Weak heuristics never exclude a page on their own:
 an article whose slug contains the word search is not treated as a search results page. 
Detection combines the exact path, its segments, the query parameters and the HTML signals.

Exclusions tab

Exclusion rules let you keep URLs out of the sitemap even though the crawler would publish them: 
a private area, a staging section, a family of parameters. 
A rule only affects the publish verdict; the crawler still follows the links of an excluded page, so excluding a listing page does not hide what it links to.

Click Add exclusion to create a rule, or the pencil icon to edit one.

Rules are matched on the decoded path and parameters, never on the raw URL string. 
A rule can be disabled without being deleted. 
Rules take effect at the next run, scheduled or started with Generate now; 
the excluded URLs then appear in the URLs tab with the custom_exclusion reason.

History tab

The History tab lists the last runs of this site, most recent first, with their trigger (manual or scheduled), status, counters, duration, and whether the run was published.

The last two generations are kept on disk; older ones are deleted after each successful publication.

What is published

The reverse-proxy serves the following URLs for the site, from local disk, with the security modules that could block a search engine crawler disabled on these paths:

• /sitemap.xml — a sitemap index when several families or several files exist, otherwise a single URL set;
• /sitemap-pages-NNNN.xml, /sitemap-articles-NNNN.xml, /sitemap-products-NNNN.xml, /sitemap-images-NNNN.xml — the family files, numbered from 0001, split when a file would exceed the protocol limits.

Example of the index generated for the site shown above:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <sitemap><loc>http://sitemap-test.articatech.unix/sitemap-pages-0001.xml</loc><lastmod>2026-09-02T13:57:45Z</lastmod></sitemap>
    <sitemap><loc>http://sitemap-test.articatech.unix/sitemap-articles-0001.xml</loc><lastmod>2026-09-02T13:57:45Z</lastmod></sitemap>
    <sitemap><loc>http://sitemap-test.articatech.unix/sitemap-products-0001.xml</loc><lastmod>2026-09-02T13:57:45Z</lastmod></sitemap>
    <sitemap><loc>http://sitemap-test.articatech.unix/sitemap-images-0001.xml</loc><lastmod>2026-09-02T13:57:45Z</lastmod></sitemap>
</sitemapindex>

A family file, with the lastmod taken from the page content:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <url><loc>http://sitemap-test.articatech.unix/</loc><lastmod>2026-08-25T08:00:00Z</lastmod></url>
    <url><loc>http://sitemap-test.articatech.unix/product-1</loc><lastmod>2026-09-02T15:52:48+02:00</lastmod></url>
</urlset>

In the image sitemap, the <loc> of each entry is the page, and its images are listed inside it, as the protocol requires:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1">
    <url>
        <loc>http://sitemap-test.articatech.unix/product-1</loc>
        <image:image><image:loc>http://sitemap-test.articatech.unix/img/product-1.jpg</image:loc></image:image>
    </url>
</urlset>

When robots.txt is managed by Artica for this site, it ends with the announcement of the generated sitemap:

User-agent: *
Disallow: /prive/
Sitemap: http://sitemap-test.articatech.unix/sitemap.xml

URLs are built on the site's canonical host name and public scheme (HTTPS when the site has an SSL listener), never on the backend address.

Safety rules worth knowing

• An empty or failed crawl never empties a healthy sitemap. 
  A backend that is down for a few hours produces a run with zero URL; 
  publishing it would remove every URL of the site from the search engines' index. 
  Such a run is not published and the previous generation keeps being served. 
  A genuinely empty site, whose home page answers and whose crawl completes normally, is published as is.
• Publication is atomic. 
  The new generation is written next to the old one and the served link is switched in one operation, 
  after the XML has been validated. 
  Search engines never see a half-written file.
• The crawler sees the backend before the reverse-proxy rules. 
  URL filters, rewrites, custom locations and caches configured on the site are not applied to the crawl. 
  In rare cases the sitemap can list a URL that the public site rewrites or refuses; add an exclusion rule for it. 
  Crawling the public listener instead would expose the crawler to the site's own WAF and anti-bot challenges.
• The crawler is polite. 
  It identifies itself with its own User-Agent, respects robots.txt when asked to, limits concurrency per site,
   slows down on 429 and 503 and honours Retry-After.
• The appliance limits itself. 
  No more than two crawls run at the same time, all sites included;
   the others wait for the next scheduler tick.
• Stopping the daemon cancels running crawls without publishing anything; they appear as interrupted in the History.

REST API

The feature is driven by the reverse-proxy daemon's REST API on the local Unix socket /run/reverse-proxy.sock. All routes are scoped by the website identifier.

Example from the appliance shell:

curl -s --unix-socket /run/reverse-proxy.sock http://x/sitemap-xml/90/status
curl -s -X POST --unix-socket /run/reverse-proxy.sock http://x/sitemap-xml/90/generate

Troubleshooting