Skip to content

Add robots.txt and sitemap.xml (both currently return HTML) - #25

Open
BigChrisCooke wants to merge 2 commits into
realignedtech:mainfrom
BigChrisCooke:seo/robots-and-sitemap
Open

Add robots.txt and sitemap.xml (both currently return HTML)#25
BigChrisCooke wants to merge 2 commits into
realignedtech:mainfrom
BigChrisCooke:seo/robots-and-sitemap

Conversation

@BigChrisCooke

Copy link
Copy Markdown

The problem

Both control paths return the app shell instead of the file search engines expect:

GET /robots.txt   → 200  text/html; charset=utf-8
GET /sitemap.xml  → 200  text/html; charset=utf-8

Neither file exists in public/, so spaHandler in backend-go/main.go falls through to index.html — it serves any file present on disk and only falls back when os.Stat fails.

Net effect: the site has no crawl directives and no URL list, and a crawler asking for sitemap.xml gets a web page with a text/html content type.

The change

Two static files in public/. No Go change needed — the server already serves whatever exists; it just had nothing to serve.

  • robots.txt allows all crawlers, including AI assistants, and declares the sitemap. For a directory of alternatives, being citable is rather the point.
  • sitemap.xml lists the single public route. lastmod is omitted rather than invented, since there is no reliable substantive-edit date.

How this was found

The identical defect was fixed on realigned.io on 2026-08-16 during an SEO audit — same Go spaHandler pattern, same cause, same one-line-of-effort fix. Verified there afterwards: robots.txt returns text/plain and sitemap.xml returns text/xml, with no other behaviour changed.

Not included

Unknown paths here still return 200 with the homepage, so typos and dead links look like valid pages. On realigned.io that turned out to be more than theoretical — Google had crawled a URL that did not exist, received a 200, and filed it under Crawled – currently not indexed.

That fix is a small change to spaHandler (serve a real 404 for anything that is neither a file nor a known route), but it touches the server rather than adding static files, so it is kept out of this PR. Happy to open it separately if useful.

Verifying after merge

curl -sI https://bye-bye-server.com/robots.txt   | head -3   # expect text/plain
curl -sI https://bye-bye-server.com/sitemap.xml  | head -3   # expect text/xml
curl -sI https://bye-bye-server.com/             | head -1   # unchanged, 200

Submitting the sitemap in Search Console afterwards is what actually gets the URL discovered.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant