Docs: Add robots.txt, llms.txt and update documentation urls (#377)

Docs: Add robots.txt, llms.txt and update documentation urls (#377) #1317