Docs: Add robots.txt, llms.txt and update documentation urls

Docs: Add robots.txt, llms.txt and update documentation urls #1314