User-agent: * Allow: / Disallow: /includes/ Disallow: /*.py$ Disallow: /*.sh$ # Sitemap Sitemap: https://www.musickik.com/sitemap.xml # Crawl-delay for heavy bots User-agent: AhrefsBot Crawl-delay: 10 User-agent: SemrushBot Crawl-delay: 10 User-agent: MJ12bot Crawl-delay: 10