Do you want to hide specific areas of your platform from search engines, or do you want to tell them you have a sitemap available? Say hello to the Robots.txt file. It’s a simple text-file with huge responsibilities. This file will specifically tell search engines how to crawl and index the content of your platform.
Robots.txt is not designed to boss search engines around, they are free to do whatever they please. But it’s become a universal standard and the majority of the bigger search engines (most importantly: Google) will respect and follow the rules you provide. Want to learn how to setup your robots.txt file? We’re explaining as much as we can below!
Default Robots.txt file on inSided platforms
Tip: Use the default rules listed below to stop crawling members, sort and search pages (and save crawling budget)
User-agent: *
Disallow: /members/
Disallow: ?userid=
Disallow: ?sort=
Disallow: search_type=tag
Disallow: search?q=
Robots.txt elements
User-agent: determines for which search engines the rules apply, * indicates that the rules apply for all user-agents
Allow: determines which content is accessible for the user-agent (only works for Google and Bing)
Disallow: determines which content is not accessible for the user-agent
Sitemap: tells the search engine where it can find the sitemap.xml file. This should be an absolute URL and in .xml format. (The inSided platform does not come with a built-in sitemap functionality.)
How To Setup a Robots.txt file for your community
- Go to Control → Settings → SEO → Robots.txt (you have to be an administrator to access this page)
- Provide Robots.txt elements in the details section. The following elements are supported: user-agent, disallow, allow, sitemap
- Hit Save changes, and you’re done! You successfully configured your Robots.txt file.
Example: How to crawl all of my content
User-agent: *
Allow: /
Example: How to crawl none of my content
User-agent: *
Disallow: /
Which search engines support Robots.txt
- Google (documentation)
- Bing (documentation)
- Yahoo (documentation)
- DuckDuckGo (documentation)
- Yandex (documentation)
- Baidu (documentation)
Note: It is not required to have a Robots.txt file. Search engines will crawl all pages of your platform if you don’t provide any.
Beware: The inSided platform does not come with a built-in sitemap functionality. If you want to make use of a Sitemap you have to create and host your own sitemap (in XML)
A Robots.txt file is custom made for your platform only – inSided support won’t be able to assist you with issues related to your robots.txt.