By default your Faspex site is designed to remain as hidden as possible by preventing search engines such as Google from indexing it and displaying it in search results.
This is thanks to the
noindex HTML meta tag which removes your Faspex site from all of a search engine’s search results, regardless if other websites link to it:
<meta name="robots" content="noindex">
This tag is located in the HTML header section of each page of your Faspex site. When a web crawler accesses your site it checks for the
noindex tag, and when it finds the tag it removes your site from the search engine’s search results.
However, in order for the
noindex meta tag to work, you must ensure you are not blocking search engines from seeing the
noindex tag with a
robots.txt file provides instructions to web crawlers, and a common way of preventing search engine indexing is to place instructions in this file to disallow web crawlers.
For example, a
robots.txt instruction that disallows web crawlers on every page of the website would look like the following:
This isn’t always effective though, because if other websites link to your website it can still be picked up and indexed by search engines.
robots.txt file that blocks access to your site also has the consequence of preventing web crawlers from reading the
noindex meta tag on your pages, so the full removal of your website from search results will not take place.
Therefore, for maximum hiding of your Faspex website, ensure that your
robots.txt file allows access to your Faspex pages.