Hiding Faspex from search engines

Description

By default your Faspex site is designed to remain as hidden as possible by preventing search engines such as Google from indexing it and displaying it in search results.

This is thanks to the noindex HTML meta tag which removes your Faspex site from all of a search engine’s search results, regardless if other websites link to it:

<meta name="robots" content="noindex">

This tag is located in the HTML header section of each page of your Faspex site. When a web crawler accesses your site it checks for the noindex tag, and when it finds the tag it removes your site from the search engine’s search results.

However, in order for the noindex meta tag to work, you must ensure you are not blocking search engines from seeing the noindex tag with a robots.txt file.

Explanation

A robots.txt file provides instructions to web crawlers, and a common way of preventing search engine indexing is to place instructions in this file to disallow web crawlers.

For example, a robots.txt instruction that disallows web crawlers on every page of the website would look like the following:

Disallow: /

This isn’t always effective though, because if other websites link to your website it can still be picked up and indexed by search engines.

A robots.txt file that blocks access to your site also has the consequence of preventing web crawlers from reading the noindex meta tag on your pages, so the full removal of your website from search results will not take place.

Therefore, for maximum hiding of your Faspex website, ensure that your robots.txt file allows access to your Faspex pages.

Have more questions? Submit a request

0 Comments

Article is closed for comments.
Powered by Zendesk