SEO goes far beyond the selection of keywords and The creation of backlinks. There are also A technical aspect of SEO who will have a significant impact on your ranking in search engines. This is where the Robots.txt file appears. When a search engine robot explores a website, it uses the Robots.txt file to determine what parts of the site must be indexed. If your Robots.txt file is not properly optimized, it can cause major referencing problems For Your website. This is why it is important that you understand exactly how it works and what you need to do to make sure youHave a properly configured robots.TX.
What is the Robots.TXT file?
A Robots.txt file is a text file which lies in the root repertoire of Your website and which gives robots search engines instructions on The pages that they can Explore and index.
Search engine robots Try troUver pages available on the public web, that they can Include in their index. When visiting a website, the first thing they do is to research and Check the contents of the Robots.TXT file.
In function Specified rules In the file, they create A list of URLs that they can explore but also index.
What’s going on if you don’t have a robot.txt file ?
In the absence of a Robots.txt filesearch engine robots suppose that all the pages of the website in question, and which are accessiblecan be explored and added to their index.
What happens if the Robots.txt file is not well formatted?
It depends on the problem. If search engines cannot understand the content of the file because it is poorly configured, they will still access the website and will ignore what is in the Robots.txt file.
What happens if I accidentally block the access of search engines to my website?
It’s a big problem. To begin with, they will not explore and do not indicate The pages of your website And, gradually, they will remove the pages Already available in their index.
Where is the Robots.txt file on a site?
When they arrive on a site, exploration robots know that they must Search for a robot.txt file. But they only seek this file specific location : THE main repertoire (generally Your root domain or your home page).
If an explorer robot goes to the site www.exemple.com/robots.txt and cannot find a robot file at this place, He will assume that the site do not have it and will proceed to exploration of everything which is on the page (and perhaps even from the entire site).
Even if the Robots.txt page existed, for example, at the address example.com/index/robots.txt or www.exemple.com/accueil/robots.txt, it would not be not discovered by robots and The site would therefore be treated as if he had No file Robots.txt. So that your file Robots.txt be found, Include it Always in your main directory or your root domain.
Now let’s see How to access (or create) and Optimize the Robots.txt file on one WordPress site.
As we have specified, the Robots.txt file is generally found in the root folder of your site. You will have to connect to your site using of an FTP customer type ” Fringeilla“, Or by using the file manager of your control panel to view it. It is simply an ordinary text file that you can then open with a notebook.
If you don’t have Robots.txt file in the root repertoire of your siteyou can create one. You just have to Create a new text file on your computer and theSave under the name of robots.txt. Then just it download in the root folder of your site.
What does an ideal robot.txt file look like?
The format of a Robots.txt file is very simple. The first line generally designates a User agent. The user agent is the Name of the robot research with which you try to communicate. For example, Googlebot or Bingbot. You can use the asterisk * to give instructions to all bots.
The next line contains authorization or refusal instructions for search engines, so that they know Which parties you want them index and which you don’t want them to index.
Here is an example:
Note that if you Use a plugin as Yoast Or All in One SEO,, The sitemap will be added automatically to your website. If this fails, you can add it manually as in the example above.
What do I have to “prohibit” or “noindex”?
In his instructions to webmasters,, Google advises of do not use their Robots.txt file To hide from low quality contents. Thus, the use of your Robots.txt file For prevent Google from indexing Your pages of categories, dates and other archive pages is not necessarily a wise choice.
Don’t forget that The goal of the Robots.txt file is to indicate robots what they have to do with content that they explore on your site. He Do not prevent them from exploring your site.
In addition, you do not have no need to add your WordPress connection pagean administration directory or a registration page in robots.txt, because Connection and registration pages have it Noindex tag added automatically by WordPress.
We also recommend that you Do not allow the Readme.html file in your Robots.txt file. This Readme file can be used by someone Who tries to know which version of WordPress you use. If it is a person, it can easily Access the file Simply sailing. Moreover, The insertion of a “disallow” tag can Block malicious attacks.
Although you can display the contents of Your ROBOTS.TXT file By accessing the file URL, The best way from it Test and validate it is to use The dedicated test tool made in Google.
- Access the robot tester tool.txt.
- Click on the test button.
If all is well, the button Test becomes green and the label becomes “authorized”. If there are a concern, the line that causes the problem will be Demonstration.
You can also Use the URL tester (at the bottom of the tool) to enter an URL of your website And test whether it is blocked or not.
You can make changes to the publisher and Check the new rules. But, so that these are Applied to your Robots.txt file Live, you must nevertheless TO MODIFY Your file with A text editor and download the file in The root folder of your website (as explained above).
In order toinform google that you have Make modifications to your Robots.txt fileclick on the Submit button. Then click on the Submit button in the contextual window.
