Fix crawl errors for search engines


According to what I read in Google’s help documentation, many crawl errors may cause a decrease in Google’s search engine crawl rate for your site, and perhaps the same applies to other search engines, so you should pay attention to these things. In this post, we will discuss fixing crawl errors, which may have several reasons As for the explanation, we will rely on the Google Webmaster Tool and two WordPress plugins to check for broken links and redirects. For me, I did not care about these things in the past because I was simply immersed in developing templates 🙂 Getting to the heart of the matterTo remind you, crawling errors are one of the reasons for the low rate of search engines crawl, and this may affect the speed of archiving your site, according to the following report: Crawl Statistics Report (Websites)Crawl errors have nothing to do with templates, not from near or far. To simplify the matter, when the search engine crawls to a specific page, a question arises whether it exists or not, here your site responds, available or not, that’s all there is to it.

Causes of crawling errors

After updating my site recently and reviewing it in the webmaster tool, I noticed that these reasons are multiple. I will mention only the ones that I encountered on my site. The reasons may differ from one site to another, and they are as follows:

  1. Change the composition of permalinks to articles
  2. Change cute link to ratings
  3. Delete specific articles or pages
  4. Archive numbering problem

The reasons mentioned above are for WordPress, and there are other reasons, such as problems with your hosting or your server, or problems with the DNS. These things are handled with the domain provider or the company that hosts your site.

Illustrative image of some errors that have appeared to me recently in the Google Webmaster Tool

Search Console Crawl Errors

Crawl bug fixes

Fixing these errors depends on the type of error that you are facing, for example, to solve the problem of deleted pages, we transfer them to the home page and permanent links, we correct them and so on until all errors that appear in the webmaster tool are corrected. In order to do the repairs, we will need two WordPress plugins

The additions we will need are as follows:

  1. Add  Broken Links to fix some links
  2. Add  301 Redirects to redirect pages, articles, or categories
  3. Include a condition in the htaccess file to fix some links

Install and activate the plugin from the WordPress control panel. After activation, the first thing is to check for broken links by adding Broken Links. Go to the plugin page as shown in the image

View Broken Links - WordPress Magazine

Broken links do not appear immediately after activating the extension, you need a little time, refresh the page until you see the broken links as shown in the following image

View Broken Links list - WordPress Magazine

The image above is only an example of what will appear to you in the results of the broken links that the extension found. As for the errors that appear, they are either for permanent links and comments links that no longer work. As for the articles or pages that you deleted, they will not appear in these results, but will appear in the Google Webmaster Tool (Search Console)

Remove comment links

As for the links of people who comment on your site, whose status appears as Server Not Found , 403 Forbidden, or 410 Gone , delete them directly, as you do not need them, as shown in the following image:

View Broken Links unlink - WordPress Magazine

You can choose many links at once and then delete them directly. The method is easy and does not require explanation. You can also delete the links in which you find the 302 or you can delete all the comments links, but this method is done after the completion of repairing the links on your site

Fix the structure of permalinks

If you change the composition of your permalinks to your site and you are inserting links to certain articles in other articles, the links that you have inserted will not change automatically, meaning that you must repair those links to match the new structure you made. With regard to repairing these links, I will refer to one example, because the matter varies from one combination to another

For example, I use the current permalink structure only the article name, as shown in the following image:

crawl error permalink check - WordPress Magazine

Previously, I used the month and year as shown in the link below

https://www.mwordpress.net/2011/08/use-google-app-engine-like-cdn/

To fix this link I will modify it by adding broken links as shown in the following image:

View Broken Links edit link - WordPress Magazine

When the mouse is hovered, an Edit Link appears. We click on it to modify the link, as shown in the following image:

View Broken Links fix permalink - WordPress Magazine

As shown in the image above, I removed the month and year from the link, after clicking on Update, the following will appear:

View Broken Links status after fix success - WordPress Magazine

Pay close attention to this process for all broken links on your site if and only if the problem is in the composition of permanent links

Now we will use a code in the htaccess file to redirect visitors from the search engine or the search engine itself to the correct links for me use the code below

RewriteEngine On
RedirectMatch 301 ^/([0-9]{4})/([0-9]{2})/(.*)$ https://www.mwordpress.net/$3

This condition or law, in other words, is valid only if you use the combination (month + year) only

But if you are using the formula (day + month + year), you can use the following condition:

RewriteEngine On
RedirectMatch 301 ^/([0-9]{4})/([0-9]{2})/([0-9]{2})/(.*)$ http://www.mwordpress. net/$3

Do not forget to change the site link in the code above to your site linkAs for the structure of permalinks, it varies from one site to another, for example, there are those who use the name of the classification, and there are those who use a text before the structure of the link, etc.

Now after fixing the links, we go to the Google webmaster tool (Search console), then we check the links that have been repaired, then mark them as shown in the image below, then click on “MARK AS FIXED”

Search Console Crawl Errors mark link as fixed - WordPress Magazine

After that, a confirmation message will appear, press the OK button, as shown in the following image:

Search Console Crawl Errors mark message - WordPress Magazine

Of course, the message says that it may appear later if you do not solve this problem and this is what we did above. After confirmation, the link will be hidden from the list

If you follow these steps, these articles will no longer appear in the Search Console.

Request to remove articles from search engine

As for the articles, pages or categories that you have removed from your site, these pages will not disappear from the site’s errors in the webmaster tool () until you request their removal from the search engine or prevent access to them through the robots.txt file. For a request to remove a link, it is done as follows:

Search Console Remove URLs - WordPress Magazine

After entering the link in the field, click on Continue, and another page will appear. Do the following:

Search Console Remove URLs Submit request - WordPress Magazine

These pages are not deleted directly. It takes time to be deleted from the important search engine. After the removal request, we block these links through the robots.txt file.

For a method with these pages, you can take a look at the robots.txt file. The method is very simple. All I have to do is add the word Disallow: and put the article link, do not put the full link, only the non-existent part of the article link as shown in the following code:

Disallow: /how-to-add-adsense-to-sidebar-in-wordpress-part3/

Nice link fix

In the event that you change the slug link to articles, categories, or pages, you must redirect this link from the old to the new by htaccess because it will simply appear to you also in site errors, the directive is as follows:

For example, a classification of problems and solutions I changed the nice link to it from “problem-and-solution” to “fix-problem” as shown in the code below

RedirectMatch 301 /problem-and-solution/(.*) /fix-problem/$1

The directive accepts Arabic words. I tried it for a page about the blog, as shown below

RedirectMatch 301 /about-blog/(.*) /about/$1

Archive numbering problem

As for the problem of numbering the archive pages, classifications, or honors, in fact, I did not find a solution to it except to prevent access by adding Yoast. I do not see any meaning for these pages in the search engine, so I prevented access to them by search engines. If you want the method, all you have to do is activate the feature next :

Titles Metas Yoast SEO noindex subpage - WordPress Magazine

Redirect links – 301 Redirects

This method is considered a secondary method in case you want to redirect broken links to the home page in order not to lose visitors

Now after taking all the necessary measures above, we are dealing with the pages and articles that have been deleted from the site and are no longer available, we will collect these links from the Broken links extension and the links in the Google Webmaster Tool (Search Console)

View Broken Links No Longer Available - WordPress Magazine
Search Console Crawl Errors list 404 page - WordPress Magazine

Now, as shown in the two pictures, there are links that are no longer available on the site. We collect them in a text file on your device to arrange them and redirect them to the home page or any other page. The important thing is this list that you have collected. You can create a list similar to the links that appear on your site and this is an example:

https://www.mwordpress.net/2010/11/how-to-add-adsense-link-unit-in-wordpress/
http://mwordpress.net/2010/09/15-wordpress-slide-show-plugins/
http://mwordpress.net/category/adsense/
https://www.mwordpress.net/2012/02/new-revisions-for-newstoday-theme/comment-page-2/
https://www.mwordpress.net/2011/04/how-to-display-your-most-popular-posts-with-thumbnails-in-wordpress/
https://www.mwordpress.net/amnews-theme/
https://www.mwordpress.net/2011/03/ways-to-retrieve-the-admin-password-to-wordpress-blog/
https://www.mwordpress.net/add-new-category-post-photo-video/
https://www.mwordpress.net/2012/02/new-revisions-for-newstoday-theme/comment-page-6/
https://www.mwordpress.net/2012/02/new-update-amn-u001-for-theme-amnews/comment-page-2/

Modify the list and remove the domain name to become as follows:

/2010/11/how-to-add-adsense-link-unit-in-wordpress/
/2010/09/15-wordpress-slide-show-plugins/
/category/adsense/
/2012/02/new-revisions-for-newstoday-theme/comment-page-2/
/2011/04/how-to-display-your-most-popular-posts-with-thumbnails-in-wordpress/
/amnews-theme/
/2011/03/ways-to-retrieve-the-admin-password-to-wordpress-blog/
/add-new-category-post-photo-video/
/2012/02/new-revisions-for-newstoday-theme/comment-page-6/
/2012/02/new-update-amn-u001-for-theme-amnews/comment-page-2/

Now we will redirect these links by adding 301 Redirection. As for adding links, the method is simple as shown in the following image:

301 Redirects add redirect link - WordPress Magazine

Links are listed by number

  1. Put the broken link
  2. Select the page you want the link to be sent to in case of an order
  3. Then save the settings

After completing the process

Head to the Google webmaster tool and then mark all the links that have been redirected to the home page or the page you want as shown in the image

Search Console Crawl Errors fix all links - WordPress Magazine

Errors must be reviewed periodically until they are completely fixed. May God help you and have mercy on yo

Leave a Comment