In most cases the answers are to do with the structure or format of their website and how it responds to our spider.
To help users understand and address these problems we've started to automate some standard checks we should normally do manually to help resolve these issues.
The diagnostic tool runs through a series of tests designed to mimic the behaviour of our spider.
We test things like
- Accessing your server
- The server response codes
- Parsing of the HTML content
- Important tags such as
- titles and headers
- canonical urls and
- http refresh
- Number of urls found
|Exampe diagnostics output|
You can access the diagnostic tool when you download your sitemap and from the help section.
In this release we also fixed a couple of bugs.
- Fix: processing of http equivelent refresh meta tag.
- Fix : noindex / nofollow rules to ensure pages that we're not indexed we're still followed where there wasn't a nofollow value.