New : Follow meta refresh tag e.g. http-equiv="refresh"
We found quite a number of websites using meta tags in their homepage to redirect to another page. Previoulsy we were not detecting this so our spider only found a hompage.
We now take the http-equiv tag url and follow it. e.g. :
<meta http-equiv="refresh" content="0; url=http://example.com/">
New : Automatically follow both www. and non www. urls.Some users were not understanding what domain they were using and where they were specifying the full URL within pages mixxing the www and non www equivelent.
e.g. http://www.xmlsitemapgenerator.org and http://xmlsitemapgenerator.org
If you enter one we now follow both by default but we don't duplicate your urls. We list them once using the domain you entered on the setup page.