Sitemap generation performance issues

Sunday, 13 April 2014

We would like to apologise for the recent variable performance over the last few days. We believe we are now starting to get it back under control....

The recent improvements created some additional problems when dealing with slow websites and hung connections. Some websites started locking up resources that were not being released quick enough  to serve other sitemap requests.

To address this we have added a number of controls to the existing list which are summarised below.

Max of 1000 pagesExistingThis was  recently increased from 250 to 1000
Page size of 80KUpdatedThis was 100K, but to improve performance we had to bring it down. The majority of websites should be unaffected by this.
Response time 30sNewYour website must respond to a request and send all of the page content within 30 seconds. For the majority websites this should not be an issue.
3 concurrent downloadsNewThis has been increased from 1 to 3 to improve sitemap generation speed.

We are continuing to monitor the situation and make further improvements over the coming days.