Xml Sitemap Generator

Wordpress 5.5 supports XML sitemaps

Tuesday, 18 August 2020

With the release of Wordpress 5.5 there are some interesting new developments included. Not least Version 5.5 of Wordpress includes support for XML sitemaps, a long overlooked feature and the reason for many sitemap plugins existing, including ours.

While the new feature will serve many users with a basic XML sitemap need, you might want to stick around to learn why this new feature might not be suited to your SEO needs.

A quick XML Sitemap recap


XML sitemaps have a number of advantages and are a key part of your SEO strategy. In particular they: 
  • Maximise your search engine coverage and ranking by submitting all pages.
  • In doing so they improve the speed which your pages get listed.
  • Allow you to take better advantage of search engine webmaster tools
  • Receive alerts about problems in your website
  • Importantly, they allow you to provide hints to search engines about the content in your pages..

XML sitemaps allow you to provide a lot of information about your website content including:

  • Web page address
  • News topics
  • Images and videos
  • When and how often a  page changes
  • Relative priority of content within your site

Coupled with RSS sitemaps you can also provide content which can be aggregated even more broadly to ensure your content is distributed and consumed.

Wordpress 5.5 sitemap feature vs plugins


The new wordpress sitemap feature is very basic and essentially just spits out a sitemap for all the published pages in your website without any user control or indeed any user interface.


“Are there any UI controls to exclude posts or pages from sitemaps?
No. User-facing changes were declared a non-goal when the project was initially proposed,”


Many sitemap plugins at least offer you a degree of control and one of the key design goals of our plugin was to offer you very granular control over your sitemap.

If you want to be able to edit your sitemap, including setting defaults by categories / tags or even edit down to the page level then our wordpress XML Sitemap plugin remains one of the few to enable you to do this.



Another key feature is that we support sitemaps in a variety of formats including XML, RSS and HTML files which maximizes support. RSS in particular is very helpful towards your SEO.


We also actively test with other plugins such as woo-commerce as well as supporting custom posts which gives your sitemap broad compatibility. Again the high level of customization applies to your custom posts types.

We're also exicted to let you know that version 2.0 of our sitemap plugin is on its way in the coming days with some pretty big additions and improvements.

Conclusion


While there is no doubt the inclusion of XML Sitemaps in the core Wordpress distribution is a welcome benfit and may suit the needs of many users with basic needs, if you are serious about SEO and need flexibility and control, you should certainly give our XML Sitemap generator plugin for Wordpress a try  and watch out for version 2!
Categories: ,

New G-Mapper version available

Tuesday, 28 July 2020

We're pleased to announce the release of G-Mapper version 3.0.5.0. This version includes the changes previously released in the online version as well as a few enhancements and bug fixes.

We have finally decided to purchase a code signing certificate to remove some of those nasty windows warning about untrusted applications! If you can help contribute towards our costs we really appreciate your support.

Diagnostic tool

We added a diagnostic tool to our online sitemap generator some time ago. Due to popular demand and to help with support issues, this tool is now available directly within G-Mapper.

You can use the tool to help you diagnose problems with creating your sitemap which may also impact on your websites SEO.


You can access the diagnostic tool from the main tool bar which will scan your sitemap start/home page, or using the new context menu to scan a particular page.

Context menu

To make managing your sitemap easier and finding pages within your website, we've added a context menu so when you right click you can navigate to the page or copy the url as well as accessing the diagnostic test tool.



Canonical urls

As with the online sitemap generator, due to how various sites have implemented canonical urls we will no longer automatically redirect to alternate domains as this is causing problems due to misuse / poor implementations on some sites.

Instead we will process the page within the context of the current domain and issue warnings about canonical urls in the logs by default.



If you set the option to obey canonical urls we will assume you are spidering the correct domain and when we see a canonical url:

  • If it matches the current page we will include the current page.
  • If it is within the current domain we will ignore the current page and content and add the canonical url to the list to be spidered.
  • If it is outside of the current domain we will ignore the page and content.

Robots meta and rel nofollow / no index 

Again, as with the online sitemap generator, some sites end up with no pages or missing pages due to the use of the robots meta tag and rel nofollow attribute which has created negative feedback.

By default we will now ignore these and issue warnings in the logs. If you wish for our spider to obey any robots meta or rel attributes please used the advanced settings.

Other changes

We've changed our user agent header to imitate common browsers as some websites we're issuing a HTTP 403 not authorised error.

Increased the max page size to 300Kb (uncompressed) as some sites were not finding urls due to their size.

We've also been making some general improvements to our spider code as well as some bug fixes to try and make it more reliable and robust.

We've added more validation around the FTP upload to make it easier to diagnose issues.

You can now reach our web diagnostic tool from the toolbar and settings page.

Bug fixes

We've also been resolving a number of bugs thanks to the reports you send us. Please do include your email address when you send a report as it can be really helpful in some cases to be able to email you for more information.

  • Gzip and deflate compression not working causing excessive bandwidth use. 
  • Fixed mix date standards to YYYY-MM-DD format.
  • Disable batch edit when not on correct tab to prevent errors.
  • Ping utility error fixed.
  • Error when trying to export to folder without write permission.
  • Error exporting map with no urls.
  • Crashing when checking version and cannot reach server.
    •  

Links


Categories: ,

New Wordpress Plugin version released

Tuesday, 14 July 2020

An update to our Wordpress plugin is long overdue, but finally it is here with some fixes and a few popular requests implemented.

We've updated our testing to ensure that version 1.3.5 of the plugin works with the latest version of Wordpess (5.4) as well as remaining backwards compatible as before.

Customise sitemap filenames

One of the most requested features was the ability to customise the sitemap file names. Previously the only way to do this was to edit the plugin code which made it challenging. You can now easily configure the sitemap filenames from within the main settings page.

Simply go to the settings page, scroll to the "Output urls" section, edit the files names and hit save. Be sure to stick to simple filenames using basic alpha numeric characters and no special symbols or slashes to avoid issues.

You should also be mindful of other plugins which might generate the same/similar file names.


Set sitemap values by author

As well as being able to set a general value for authors in your sitemap settings you can now set a default per author if you want to give certain editors different values in your sitemap. This can be useful if you have a number of content authors, some who work on general pages and others who work on higher value content or who have content that changes more often.


Include scheduled posts.

Previously if a post was scheduled it would be excluded from the sitemap. Some of our users wanted to be able to include posts before they were published. You can now configure the plugin to include standard pages and posts which are scheduled.


Other fixes

  • Erroneous bug output due to undefined authors variable.
  • Incorrect sitemap values due to unique ID collisions
  • Page options do not work for multisite/network.
  • Settings link error on network installations of plugin.
  • Google ping url error

Get the latest version of our Wordpress XML Sitemap Generator Plugin.

We hope you find these updates useful. If you do find our plugin valuable please consider making a small annual contribution towards our costs.


Categories: ,

Key sitemap generator changes

Saturday, 9 May 2020

We've been making a number of changes to our online sitemap generator. Some of these will filter down to G-Mapper over the coming weeks. It's important to understand these recent changes as they impact how we spider your website.

Hosting updates

We recently completed a migration to new hosting within Microsoft Azure to simplify management and deployment of services. We hope that this will reduce down time and deployment errors as well as allow us to scale more easily. 

Most of the major changes are now complete and so any minor updates should cause minimal disruption

Canonical urls

Due to issues with how various sites have implemented canonical urls we will no longer automatically redirect to alternate domains as this is causing problems due to misuse / poor implementations on some sites.

Instead we will process the page within the context of the current domain and issue warnings about canonical urls in the logs by default.

If you set the option to obey canonical urls we will assume you are spidering the correct domain and when we see a canonical url:

  • If it matches the current page we will include the current page.
  • If it is within the current domain we will ignore the current page and content and add the canonical url to the list to be spidered.
  • If it is outside of the current domain we will ignore the page and content.

 


Robots meta and rel nofollow / no index 

Some sites end up with no pages or missing pages due to the use of the robots meta tag and rel nofollow attribute which has created negative feedback.

By default we will now ignore these and issue warnings in the logs. If you wish for our spider to obey any robots meta or rel attributes please used the advanced settings.


 

Diagnostic tool 

The diagnostic tool has been updated to give a more realistic view of how our spider sees your page. When you use this tool it will now output a verbose log of how it processed your page. These details will also be included in the log which we include in your sitemap download file.

When a spider session returns no results or only one page the log details from homepage will be displayed on the results page. This can often be due to the website blocking our spider.



Other changes

We've changed our user agent header to imitate common browsers as some websites we're issuing a HTTP 403 not authorised error. If your site still blocks us with a 403 we will try to detect this and notify you.

Gzip and deflate compression were not working and causing excessive bandwidth use. We have now resolved this which will improve performance.

This has enabled us to increased the max page size to 300Kb uncompressed as some sites were not finding urls due to their size. This will be notified as a warning as this is exceptionally large for a page.

If your site uses javascript to detect bots and blocks our spider we will try to detect this and notify you. Specifically we're noticing a number of sites use the "testcookie nginx" module. We are working on a better solution to address this but it required further investigation.

We've also been making some general improvements to our spider code as well as some bug fixes to try and make it more reliable and robust, this will hopefully benefit G-Mapper users once it rolls out to our Windows solution.

Support us


Don't forget we have now updated our service so that you can contribute and get some benefits. Please consider supporting us to help keep the project alive.

Support us.

Categories: ,

Become a contributor and get extra benefits

Tuesday, 28 April 2020

We provide all our sitemap services for free and for the love of it, but running this service on advertising revenue and donations is difficult as they are unpredictable and in recent times they have not covered our costs.

In May 2018 new European Union Data Protection regulation had a significant impact and with recent world events we now find ourselves under significant financial pressure.

As a result we have brought forward plans to better manage our capacity and introduce our patron scheme which will allow people to contribute to the service and get extra benefits.

A basic free service will be maintained so that over 80% of our users will be able to continue with little or no impact while heavier users will be able to become a contributor.

Patron contributor tiers

Initially we will have 5 tiers of service which will suit the majority of our users. If you require  additional services, please contact us.

All our contributors will benefit from having

  • No google adverts, 
  • No captcha human check to simplify the user experience and 

Each usage tier will allow for more pages, more websites and a greater number of spider sessions in a 28 day period as follows :


  • Free - up to 250 pages for up to 1 website
  • Bronze - up to 500 pages for up to 1 website
  • Silver - up to 1000 pages for up to 5 websites
  • Gold - up to 2000 pages for up to 10 websites
  • Platinum - up to 5000 pages for up to 20 websites

Payments can be made using Paypal which includes card transactions.
We will continue to review these tiers over the coming months.

For more details about the tiers please visit our contributor page.

More about our costs

We hope that by introducing this new model we'll be able to cover our basic costs include things like:

  • Domain names and SSL certificates.
  • Web server and database hosting
  • Bandwidth and storage costs
  • Development and testing tools
  • Software licences
  • Email services
  • Marketing

We need 1,000's of supporters just to cover these, before we consider time and effort invested in the service, so we really appreciate any support.

Our aims

Of course we want to do more to improve the service and by becoming a contribute you can help.

  • Ensuring the existing service is financially sustainable.
  • Support upgrading our hosting infrastructure.
  • Adding resilience and redundancy.
  • Invest in development and updates to the service.

We hope that these changes will ensure the future of the service and thank you for your continued support.

Upcoming service changes

Wednesday, 8 April 2020

Over the last few months we have bee looking at the sustainability of our various sitemap solutions.

Since the introduction of new EU Data Protection legislation in 2018 our ability to generate revenue through advertising has become limited.With dwindling advertising revenue from current streams and limited donations we are increasingly finding ourselves reaching in to our own pockets to fund the service.

We have been looking at options to allow the continued running and maintenance of the services as well as ways to invest in a number of key areas such as infrastructure, developer tooling, support materials as well as new services.

Having experimented with a number of models we believe that providing a patron model for a percentage of our community would be the best way forward, where we can generate more regular  reliable income and offer some basic benefits as a gesture of goodwill.

With recent world events surrounding the pandemic, the need to address these challenges is ever more pressing as we find ourselves under increased financial pressure. In order to address these issues we will be accelerating the implementation of these new services as well as implementing some cost cutting measures.

  • Reducing the number of pages that our spider will include in a sitemap as standard to reduce our infrastructure overhead costs.
  • Tweaking our fair use policy to ensure we can continue to serve the community at large.
  • Limiting the account features / saving of sitemaps
  • Introducing our patron scheme which will allow contributors to support the service and also get access to more features / resources.

The final details of what this entails are still being worked on, but from our initial modelling the good news is that for over 80% of our users the service will (to most intents and purposes) be unaffected.

For the remaining users, we have identified that they are more frequent and higher volume users of the service, however the patron model will allow them to go over these limits for a nominal contribution in line with our current donation model.

Further details and final arrangements will be published over the coming days, but we felt it was important to give you some notice of these incoming changes.

We thank you for using the service and your continued support us and we wish you, your colleagues and family all the best during this difficult time.
Categories: ,

What has been happening?

Thursday, 19 March 2020

First of all we're sorry we've been quiet for a while. The service is very much run on a volunteer basis and some sometimes other priorities take over.

We rely on the support of our community and advertising revenue which of late has decreased due to stricter rules on the use of personal data. We thank you for your continued support.

What's new?


We've updates our WordPress plugin having tested it for the most recent version of wordpress as well as publish a new version 1.3.5 which we would welcome input from our community.

1.3.5 should be considered test and not for use in a production setting.




Online sitemap generator

We've made a number of improvements to our service to improve compliance with data protection legislation. Including :

  • Privacy policy updates.
  • Consent and use of cookies.
  • Automatic download of your data.
  • Right to erase your data

Some people use our service heavily and seemingly for commercial purposes. To help moderate use and make it fair as well as keep our cost sunder control we have implemented a fair use policy and will limit the number of times the generator can be used in a given period. We've been testing and tuning this over the last few months.

Most people wont hit this limit but if you do simply wait a few days and you'll be able to use the service again. We're hoping to introduce a premium service shortly for users with greater needs.

We've also fixed a couple of bugs. In particular the bug which included external links being wrongfully included in the sitemap when they were part of a redirect.

G-mapper

G-mapper receives the same bug fixes as out online version and we've also updated some of libraries and code to more recent versions. We've also addressed some of the color coding issues and user interface performance issues for larger sitemaps.

What's coming?


Over 2020 we're hoping to offer improvements to both our online sitemap generator and G-Mapper, including closer integration and more advanced features and options. We'll also officially release version 1.3.5 of our WordPress plugin once this has been tested further.

Don't forget to follow us on social media to stay up to date with news and updates.
Categories: , ,
Subscribe to: Posts (Atom)