WordPress Plugin directory - Gone!

Thursday, 16 June 2022

As with many of my projects, my Wordpress XML Sitemap Plugin started as something for one of my own websites as there were some serious limitations with using other plugins for effective SEO. Even today XML Sitemap offerings for Wordpress are limited.

Since then my plugin went from strength to strength gathering more users. In early 2021 I re-wrote a large part of the plugin to improve maintainability and extensibility as it had become a bit "organic". When I was last tracking usage we had over 100,000 websites using our plugin, which is great for something I put together for only 1 or 2!

The Plugin Directory "process" problem

In recent times the Wordpress "plugin team" have taken a more active interest in plugins, rightly so, as one of the top ways Wordpress websites get hacked is via plugin vulnerabilities. I'm well aware of this as I run several websites.

However, the way Wordpress have started "policing" their plugin directory is not conductive to the maintenance of a free open source plugin community. Frankly it is tiresome at best, belligerent at worst and not something I have the time to deal with.

Their "nit pick" approach, without categorization or gradation is unhelpful. It's a terrible "process" and I use the word loosely having been involved in numerous security audits before.

To be clear, I have always responded to critical bugs and security concerns as quickly as possible and continue to do so. At this time I am not aware of any, but, the back and forth with their team and constant threats to delete the plugin unless I act on their every email is frankly ridiculous.

Many of their emails are are around trivialities and without proper consideration for context. For example there is an issue with this function, can you see what it is? 



$allowString isn't escaped. 
 
Technically a valid point, but not a serious or exploitable issue in this context, but respond to their email or else.
 
If they did the job properly and carried out an audit that would actually be helpful, but they don't, you fix one "issue", make a release, and then a week later they say, "oh you missed something" and so it goes on.
 
I've had months of this crap on a handful of trivial issues and I've had enough of it.
 
As I say, critical and high severity issues get fixed as immediately as possible. Everything else goes on the backlog and gets released in the next version. I don't have a development and testing team running sprints every week. It's literally just me and a few helpful contributors who weigh in from time to time. Sometimes it can be months between releases.

Don't worry, the plugin is still here!

As a result of this time drain, and pitiful "process",  I have taken the decision to remove my plugin from the Wordpress Plugin Directory and make it available for direct download from my website instead. 

I will still maintain and support it, just not on the plugin directory.

I know this is a bit of a ball-ache for people who maintain websites, and I will try to make this as painless as possible, but if you have a complaint, please take it up with Wordpress and tell them to sort their processes out.

I also hope to publish the source code on GitHub at some point so that people can get involved and communicate this way, but in the mean time, you can contact me via the XML Sitemap Generator website or join the telegram group.

I don't always respond immediately and can't promise to respond to everyone, in particular for feature requests, but I do keep a note for future releases.

Security audits don't come cheap so if people don't donate enough (and they don't) then it wont happen. It is good practice to review 3rd party code yourself before including in a website anyway, unless there is a 3rd party audit you can depend on.

If someone did want to donate to pay for an audit, or better still carry out the audit so I can respond to any issues and publish the results, I would be open to this.

The future of Wordpress plugins?

It's interesting to think about what this means for the future of Wordpress Plugins in general. 

Wordpress do not allow you to commercially sell your plugin from their directory, let alone provide any support for it as app stores do. They require that plugins are free and open source, while at the same time requiring development and support regime to deal with their every whim.

These two things just aren't compatible.

While some developers have found ways to work around Wordpress rules to commercialize their plugin, I can see plugins disappearing and other great ideas never being realized which is a sad thing for the open source community, but then there are other platforms and mechanisms out there.

Please support my work

Thank you to everyone who has and continues to support my work. A security audit would cost money and if that's what people want it's going to take donations, or a small number of very generous people.

You can support me via PayPal or Buy Me A Coffee. Some suggestions are below :

Individuals
£5 GBP
~ $6.30 USD

Sole traders
£10 GBP
~ $12.60 USD

Small businesses
£20 GBP
~ $25.20 USD

Larger businesses
£50 GBP
~ $61.67 USD


You can also stay in touch via social media, including our new Telegram channel:

Facebook: @XmlSitemapGenerator
Twitter: @createsitemaps
LinkedIn: xmlsitemapgenerator-org
Telegram: t.me/GoogleXmlSitemaps


New Wordpress Sitemap Plugin v2.0.5

Saturday, 30 April 2022

After a period of quiet I have released a new version of our plugin. Initially this was version 2.0.4 but following a critical bug 2.0.5 was released to hot fix this issue. In hindsight, this should have been 2.1.0 due to the number of code changes.

Sorry to those who were impacted by this issue. It slipped the net during testing and is part of the reason for this blog.

In this new version much of the work was focused on updating coding standards and conventions required by Wordpress as recently their enforcement has become more aggressive with threats to close the plugin.

The short back story is that I started this plugin for a website I was running  and decided in the spirit of open source to make it available. Over time it has grown in popularity and we now have over 100,000 websites using it.

Version 2.x of the plugin saw a major overhaul of the code to improve how it worked from its rather hacky 1.x days and in this more recent version we have been improving code quality.

What's changed in 2.0.5? 

While there were not really any new features, a lot of underlying code was updated which is what lead to the code breaking.

  • New : Set an empty sitemap file to disable a given map
  • Fix : Index file showing news feed twice
  • Fix : Archive provider rendered sitemap regardless of setting
  • Fix : Erroneous variables producing debug warnings in archive provider.
  • Fix : Adopted Wordpress sanitize and escape filtering functions
  • Fix : Moved to using enqueue scripts instead of inline scripts
  • Fix:  Addressed a problem with the stable tag label.
  • Fix : Removed google analytics
  • Fix : Remove the latest archive from XML index.

Are we going to be deleted 😔?

The plugin is at risk of being deleted by Wordpress. Despite being open source they are expecting plugins to operate more like businesses, requiring plugin publishers to adopt more of their standards in terms of coding, updating, releases, etc. and that requires resources.

As a side project that I released to the community, maintaining and supporting the plugin as its user base has grown has become increasingly challenging and at times I do not have the time, or skills to undertake what Wordpress are asking for.

Please help and contribute

I want to keep the project going, for it to be freely available, open source and supported by the community, so I am really relying on good will and generosity of the community to keep it alive. Please consider contributing to the project.

Contribution suggestions
Individuals with a small website
£5 (about $6.30 USD)
Contribute now
Small business and professional websites.
£10 (about $12.60 USD)
Contribute now
Commercial websites and services
£20 (about $25.20)
Larger websites, multiple websites, or use our plugin for professional services / consultancy.
Contribute now
Larger businesses and absolute legends
£50 and over.
If you are feeling particularly generous and legendary, or are one of the very large commercial websites that use our plugin.
Contribute now


Thank you in advance for your support

 

Priorities for resources ....

The only way the plugin can continue is if the community gets behind it.
 
  • First and foremost it needs some funding to allow me to bring in the skills / resources to bring it up to speed and maintain it moving forward. In particular I would like to get a recognized 3rd party to review the code for new releases to help keep Wordpress happy. This will also hopefully give you confidence that the plugin is up to standard too.

  • Secondly, I will need to put more rigor around testing and releasing, so in the not too distant future I will be setting up an "early access"(or similar) programme with incentives and rewards for helping out, although obviously this will be subject to funding being successful.

  • Thirdly, improving the support resources and community around the plugin. I have tried to do this with reddit and telegram, but struggled and really it needs some time investing to get it right.

  • Forth, (and it probably should be first) I can buy my partner some flowers and make up for all the time I invest in "that X X sitemap thing" I'm always working on. 😂

 


G-Mapper XML Sitemap Generator Update

Friday, 12 November 2021

We've been working on new code for this version for some time and this release (version 3.1) includes the spider back-end changes discussed in our previous online sitemap generator release, as well as some improvements to the user interface.

This version is first being released in Beta as it is not yet fully tested. It should install alongside your existing G-Mapper installation so you can run both. Note, it does not import your existing sitemaps at present, this is something we hope to fix when we move from Beta.

Editor interface improvements

We've made some significant changes to editing sitemaps to improve the user experience.

XML Sitemap Editor

The key change being that instead of using the "manual meta" option and overwriting values, you can the relevant column directly in the grid and overwrite the existing / default value. We highlight these to make them easier to see. If you then delete this change the original / default value will return without needing to re-spider.

As always you can resize the grid columns, but now we save your settings so that next time you return they are retained. This solves perhaps one of the biggest annoyances for many users.

We also made it so that G-Mapper will remember it's startup size and position instead of defaulting to maximize .

Export button

Sitemaps used to be exported to the default location after spidering and you could use the manual export button if you wanted to export to a new location. This was a pain because after editing your sitemap you had to manually export or re-spider.

XML Sitemap Export

We've updated the export button to export your sitemap to the default location so you can use it any time to export the latest version of your sitemap. If you want to export to an alternate location use the export button drop down and select "Select folder".

FTP improvements

As of 16th November (an update to the original release) we've  tweaked FTP settings to add support for FTP over SSL and SFTP connections. We also added an option to allow you to chnage the port number for non-standard setups. In theory FTP and FTPS (FTP over SSL) should auto-detect and work with your server settings, but we would welcome feedback on this.

Sitemap FTP upload
 

Other changes

As part of the spider engine update a few other things have changed. Our spider will no longer act based on the "nofollow" rel attribute for internal pages. If you do not wish for a page to be included you should use a robots meta tag with the "noindex" property value and ensure that you have set the " Obey robots" meta tag option in your sitemap settings.

We have deprecated the options for "iframe", "framesets", "area" tags and will now extract all links from a page and process based on the global rules. Please use regular expressions or other available rules to filter your pages.

Technical stuff.

As with the changes to the online version earlier this year, we have made some technical changes under the hood to improve performance and resolve a number of issues. :
  • Re-writing of the spider HTTP stack, in particular, moving to the new .Net SocketsHttpHandler to improve the performance of requests.
  • Changing from exclusively in memory processing to streaming spider results disk, which allows for more concurrent processing and larger sitemaps without running in to memory issues.


We hope these changes will lead to a more performant and stable spider engine and have fixed a number of bugs.

Download the latest version

Download the latest version from the G-Mapper website.

 

What's coming next?

This was a fairly big update with lots of code rewrites but not many new features, however it does open the door to porting many of the online features to the Windows version including external link checking, SEO and other Reports. 

We're also going to be moving to a hybrid cloud model to help simplify how we manage and maintain more of the application and share more features between the online and desktop versions.


Wordpress Sitemap Plugin Update (2.0.2)

Thursday, 19 August 2021

We've finally got around to updating our WordPress plugin addressing a number of bugs which have prevented some people from upgrading without workarounds.

Again, we're sorry it has taken so long, this is due to juggling various priorities and personal issues.

In early 2021 we made one of the biggest updates to our plugin re-architecting much of it and adding new features and flexibility including :

  • Support for sitemap index files
  • Support for image sitemaps
  • Support for news sitemaps
  • Support for larger sitemaps

 

A patch release (2.0.1) addressed some bugs but others since came to light requiring manual patching or rolling back to a previous version which has been an issue.

We are hopeful that this 2.0.2 patch release has addressed the outstanding issues to enable more people to upgrade. Key issues addressed include:

  • PHP debug / log errors xmlRenderer.php when images disabled.
  • Post settings not updating.
  • News sitemap missing language tag.
  • News not working due to bad database upgrade script.
  • Outstanding activation issues on network installs.

 

You can download the latest version from wordpress


If you have any further issues please contact us via our website or join our telegram group. We did have a reddit group but that has once again been banned. Apparently sitemaps are just to controversial for Reddit.

 
Don't forget to follow us on social media to stay up to date.


Overdue online update

Saturday, 14 August 2021

We've been working on an update for some time to address some bugs and performance issues since our last release. This lead to some pretty big rabbit holes and major reworks of code and re-architecting  how we use Microsoft Azure infrastructure, which has consumed many evenings and weekends, but finally there is light at the end of the tunnel.

Most of the changes in this release were technical under the hood issues that many users wont appreciate although the service is running more smoothly and faster, not withstanding their may be teething issues as we migrate users on to the new release.

Key fixes and changes include

  • Prioritization of sitemap jobs for contributors.
  • Contributor can configure more concurrent requests for faster processing.
  • Longer timeout periods for larger sitemaps.
  • "My sitemaps" page performance improved.
  • Fixed the transient download link issues.


Please note, the more concurrent requests you have the more pressure our spider will put on your website so please use this carefully.

Technical stuff.

Most of the time went in to key technical changes which aren't worth writing up in huge detail but for interest :

  • Re-writing of the spider HTTP stack, in particular, moving to the new .Net SocketsHttpHandler to improve the performance of requests.
  • Changing from exclusively in memory processing to streaming spider results disk, which allows for more concurrent processing and larger sitemaps without running in to memory issues.
  • Migrating the entire spider process to Azure Functions to allow spider sessions to run for longer without timing out and dropping pages.

These key changes were have led to a more performant and stable spider engine, but there was quite a learning curve in some areas which led to lots of time invested in research and bug resolution.

Once we had stabilized the new setup we were able to resolve a number of additional bugs and issues that people had reported, but were transient or hard to reproduce due to stability issues.

External link checker

One issue we have not been able to resolve is the performance of external link validation. We lost a lot of time to this one and despite re-writing our HTTP stack we were unable to resolve it. After many hours we discovered that the problem relates to an issue (feature) of Microsoft Azure. Microsoft is throttling DNS requests and so no matter how performant our code their DNS server is a serious bottle neck.

We have a number of strategies we are looking at to resolve this however it may take some time and unfortunately in the meantime we have disabled this feature to prevent it impacting on other areas of the service.

We haven't forgotten G-Mapper and Wordpress.

In better news, we will be adding a few new features and at some point hope to migrate this code over to G-Mapper so that our Windows users can benefit from these improvements. In the meantime, we're aware that our Wordpress users have been waiting on some bug fixes for some time and so this will be our next priority.

Infrastructure upgrade

One thing that has become clear is that we need to move to beefier hosting and database plans which will ultimately cost more money, almost doubling our costs. We hope that by gaining more contributors we can do this before the end of the year.

Please support us by becoming a contributor  or supporting us in other ways.

Join our community

To help with managing support we've created and online community. Please get involved as it will be a real help to maintaining the project and is a good way to get the latest news and updates and interact with other users about our services, SEO and other related matters.


Ways you can help support us

Thursday, 1 July 2021

This project only keeps going with the contributions from our community and we are very greatful for for your support. We now have more ways in which you can help support the project below.

Contribute / donate

 

Wish list

We have a list of things we need for managing and running the service which you may wish to purchase to help us out.

 

Affiliate links

When you buy from some companies we can get a small % comission when you use are links. A free and easy way to support us and with some you even get a small incentive / discount.


Join our community

Please get involved with our communitues to get the latest news and updates and interact with other users about our services, SEO and other related matters.


Social media

As well as our community, social media is a great way to stay in touch and you can help us out by following us online and giving is a shout out.

 

Coming soon ....

We'll  be adding new ways to contribute over the coming months and we have some exicint ideas about crypto currencies too, so stay tuned.