WordPress now has 4 versions in 3.x line. With slow adoption rate for previous two major versions, despite great 3.2 release, question is will the new WordPress 3.3 manage to persuade users to upgrade? This benchmark will try to give, at least, partial answer to that.
WordPress Release Stats
Before we get to test results, first thing is to see how this 4 version compare when it comes to size, number of files, hooks, requirements and few other things. Latest version is the version we used to test here.
| WP 3.0 | WP 3.1 | WP 3.2 | WP 3.3 | |
|---|---|---|---|---|
| Release Date | 2010.06.16. | 2011.02.23. | 2011.07.04. | 2011.12.12. |
| Latest Version | 3.0.6 2011.04.26. |
3.1.4 2011.06.29. |
3.2.1 2011.07.12. |
3.3.0 2011.12.12. |
| Archive Size | 2.83MB | 2.95MB | 3.79MB | 4.05MB |
| Unpacked Size | 7.95MB | 8.24MB | 9.43MB | 9.97MB |
| Number of Files | 758 | 835 | 948 | 936 |
| Number of Hooks | 1344 | 1469 | 1506 | ~1580 |
| Included jQuery | 1.4.2 | 1.4.4 | 1.6.1 | 1.7.1 |
| Included Themes | TwentyTen | TwentyTen | TwentyEleven TwentyTen |
TwentyEleven TwentyTen |
| Required PHP | 4.3 | 4.3 | 5.2.4 | 5.2.4 |
| Required mySQL | 4.1.2 | 4.1.2 | 5.0 | 5.0 |
Test Environment
For tests I have used WordPress 3.0.4, WordPress 3.1.4, WordPress 3.2.1 and WordPress 3.3. Test machine is CentOS powered VPS server with PHP 5.3.3 and mySQL 5.5.14, with PHP memory limit set to 256MB. PHP runs with eAccelerator enabled. All WP installations used only GD Press Tools 4.3.1 Pro plugin, and all had exactly the same content. GD Press Tools Pro had all its security related features activated, and they add about 3 SQL queries per page, and these queries are counted in the results below. Since I run GD Press Tools Pro as a part of all my WordPress installations, I think of it as logical and essential expansion of WordPress.
Each test for each WP version is repeated 10 times, highest and lowest results are removed, and the average is calculated from the rest of the measured values. Same method is used for server and client (browser) side. For client side measurements I have used Firefox 8.0, and before each measurement, all cached data was emptied. I use my faithful Lenovo IdeaPad Y560 with Intel Core i5 and 8GB RAM. Testing is done on 4 most used pages in WordPress: dashboard, posts list, post editor and comments list. Other pages are similar to these 4, and they will not show much difference in overall results.
If you compare results in this benchmark with previous one comparing 3.0 vs 3.1 vs 3.2, you will see that results for these 3 versions are now different. Difference comes from using latest versions of each WordPress, using differently configured server, and new browser. It is interesting to see how much faster is Firefox 8.0 over 5.0 used for old benchmark.
Server Side Execution
For client side I have measured 4 elements: used PHP memory to generate the page, number of SQL queries executed, time server needed to generate the page and number of hooks (both actions and filters) registered for execution. As with any version of WordPress, each new version uses more memory than previous one. But, while first 3 tested versions had number of executed SQL queries in decline, new WordPress 3.3 uses 4 to 9 queries per page more. Number of queries in the test includes 3-4 SQL queries executed by GD Press Tools Pro for security purposes, and it is the same for all 4 tested versions.
With all this, execution time on server-side for WordPress 3.3 is about 10%-15% higher than in earlier version. Number of hooks is expected to rise, just like the used memory. But again, there is a big jump with new WordPress. You can see all data on the charts below.
What this mean? In terms of server performance WordPress 3.3 is a big step backward from WordPress 3.2. I am not going into analysis why this happened, or if it was really needed, but the fact is that new WordPress doesn’t add any feature that warrants such change on the server-side.
Client Side Execution
Pages generated by new WordPress are only a bit larger than before, and the same goes for JavaScript used. Interesting thing is that on post edit page new WordPress adds less JavaScript, all thanks to new editor and quick tags code. Hopefully, we will see more JavaScript improvements like this in next versions. But, execution time is another story, and overall, this new version is slower even than WordPress 3.1 in some cases. Similar results are with cached execution, and new WordPress on average is slowest in the 3.x line.
As before, Firefox even with latest stable version 8.0 is still slower than Opera 11.6 or Chrome 16/17. It is not as drastic difference as before, but with Opera or Chrome, pages are loading some 20%-30% faster.
What these results mean for end-user page loading? The difference is not big, we are talking about half a second slower execution. That is hardly noticeable, but it proves the conclusion from the server-side benchmark: new WordPress is slower than previous one, while in the same time you will not see why that is needed, what new features are so important to justify this performance drop.
Overall performance index
Last chart shows two interesting, overall comparison indexes. First one is for server-side performance, and other for client side. They take into account individual results for each test and each page. WordPress 3.0.6 is referent version for this test and results for it are normalized to 1.0 for all pages and tests. Lower value is better.
As you can see, speed index for each of the pages shows that WordPress 3.2.1 remains superior to new WordPress 3.3 in terms of performance. If we want to compare overall for client and server-side, look at the last chart bellow. Overall, WordPress 3.3.0 managed to cling to second place thanks to some of the improvements, like WP Editor API and some other JavaScript changes, but it is very close to WordPress 3.1.4 performance. WordPress 3.3 is some 7% on server-side, and (too) big 14% on the client side off from WP 3.2.1.
Website front-end
I didn’t test new WordPress on the front-end. From using it in the past few weeks while in Beta and RC, I didn’t noticed any changes. I imagine that using some of the permalinks structures will improve speed due to permalinks changes, but other than that all 4 WordPress versions in 3.x line have very similar performance on the front end.
Conclusion
I can’t end this benchmark on a positive note as I did back in July for WordPress 3.2. I can understand that WordPress is evolving and that new features can need more resources. But, in WordPress 3.2 six months ago, features and changes were followed by considerable optimization, and that resulted in faster WordPress. This time, we got no significant new features or improvements, apart from cosmetic ones (and some of them are questionable at best) and some changes that affected only some features (uploader and editor), and yet new WordPress is gone back a full year in terms of performance, and it is comparable to WP 3.1 or even WP 3.0 in some cases.
I am sure that some developers will say that these results are not important. I agree that half a second is not much in terms of speed and that most users will not even notice the difference, but it shows that current development of WordPress is not done right. Adding new features and sacrificing speed and resource usage is not a good way to go. Big part of the blame is on development cycle, that is targeting 2 major versions each year (well, we got whole 3 major versions this year alone). Because of that, testing and optimization is not high on priorities list, developers don’t have enough time for everything. Even now, 3.3 is about one month late from projected development roadmap.
I really hope that this new WordPress performance is a ‘glitch’ in the development, and that next WordPress 3.4 will get it back on track. I also wish that WP 3.4 takes more time, and not to be rushed by May or June just for sake of having new WordPress. Lets have 9 to 12 months development cycles, and I am sure that released product will be better and more appealing to end users to get with faster upgrades.
For now, WordPress 3.2.1 is clear winner in terms of performance and resource usage between all 4 major releases in 3.x line.
Edit:
Server side Memory and Hooks charts replaced with new ones where Y axis starts from 0.
I don’t think the testing here is nearly thorough enough to justify the rather harsh conclusions.
On my test install, 3.3 runs a lower memory footprint and less queries on the dashboard, than 3.2.
There are too many variables here, like dashboard meta boxes, post meta boxes, number of comments, and the like.
Likewise, we’ve occasionally split one query into two to actually improve performance. Query counts are not a good way to judge performance, unless you start to see hundreds of queries, rather than 30 queries versus 31, or whatever it may be.
EAccelerator could be muddying the results. So could basic things like your server setting (or stripping) caching headers.
Many of the graphs start the vertical scale (Y-axis) at a rather high number, thus causing a line to be double in length for 3.3, when in reality, the change was around 5% in either direction.
Running GD Press Tools Pro alongside this test just doesn’t make sense. Sure, it’s the same code, but that same code might not perform the same on different versions of WordPress.
Some memory footprints don’t add up. I can tell you right now that the dashboard in 3.1 and the dashboard in 3.2 do not have the same memory footprint. But then again, it all depends on what’s cached in the database. Were transient caches cleared (or consistently primed) before testing?
Were other factors isolated? Could WordPress have been doing an update check during some tests, but not in others?
If there is really a “big step backward” I would love to see some better data.
Thanks for the comment Andrew. I have used same methodology for all 4 versions and same as with all previous benchmarks over the past 2 years, and previous WordPress did great as it did now. Why no one complained about the results back then? I understand that no one likes to see that new version is behind in terms of performance from old one, but that is the case here.
GD Press Tools Pro was running because it was used to do the tests. And I made sure that it had exactly the same impact on performance of all 4 versions. This included disabling (manually through code changes) anything that was specific for some of the 4 tested versions. So, it added same number of queries and used same memory on all 4. All 4 were tested on same server, under same conditions (no updates or other stuff were causing problems), no plugins for caching, and yes, all transient data was cleared before each test (there is a cleanup tool in GD Press Tools Pro for this). Also, each test was done 10 times and lowest and highest results were removed before getting average value from the rest 8 tests. And it took me almost 20 hours to test all 4 versions. I did my best to create same test environment for all 4 versions, and I listed what I did to make the tests. I know it might not be perfect, but it was the same for all 4 WordPress versions, and the results are presented as they are. And charts are not to blame, the Excel made them like that, and again, all 4 versions are on the chart, it is not discriminate to 3.3, and numbers are added bellow each chart.
As for my conclusions, I stand behind them. Maybe it was a bit harsh, but if you look performance improvements from 3.0 to 3.2, and than you get 3.3 with difference like this, I don’t know how else to call it. If you think you can make better test that would take more things into account I would like to see it. If you have some suggestions what to change in the testing process, I will do my best to use that next time.
And after all, performance is not the only thing that will get people to install new WordPress. If that was the case we would be stuck with 2.3 or 2.5 that were much faster and used less resources. Each new version is sum of both performance and features. If the features are good enough, people will be happy to make a trade off and sacrifice 5% of performance for much better features. If that is the case with WP 3.3, we will see in the next few months.
From a practical “in production” perspective (on 35+ sites) – It runs dog slow, period – Immediately after the upgrade. So much so, that for the first time (since WordPress launched), I’ve had more than one client ask to be moved away from the self hosted WordPress platform – For me at least, this is VERY alarming.
In my case, I am lucky, I backed up the databases and all files prior to the upgrades, so will be able to (relatively) downgrade back to 3.2.1 – And keep clients happy.
The author is not being harsh. Instead he is pointing out a serious performance issue that needs to be addressed. I’d rather have people who speak out, to help get things improved, than those who may not (as they may be afraid to appear as harsh).
Right after the upgrade, I would imagine that the PHP opcode cache needs to be rebuilt so the first request after the upgrade would take a while. I am not sure what would slow down subsequent requests though but I guess you’re experiencing the heavy processing requirements of WordPress too. While the WordPress platform is great for quick development, I wouldn’t host it on my old laptop like I would run some of my simple PHP scripts paired with static HTML files.
I have to agree with Andrew on this matter. Firstly i have not read the comments below although i would like you to make the hosting situation clear, was it a shared host? or was this a dedicated server? and if not then these results are absolutely invalid, as there is no way to control server load outside your own environment.
Unless results have been done in a 100% rigorous manner, your putting your name against something that does not stand up.
I have repeated this, but I have made sure to make the environment absolutely stable and isolated during the testing, I have used VPS server and all measurements repeated 10 times to make sure. Results stand, and most users I have talked with can confirm from practical experience that new WP on admin side is slower. I have no idea why people find it hard to believe these results, but they are correct. If you find better way to do the test I would like to see the results.
…as usually, i update my site when the RC version became available. No performance problem through all the RC period, but after installing the final version of the performance is significantly decreased. As reported by many online site speed tools.
Thanks for the comment. I have got some calls today by few clients that made the upgrade, and they also noticed slight performance changes after update. This is not much, but it can be measured. As my tests shows that is 7% or 8% average. It is not much, and it is not that noticeable during active work, but it is important to know the results of tests for any software product, so it can be improved later.
As Andrew Nacin noticed some charts were not starting with 0 value on Y axis. This is now corrected, and new charts for server side memory and hooks are generated to replace old ones.
Very interesting. Overall the 3.0 vs 3.1 vs 3.2 data looks pretty similar to the results I got when doing similar tests.
Just curious how many posts, comments and users were present for these tests? From personal experience, it does make a difference. For example, going from 3.1 to 3.2 some operations were slower on sites with small user tables (< 30 users), small comments tables (<5 comments per post on average). But on sites with larger datasets, those operations sped up.
Also, when running tests for relative performance, I would try and factor out network time altogether — even when testing against a virtual machine or localhost. Since all I actually cared about was the *relative* difference, it made the results more accurate, because I would often see a ~20ms difference in network latency.
Thanks for the comment Gabriel. Demo database used has 50 posts, 5 pages, 5 categories, 20 tags and 5 users/authors, all data is based on posts from Dev4Press, but modified to be used for a demo. I think that some query changes in WordPress 3.2 are better suited for websites with a bit more data.
As for the network latency, I have very fast Internet connection and very low latency on it (my test server wasn’t localhost, it was actual server). Firebug shows this in the loading timeline, and I was looking at that as well, and for these tests that wasn’t causing problems with measurement. I had very stable connection and virtually identical network latency for the duration of the tests. Normal latency oscillation wouldn’t make much difference anyway, since I used average ratings from 10 (8 used) measurements.
As I commented in a previous post, I can’t update WordPress automatically — but this is not what this comment is about. This time, I did the update, after being stuck with 3.0.5, I think, for months. (I just hope they don’t release 3.3.1 next week. hehehehe) The quickest time, of course, was to upload all files, substituting them. Since the zip file doesn’t have a wp-config.php, that works for me every time. But I had wondered something, and the table in the beginning of this article sheds a light. For instance: 3.2 has twelve more files than 3.3. So someone upgrading “my” way from 3.2 to 3.3 would be stuck with at least twelve unnecessary files, and possibly more. The only way I can think of to find them would be checking every folder and subfolder, looking for files with a different date. And even this is very far from a flawless method. Probably not a huge problem (I’m no specialist), but definitely something that should get a future manual maintenance.
Thanks for the comment. Yes, that is the problem when auto upgrade is not an option. phpBB had some script generator a while back to get differences between installed phpBB and update to download only files that you need and to delete extra files. That would be useful in WordPress for cases like this. You get a archive with changes only, and list of files/folders to delete without missing some in the manual process.
When I made the upgrade to 3.0.5 I did it manually, just replacing the files that were listed as changed from 3.0.4, since it were under a dozen, if I recall correctly. Since now I was going all the way from 3.0.5 to 3.3, there was no way I could think of to list every file that was changed, so the future “manual” maintenance will be required. But just because there will come a time when I will be bothered by the knowledge of all those loose files just being there. 🙂
There is actually a way you can determine what files should be removed. Download locally 3.0.5 and 3.3.0, unpack them in separate folders, and use some compare program like KDiff or WinMerge to find what files are missing from new one.
Also, good to see that you are moving from old 3.0 release to latest WordPress. Hopefully more users and websites will do the same despite WP 3.3 performance drop, it is still faster than 3.0. It is time to retire 3.0 line.
Thanks for the benchmarks, very useful for me.
When there is a new WordPress release I usually wait a few weeks before upgrading to make sure the plugins I use are working properly, but I may skip WordPress 3.3 completely as I already have server load issues on my current WordPress 3.2 installation.
Thanks for the comment. I would recommend you try some caching plugins (if you don’t use that already) to try and improve server load problems. Such plugins can do much to improve things. I always use W3 Total Cache.
Thanks for the tip. I also use W3 Total Cache, CloudFlare and try to optimize the database regularly (with Better Delete Revision plugin) in order to try to reduce the load.
I’d be really careful with that.. If there’s a major bug found that effects WP 3.2, they will fix it in 3.3.x. That means if you don’t keep up with the latest version of WordPress, you will be vulnerable.
…there is in 3.3 a way to safely remove /blog/ from multisite installation?
That is the first time I see a benchmark done on a back-end like this!
I agree with Andrew that the data does not show much. Anyway, to me, no matter if the stats are good or not, I just don’t care. A pure non-sense to me.
I would be really impressed to find only one user would seriously analyze this data before making a decision whether (s)he should update or not.
I would have surely enjoyed a front-end benchmark with a theme that has no image, css or javascript (to test only execution and not loading). I just can’t recall the name of this theme but it’s provided by WordPress or Automattic. That would have brought a plus-value.
Just my 2 cents!
Thanks for the comment. As I said before, benchmark is only one of the things to consider with every update, and it doesn’t mean much if you like what the update brings. But, considering how many users are having problems and reporting speed issues on the admin side with WP 3.3, that indirectly confirms the tests results. I don’t get why some people take it badly to find out that new version is slower than previous one, but remained silent 6 months ago when same test showed (as this one does) that WordPress 3.2 was significant improvement. It is fair to take results as they are, and admit that new WordPress is not as fast as previous one, after all the difference is not that bad, just about 6-7%. Problem is that this is first time after WP 3.0, that we got UI for new WP put before the performance, and I am sure that most WordPress users will not like that.
As for the frontend, there is almost no difference in speed and resource usage between WP 3.1, WP 3.2 and WP 3.3. I did some preliminary tests, and those results didn’t show anything important. I imagine that WP 3.3 would be faster when post name permalinks are used, but other than that, no changes are made to make the difference. I will try to find some time in the next 2 weeks to complete at least some of the front end tests, and I will publish those also.
1) Why not benchmark a fresh install (you have a plugin installed and who knows what impact it can/could have) ?
2) If your conclusion was “WordPress’ back-end is slow”, yes, we could debate. That said, the provided stats do not show me any clear sign that it’s going in a bad direction.
3) I’ve not seen any stats about how many pages, posts or comments you have.
I am not taking it badly! I’ve, personnally, never had any loading issue with my admin section that would lead me to think WordPress is THE problem.
That said, I would love to see a “slow” working WordPress install and investigate the “why”. I would not be surprised to see that the slow loading is caused by something else.
That said, there is surely room for improvement!
Cheers.
Sorry about that, but my remark about ‘people taking results badly’ was not directed at you.
1. Plugin was installed on all 4 installations, and I made sure that it had same impact on all 4. Also, plugin was actually used to get benchmark results.
2. Yes, that is thing to debate. I have been working with WP for 5 years now, and I make my living on WordPress, and I don’t like the current development direction of WP and I am afraid that this test results are only the start. I will wait for the official announcement for 2012 development cycle, but I am afraid that putting UI changes up front will continue. UI changes almost always bring speed performance issues.
3. I have listed stats on the demo site content in the comments above, here it is again: 50 posts, 5 pages, 5 categories, 20 tags and 5 users/authors. I am sure that results will be different depending the the size of the database, but I can’t make tests that broad, I would need weeks to do it.
As for the slow WP sites, I have detailed measurements for Dev4Press website over period of last 3 months, including server load data and few other things (since I am only one who can access admin side, I have debugger running on admin side non stop), and I have average 9% slower load time, 5% memory usage increase on admin side comparing last 4 days with WP 3.3 and average 0.3 seconds slower loading of all admin pages (I use same browser for last 3 months, Opera 11.51). I have talked with many of my clients and users, and very few have said that they don’t see the difference, most of the other confirm without any reservation that new WP is slower on the admin side. That said, many have confirmed that they reverted back to WP 3.2.1. Also, check some of the threads in WP.org forum to see similar experiences there.
I really don’t know what exactly why is new WP slower, but I would not call it a problem either. Slow down is caused by the changes and features added. I am writing a review of new WP for next week, and I will try to cover everything I can say about new WP. Even with all this, I still think that WP 3.3 is overall a good release: we got new uploader, new editor API and some interesting core changes. I don’t like the UI changes, and I think that they are responsible for performance drop, but I know that many users will like the UI changes. My biggest problem is that WP development team is focusing too much on the UI, not enough on the really important things like posts/taxonomies/terms management and posts relations. As long as we have short 5-6 months release cycles, UI will always be more important than other things.
Milan
I’m surprised by certain highly critical voices regarding the methodology of this benchmark and the validity of its results.
Let’s divide the results into two categories: performance related ( execution time, page load time ) and statistical data ( memory used, number of queries, number of hooks ). I’ve read this post many times and I did not find any place where Milan states that memory footprint, number of queries, or number of hooks affected the performance in any particular way. He only refers to their numbers. Those numbers can be used later to find out how they MAY affect performance but this is beyond the scope of this discussion. Andrew said: “Query counts are not a good way to judge performance, unless you start to see hundreds of queries, rather than 30 queries versus 31, or whatever it may be.” Agreed 100%, but I did not notice that such a judgement was expressed by Milan in his text.
Andrew, I do not understand your statement: “Running GD Press Tools Pro alongside this test just doesn’t make sense. Sure, it’s the same code, but that same code might not perform the same on different versions of WordPress.” This plugin was used for recording of data only. Are you saying that you can come up with a measuring software which will read timers differently and therefore show different values? It is not that I don’t agree with other points you’ve made. I wouldn’t use eAccelerator either and update checks not being disabled could affect the results. Being sensitive and concerned is understandable. You are a co-father of this release, after all.
When it comes to performance it is indisputable that 3.3 is slower than 3.2. You can see it with a naked eye. We all know that development of 3.2 was focused on optimization and performance and that job was very well done. Nobody said that 3.3 is “bad” or outright “slow” as Maxime suggests. 3.3 is relatively slower than 3.2 and that is the fact. So please, do not kill the messenger. Milan did an excellent job doing the test and presenting its results. If you know of other WordPress benchmarks in existence, tell us.
Thanks for the comment Frank!
As you say WP 3.2 was focused on performance improvements and it succeed. Now we get new UI centric release with mostly unneeded changes that completely invalidated all the work done with WP 3.2 on resource usage front. And that is my biggest concern at this point.
Any chance for an updated benchmark with WordPress 3.5, 3.4 and 3.3? Thanks in advance!
Interesting benchmarks, we haven’t done similar benchmarks, but our customers are noticing things being faster actually.
I am curious on the benchmark setup.. did nothing change between the tests on the harness?
Same version of PHP, Apache/Nginx/PHP ? I think you said PHP 5.3.3.. why such an old version?