How the internal cache works?

Whenever WordPress runs function(s) that get posts, pages, categories or any other data object, result is than stored in instance of the WP_Object_Cache class. Each object is stored in it’s own group and it’s identified by ID. So, if you get post ID 1, it will be stored in this class instance. Next time on the same page, you need again same post, WP will search for it in the cache, and if it’s there it will get it for you, if not SQL query will be run to get it. Depending on number of stored objects, this cache can grow significantly.

Instance  of class WP_DB is main database object, and also stores all SQL queries executed through it. Most WP pages run 10 to 200 or so queries, and queries are stored as strings. With small number of queries this object can grow to couple of megabytes. For this object you can disable saving queries, but in many cases plugins will enable it, so you can’t be sure really that queries are not stored.

And all this is great if you only have code to generate pages, normal admin use, in fact normal use of  WordPress. If you have couple of hundreds or even thousands of posts/pages, none of the normal WP operations will cause problems with use of cache. But, what if you have website with 200.000 or more posts?

Out of memory problems

For large websites, first piece of code that will fail even if you use 256M for PHP is Google XML Sitemaps plugin when attempts to make a XML file. To get URL for each post, it will run get_permalink() function, that is running get_post() that is caching objects. So, each post will be cached even if it’s used only once. And when plugin reaches 70.000 posts (approx.), PHP will run out of memory, and all 256M will be used! Not only posts, but all SQL queries will be stored. And for 70.000 posts, in this case WP can run more than 100.000 queries.

I am working on website that has 300.000 posts, sitemap generating  crashes at 65.000 with 256M used up. So, I made a new code that controls sitemap: splits sitemap in new file at every 50.000 posts (Google limit per sitemap is 50.000), and also is clearing cached data to prevent out of control use of memory.

Solution for clearing cache

Website I work on has several cleanup and processing operations that will generate large amount of queries and cache many objects during each run. Using cache will break it for sure on every run. But, in WordPress right now there is no way to disable cache. It would be good that directive like WP_DISABLE_CACHE is added to control this. And until that is done, only way to control it is actually to delete cached object. So, my processing code and also sitemaps code is using simple function to clear cache from time to time.

Here is the function that will clear cache object and stored queries:

<?php
function wp_clean_cache_full() {
  global $wpdb, $wp_object_cache;

  unset($wp_object_cache->cache);
  $wp_object_cache->cache = array();
  unset($wpdb->queries);
  $wpdb->queries = array();
}
?>

Conclusion

This is not something that you will need all the time, but it’s good thing to know what can cause website crashes that you can’t understand at first. Out of memory problems are common with large websites if the data is not handled properly.