Cache all CakePHP find queries

I have been adding some cache to Infinitas recently and found it rather difficult to cache all find queries without having to overload and rewrite the entire Model::find() method. While its not the most complicated method in the world, its not DRY rewriting it and some time in the future its likely to bite me in the arse when something changes.

A basic way to cache the queries is as follows:

  • Create a hash of the query array
  • Check the cache using the hash as a key
    • If the cache exists return it
  • If the cache did not exist fetch the results
  • Save the results using the has as a key
  • Return the results

The problem comes when using custom find methods as the call is done in two stages. First CakePHP will call the method with the $state set to before which is where query params can be processed and manipulated. Cake will then run the actual query and run the custom find a second time with the $state set to after, which is where the actual results are processed.

The way find was set up it was not easy to cache based on the query params as they were not easily accessible after the custom find query had manipulated the query. So using the ideas from above the hash of the query my not be correct as the custom find methods before calls may have modified the query.

I submitted a patch (and its merged!) which abstracts the find out a bit and makes it much easier to cache find queries now. Here is an example of how to cache all find queries (this is probably not a good idea as you could end up with a cached version of every page of pagination, sorted by ever column with every limit possible). All that needs to be done now is overload the _readDataSource() method where CakePHP fetches the data and check for cache.

protected function _readDataSource($type, $query) {
	$cacheName = md5(json_encode($query));
	$cache = Cache::read($cacheName, 'cache-config-name');
	if ($cache !== false) {
		return $cache;
	}

	$results = parent::_readDataSource($type, $query);
	Cache::write($cacheName, $results, 'cache-config-name');
	return $results;
}

So that is pretty simple having to only overload one method where the data is fetched from the data source. Everything in moderation so its probably best to limit what is cached. Filtering out a few cases like that means you wont need 4 million gigs of ram for cache and only the less frequent queries would not be cached.

Some things you might want to skip

  • Results that are paginated (possible that queries have the page param here.
  • Results sorted differently from the default
  • Results with a custom limit (eg: limit != 20)
  • Results with a high limit (eg: limit >= 100)

Its also easy to cache only queries that are find('list') for example using $this->findQueryType as a check. Another good idea is to check for $query['cache'] and then in your code you can do something like the following (the overloaded method would need to be modified to take these options into consideration, the example above does not):

Don't cache this query
$this->find('all', array(
    'cache' => false
));
Custom cache config
$this->find('all', array(
    'cache' => array(
        'name' => 'other_cache',
        'duration' => '+2 weeks'
    )
));

Read more...

How to convert SQL to CakePHP find

There are a lot of newcomers asking the same question on CakePHP's IRC channel, Stack Overflow and various other places, and that question is how to convert a SQL query into the corresponding CakePHP code.

Once you have been using CakePHP for a while the ORM becomes second nature and its pretty easy to see how a query will fit into a Model::find(). I normally break the SQL up into the various …

Read more...

Batch convert NEF (raw) to PNG

Here is a simple script for converting raw NEF files from a Nikon to PNG format. It uses the convert utility which can handle a large variety of file types and has various options for the conversion. This is just a simple script that will generate a PNG that is either 1024 px wide or high.

#!/bin/bash
##################################
#      Convert files to png      #
##################…

Read more...

Nikon raw images on Ubuntu 12.04

Since switching to Ubuntu 12.04 sometime last year I have not been able to see thumbnails of images shot in raw format. Specifically I have a Nikon camera that shoots NEF files which even after installing various tools such as UFRaw still did not work. Eventually I wrote a small bash script for converting NEF to PNG so I could see what the image was without having to open each one in UFRaw first.…

Read more...

No more posts