In redis, every object is either a binary-safe string or collection thereof.  Even if you’re storing numbers from your client library or using the INCR/DECR counter within redis, the actual values are stored in memory as strings.  In some cases, this may be fine; however, if you are storing many numbers, whole or decimal, this may add up.

  If you’re using python to store and retrieve data from redis, there’s an easy way to save loads of memory – the built-in struct library.  struct handles converting python types to binary representations of underlying C types.  In the case of numbers, this means that we can quickly and painlessly convert that sequence of characters into the underlying binary representation, which is quite often smaller.  The simple pattern below shows an example of handling non-negative whole numbers.

for number in numbers:
	if number < 2**8:
		rp = struct.pack('B', number)
	elif number < 2**16:
		rp = struct.pack('H', number)
	elif number < 2**32:
		rp = struct.pack('I', number)
	else:
		rp = struct.pack('L', number)

  In a simple test of 1M integers uniformly distributed between 1 and 10^5, this pattern resulted in a 45% reduction in memory usage. While there’s no hard-and-fast rule, I’ve had at least 25% reductions in almost every real-world application of this technique. YMMV, but given the ease of this hack and the cost of memory, it’s definitely worth a shot!

(If you have a better handle on the range or structure of data and aren’t afraid of math, you should also look at rolling your own encoding with GETRANGE or GETBIT. More to come on this in a future post.)