bigblobae – store big data blobs on AppEngine
Few days ago an idea finally came to mind that made sense for the new (maybe not so new anymore) Cloud-Computing thingi. The idea was called image-resizer and feel free to check it out. I am still not quite sure if I will take it past the cloud-computing experiment phase, but it was still quite fun to develop. I have hit several issues with AppEngine along the way, most of which were related to sandbox/production environment differences. Not pretty, but easy to solve.
One particular AppEngine limitation is that you can only store one megabyte of data in any datastore entry. Basically any row in the database cannot be bigger and since the only storage you have is the database, you are kinda screwed (like this guy, who jumps first for most of related Google searches). This is particularly painful in case you want to store big files in the database.
You are screwed until you start thinking that is. The limitation is really not a limitation of the storage, more of the transfer layer, the storage can hold virtually any amount of data, so all you have to do is fragment it a little. And here comes Mr. Smartypants. I have created something called bigblobae (read big blobs for app engine). I am certain there are other maybe smarter implementation of this out there, but this is the way I like it, simple.