bigblobae – store big data blobs on AppEngine

by jan on February 9th, 2010

Few days ago an idea finally came to mind that made sense for the new (maybe not so new anymore) Cloud-Computing thingi. The idea was called image-resizer and feel free to check it out. I am still not quite sure if I will take it past the cloud-computing experiment phase, but it was still quite fun to develop. I have hit several issues with AppEngine along the way, most of which were related to sandbox/production environment differences. Not pretty, but easy to solve.

One particular AppEngine limitation is that you can only store one megabyte of data in any datastore entry. Basically any row in the database cannot be bigger and since the only storage you have is the database, you are kinda screwed (like this guy, who jumps first for most of related Google searches). This is particularly painful in case you want to store big files in the database.

You are screwed until you start thinking that is. The limitation is really not a limitation of the storage, more of the transfer layer, the storage can hold virtually any amount of data, so all you have to do is fragment it a little. And here comes Mr. Smartypants. I have created something called bigblobae (read big blobs for app engine). I am certain there are other maybe smarter implementation of this out there, but this is the way I like it, simple.

So if you want to store your large photos, pdfs or whatever in your AppEngine application, checkout the 0.1 release of bigblobae or you can just download the binary.

From Blog

7 Comments
  1. Nice stuff!

    You might be interested in this post:

    http://blog.jteam.nl/2010/02/04/free-java-hosting-with-the-google-app-engine/

    It covers some more “workarounds” for Google’s app engine for Java.

    -Stefan.

  2. Evan permalink

    This was great.

    I just downloaded and started to experiment. I’m wondering will you support owned-to-many relationship in future?

    I had Entity1 has many Entity2 which Entity2 has BigBlob type field. An exception was raised while trying to persist Entity1: ‘Entity2 has multiple relationship fields of type org.honzasterba.bigblobae.BigBlob: thumb and image. This is not yet supported’. Or is this maybe app engine bug?

    Can you please point me?

    thanks,

    evan

  3. Evan permalink

    nvrmind my prev comment. it was the app engine bug. i added and it solved it. thanks again!

  4. NotJim permalink

    Hi,

    Have you tried storing more than 10Mb with this approach?
    Google File Service had a similar implementation.

  5. @NotJim: I have not tried files larger than approx 3,5 megs. The implementation should be able to handle files of any size, but the issues comes in request processing time. Since the time your app can spend in one request is fairly limited, the limit of the datastore now transfers to the pipes between your app and the client.

  6. Hi, Have you tried storing more than 10Mb with this approach? Google File Service had a similar implementation.

  7. No I did not, but it should work. File service was introduced only recently.

Leave a Reply

Note: XHTML is allowed. Your email address will never be published.

Subscribe to this comment feed via RSS