Ikai Lan says

I say things!

Archive for June 3rd, 2010

Introduction to working with App Engine’s low-level datastore API

with 9 comments

App Engine’s Java SDK ships with three different mechanisms for persisting data:

  • JPA – the javax.persistence.* package
  • JDO – Java Data Objects
  • The low-level API

The formal documentation has got some good examples for working with JDO and JPA, but the documentation for working with the low-level API is still a tad sparse. The original purpose of the low-level API was to provide developers a way to build libraries that could do persistence or even build persistence libraries themselves – alternative persistence mechanisms such as Objectify, Twig, SimpleDS and Slim3 all build on top of this API.

For most developers, it may be simpler to use either JDO, JPA or a third-party library, but there are cases in which the low-level API is useful. This post will be a beginner’s guide to writing and retrieving data using this API – we’ll save more advanced topics for future posts.

For those newer to App Engine, let’s define a few terms before we continue:

Entity – An entity is an object representation of a datastore row. Unlike a row in a relational database, there are no predefined columns. There’s only one giant Bigtable, and your entities are all part of that table.

Entity Kind – There are no tables corresponding to types of data. The Kind of the entity is stored as part of the Entity Key.

Entity Key – The primary way by which entities are fetched – even when you issue queries, the datastore does a batch get by key of entities. It’s similar to a primary key in an RDBMS. The Key encodes your application ID, your Entity’s Kind, any parent entities and other metadata. Description of the key is out of scope of this article, but you’ll be able to find plenty of content about Keys when you refer to your favorite search engine.

Properties – Entities don’t have columns – they have properties. A property represents a field of your Entity. For instance, a Person Entity would have a Kind of Person, a Key corresponding to a unique identifier corresponding to their name (for all real world scenarios, this is only true for me, as I’m the only Ikai Lan in the world), and Properties: age, height, weight, eye color, etc.

There are a lot more terms, but these are the ones we’ll be using frequently in this article. Let’s describe a few key features of the low-level API which differ from using a higher level tool such as the JDO and JPA interfaces. Depending on your point of view, these could be either advantages or disadvantages:

  • Typeless entities. Think of an Entity as a Java class with a Key property (datastore Key), a Kind property (String) and Properties (HashMap of Properties). This means that for a given entity kind, it is possible to have two different entities with completely different properties. You could have a Person entity that defines age and weight as its properties, and a separate Person entity that defines height and eye color.
  • No Managers. You instantiate a DatastoreService from a DatastoreServiceFactory, then you get(), put() and query()*. No need to worry about opening or closing managers, detaching objects, marking items dirty, and so forth.
  • Lower startup time. For lower traffic Java websites, loading a PersistenceManagerFactory or EntityManagerFactory can incur additional startup time cost.

We’ll cover queries in a future post. In this post, we’ll just use get() and put(). In this article, we’ll treat App Engine’s datastore as if it were just a giant Map. Frankly, this isn’t a bad simplication – at its lowest level, Bigtable is a key-value store, which means that the Map abstraction isn’t too far from reality.

Let’s create two Entities representing Persons. We’ll name them Alice and Bob. Let’s define them now:

import com.google.appengine.api.datastore.Entity;
import com.google.appengine.api.datastore.Key;
import com.google.appengine.api.datastore.KeyFactory;

Entity alice = new Entity("Alice", "Person");
alice.setProperty("gender", "female");
alice.setProperty("age", 20);

Key bobKey = KeyFactory.createKey("Person", "Bob");
Entity bob = new Entity(bobKey);
bob.setProperty("gender", "male");
bob.setProperty("age", "23");

What we’ve demonstrated here are two of the basic ways to create entities. Entity contains five constructors. We’re just demonstrating two of them here.

We’re defining Alice with a raw constructor. We’re passing two Strings: her key name as well as her kind. As we mentioned before – Entities are typeless, and we can specify just about any String as her type. Effectively, this means that the number of kinds we can have is limited only by the number of kinds that we need, and as long as we don’t lose track of them, we could potentially have hundreds of different kinds without having to create a class for each one. We could even define new kinds at runtime, if we so dared. The key name is what we’ll use to retrieve Alice later on when we need her again. Think of it as a Map or Dictionary Key. Once we have an Entity object, we need to define her properties. For now, we’ll define her gender and her age. Note that, again, Properties behave like Maps, and this means that not only can Entities have hundreds of types of different properties, we could also create new properties at runtime at the expense of compiler type-safety. Choose your poison carefully.

We’re creating Bob’s instance a bit differently, but not too differently. Using KeyFactory’s static createKey method, we create a Key instance. Note the constructor arguments – they are exactly the same: a kind and a key name. In our simple example, this doesn’t really give us any additional benefits except for adding an additional line of code, but more advanced usages in which we may want to create an Entity with a parent, this technique may result in more clear code. And again – we set Bob’s properties using something similar to a Map.

If you’ve been reading Entity’s Javadoc or following along in your IDE, you’ve probably realized by now that Entity does not contain setKey() or setKind() methods. This is because an Entity’s key is immutable. Once an Entity has a key, it can never be changed. You cannot retrieve an Entity from the datastore and change its key – you must create a new Entity with a new Key and delete the old Entity. This is also true of Entities instantiated in local memory.

Speaking of unsaved Entities, let’s go ahead and save them now. We’ll create an instance of the Datastore client and save Alice and Bob:

import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.appengine.api.datastore.Entity;
import com.google.appengine.api.datastore.Key;
import com.google.appengine.api.datastore.KeyFactory;

Entity alice = new Entity("Person", "Alice");
alice.setProperty("gender", "female");
alice.setProperty("age", 20);

Key bobKey = KeyFactory.createKey("Person", "Bob");
Entity bob = new Entity(bobKey);
bob.setProperty("gender", "male");
bob.setProperty("age", "23");

DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
datastore.put(alice);
datastore.put(bob);

That’s it! DatastoreService’s put() method returns a Key that we can use.

Now let’s demonstrate retrieving Alice and Bob by Key from another class:

import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.appengine.api.datastore.Entity;
import com.google.appengine.api.datastore.EntityNotFoundException;
import com.google.appengine.api.datastore.Key;
import com.google.appengine.api.datastore.KeyFactory;

Key bobKey = KeyFactory.createKey("Person", "Bob");
Key aliceKey = KeyFactory.createKey("Person", "Alice");

DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
Entity alice, bob;

try {
    alice = datastore.get(aliceKey);
    bob = datastore.get(bobKey);

    Long aliceAge = (Long) alice.getProperty("age");
    Long bobAge = (Long) bob.getProperty("age");
    System.out.println(”Alice’s age: “ + aliceAge);
    System.out.println(”Bob’s age: “ + bobAge);
} catch (EntityNotFoundException e) {
    // Alice or Bob doesn't exist!
}

The DatastoreService instance’s get() method takes a Key parameter; this is the same parameter we used earlier to construct the Entity representing Bob! This methods throws an EntityNotFoundException. We retrieve individual properties using the Entity class’s getProperty() method – in the case of age, we cast this to a Long.

So there you have it: the basics of working with the low-level API. I’ll likely add more articles in the future about queries, transactions, and more advanced things you can do.

Advertisements

Written by Ikai Lan

June 3, 2010 at 6:46 pm