Download - Introduction to Datastore
Introducion to Datastore
Assoc.Prof. Dr.Thanachart NumnondaAsst.Prof. Thanisa Kruawaisayawan
Mini Master of Java TechnologyKMITL
July 2010
Agenda
What is DataStore?Using DataStoreJPA in DataStore
What is DataStore?
What is Datastore?Google App Engine Datastore is a schema-less persistence
system, whose fundamental persistence unit is called Entity, composed by an immutable Key and a collection of mutable properties.
Entities can be created, updated, deleted, loaded by key and queried for properties values.
DataStore is consistent and transactional, with support to current transaction.
The DataStoreThe Datastore is not a relational database nor a
façade.Relational database technology doesn’t scale
horizontally– Connection pools, shared caching are a problem
The Datastore is one of many public APIs used for accessing Google’s
The DataStore
The DataStore
The DataStore : OperationsTransactions and Index are based on MegaTable.File persistence it's done with Google File System
(GFS).It's distributed by Chubby, a lock service for loosely-
coupled distributed systems.
BigTableBigTable is a compressed, high performance, and
proprietary database system built on Google File System (GFS), Chubby Lock Service, and a few other Google programs
Currently not distributed or used outside of Google.BigTable development began in 2004. and is now used
by a number of Google application Google Earth, Google Map, Gmail, Youtube, etc..
BigTable : DesignBigTable is a fast and extremely large-scale DBMS. It is a sparse, distributed multi-dimensional sorted map,
sharing characteristics of both row-oriented and column-oriented databases.sparse because only "not null" values are persisteddistributed in Google cloud persistent on Google File Systemmultidimensional in columns valuesordered lexicographically by key
BigTable : Design
Tables are optimized for GFS by being split into multiple tablets - segments of the table.
BigTable is designed to scale into the petabyte.Each table has multiple dimensions (one of which is a
feld for time, allowing for versioning and garbage collection).
It allows an infnite number of rows and columns.
Google File SystemGFS is a proprietary distributed fle system developed
by Google.It is designed to provide effcient, reliable access to
data using large clusters of commodity hardware.GFS grew out of an earlier Google effort, BigFiles,
developed by Larry Page and Sergey Brin in the early days of Google, while it was still located in Stanford.
Using DataStore
DataStore OperationsDatastore operations are defned around entities (data
models) which are objects with one or more propertiesTypes: string, user, Boolean, and so onEntities may be recursive or self-referential
Entity relationships are one-to-many or many-to-many.Entities may be fxed or grow as needed.
DataStore Storage ModelEvery entity is of a particular kindEntities in a kind need not have the same propertiesOne entity may have different “columns” from another in
the same kind!Unique IDs are automatically assigned unless the user
defnes a key_name
Compare DataStore with Others
DataStore Storage ModelBasic unit of storage is an Entity consisting of
Kind (table)Key (primary key)Entity Group (partition)0..N typed Properties (columns)
Datastore QuotasEach call to Datastore counts towards the quotaThe amount of data cannot exceed the billable
Includes properties and keys but not the indices
CPU and Datastore CPU time quotas apply
Using the DatastoreApplications may access the Datastore using the JDO
or the JPA classes.The JDO and JPA classes are abstracted using the
DataNucleus APIOpen source Not very popular Support for Java standards Poor documentation
JPA in DataStore
Setting Up JPAThe JPA and datastore JARs must be in the app's
war/WEB-INF/lib/ directory. A confguration fle named persistence.xml must be in
the app's war/WEB-INF/classes/META-INF/ directory,A confguration fle tells JPA to use the App Engine
datastore.The appengine-api.jar must also be in the war/WEB-
INF/lib/ directory.
persistence.xml: Example
<?xml version="1.0" encoding="UTF-8"?>
<persistence version="1.0" xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd">
<persistence-unit name="thaijavaappPU" transaction-type="RESOURCE_LOCAL">
<provider>org.datanucleus.store.appengine.jpa.DatastorePersistenceProvider</provider>
<non-jta-data-source/>
<properties>
<property name="datanucleus.ConnectionURL" value="appengine"/>
<property name="datanucleus.NontransactionalRead" value="true"/>
<property name="datanucleus.NontransactionalWrite" value="true"/>
</properties>
</persistence-unit>
</persistence>
<?xml version="1.0" encoding="UTF-8"?>
<persistence version="1.0" xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd">
<persistence-unit name="thaijavaappPU" transaction-type="RESOURCE_LOCAL">
<provider>org.datanucleus.store.appengine.jpa.DatastorePersistenceProvider</provider>
<non-jta-data-source/>
<properties>
<property name="datanucleus.ConnectionURL" value="appengine"/>
<property name="datanucleus.NontransactionalRead" value="true"/>
<property name="datanucleus.NontransactionalWrite" value="true"/>
</properties>
</persistence-unit>
</persistence>
Getting an EntityManager InstanceAn app interacts with JPA using an instance of the EntityManager.
import javax.persistence.EntityManagerFactory;
import javax.persistence.Persistence;
public class EMF {
private static final EntityManagerFactory emfInstance =
Persistence.createEntityManagerFactory("transactions-optional");
public static EntityManagerFactory get() {
return emfInstance;
}
}
import javax.persistence.EntityManagerFactory;
import javax.persistence.Persistence;
public class EMF {
private static final EntityManagerFactory emfInstance =
Persistence.createEntityManagerFactory("transactions-optional");
public static EntityManagerFactory get() {
return emfInstance;
}
}
Entity Class : Example@Entity
public class GuestList implements Serializable {
…
@Id
private String id;
@Basic
private User author;
private String content;
@Temporal(javax.persistence.TemporalType.DATE)
private Date visitDate;
…
// Getter and Setter methods
}
@Entity
public class GuestList implements Serializable {
…
@Id
private String id;
@Basic
private User author;
private String content;
@Temporal(javax.persistence.TemporalType.DATE)
private Date visitDate;
…
// Getter and Setter methods
}
Queries and IndicesA query operates on every entity of a given kind.
Specify zero or more sort ordersSpecify zero or more flters on property values
Indices are defned in the App Engine confguration flesResults are fetched directly from these indices; no indices are
created on the flyWEB-INF/datastore-indexes.xml - non-standard fles
Normalization is not recommendedOptimization techniques for RDBMSs may result in poor
Datastore performance!
Query : Example EntityManager em = EMF.get().createEntityManager();
try {
Query query = em.createQuery("SELECT o FROM GuestList AS o");
@SuppressWarnings("unchecked")
List<GuestList> results = (List<GuestList>) query.getResultList();
for (Object obj : results) {
GuestList guest = (GuestList) obj;
String nickname = guest.getAuthor().getNickname();
out.println(nickname + " " + guest.getId());
}
} catch(Exception ex) {
out.println(ex);
}
EntityManager em = EMF.get().createEntityManager();
try {
Query query = em.createQuery("SELECT o FROM GuestList AS o");
@SuppressWarnings("unchecked")
List<GuestList> results = (List<GuestList>) query.getResultList();
for (Object obj : results) {
GuestList guest = (GuestList) obj;
String nickname = guest.getAuthor().getNickname();
out.println(nickname + " " + guest.getId());
}
} catch(Exception ex) {
out.println(ex);
}
Entity Relationships
Models association between entities.There are four types of relationship multiplicities:
@OneToOne@OneToMany@ManyToOne
Supports unidirectional as well as bidirectional relationshipsUnidirectional relationship: Entity A references B, but B doesn't
reference A.
Example : ManyToOne Mapping
Example : OneToMany Mapping
Transactions and Entity GroupsTransaction = Group of Datastore operations that either
succeed or failEntity groups are required because all grouped entities are
stored in the same Datastore nodeAn entity may be either created or modifed once per
transactionTransactions may fail if a different user or process tries an
update in the same group at the same timeUsers decide whether to retry or roll the transaction back
Transaction in JPA : Example
Book book = em.find(Book.class, "9780596156732");
BookReview bookReview = new BookReview();
bookReview.rating = 5;
book.getBookReviews().add(bookReview);
Transaction txn = em.getTransaction();
txn.begin();
try {
book = em.merge(book);
txn.commit();
} finally {
if (txn.isActive()) {
txn.rollback();
}
}
Book book = em.find(Book.class, "9780596156732");
BookReview bookReview = new BookReview();
bookReview.rating = 5;
book.getBookReviews().add(bookReview);
Transaction txn = em.getTransaction();
txn.begin();
try {
book = em.merge(book);
txn.commit();
} finally {
if (txn.isActive()) {
txn.rollback();
}
}
Unsupported Features of JPAOwned many-to-many relationships, and unowned
relationships."Join" queries.Aggregation queries (group by, having, sum, avg, max, min) Polymorphic queries.
ResourcesGoogle App Engine for Java HOWTO, Andrew Lombardi, Mar
2010The Softer Side Of Schemas, Max Ross, May 2009Official Google App Engine Tutorial,
http://code.google.com/appengine/docs/java/gettingstarted/Programming Google App Engine, Don Sanderson, O'Reilly,
2010