Using JDO 2.0: JDOQL Part I

Welcome to Part I of my four-part series on JDOQL. The articles in this series illustrate the new capabilities which the forthcoming JDO 2.0 standard is bringing to JDOQL. 
The series is comprised of:

JDOQL Part I - New operator and method support, paging of query results, and datastore-delegated "deletion by query". 
JDOQL Part II - Projection, aggregation and the grouping ad de-duplication (making distinct) of query results. 
JDOQL Part III - A new single-string representation of JDOQL, along with implicit parameter and variable definitions for more concise queries. 
JDOQL Part IV - Using SQL as a query language - power at the expense of tight coupling between your code and the database schema.

This particular article uses the separately discussed domain object model and data set for Hammer.org - The Online Auction Domain. Here is a quick link to the UML diagram for the persistent domain.

Other useful resources include SolarMetric's 2-page JDOQL 2.0 Quick Reference and the unprintable PDF Edition of my own book, Java Data Objects (Addison-Wesley).

Filter Basics

Several improvements have been made to the set of supported operators for JDOQL filters, as well as support for additional method invocations on String, Collection and Map types.

Intro to JDOQL

Some readers may want more background on JDOQL before looking at the improvements delivered by JDO 2.0. A good treatment of JDOQL is available in several books dedicated to the JDO standard. The unprintable PDF Edition of my own book, Java Data Objects (Addison-Wesley) is freely downloadable. Go straight to Chapter 8 for discussion of JDOQL.

JDOQL provides a powerful query mechanism which is expressed in terms of the persistent domain model and is therefore independent of the structure of the underlying database. The language has been designed specifically so that it can be translated into efficient SQL by a JDO implementation.

In essence (and this really is a brief simplification) each JDOQL query has a set of Candidates and a Filter clause, and might have other clauses governing the ordering of the results and the definition of incoming parameters, etc.

The Candidates are usually the set of all persistent instances of a stated class (including subclasses).

The Filter is a boolean expression written according to the JDOQL grammar, which uses much that is familiar from Java expression notation. Individual candidate instances are included in the Collection returned by query execution if the boolean filter expression evaluates to true for that instance.

Queries are created through newQuery() methods on the PersistenceManager interface. A query is actually an instance of the Query interface, which provides methods for setting the various clauses of the query and for executing the query with specified parameter values.

JDOQL (#1) - kodo workbench screenshot

Here's a simple query which searches for all AuctionItem instances whose corresponding Location has the isoCode 'UK':

// define the query

Query q = pm.newQuery (AuctionItem.class,

 "location.isoCode == 'UK'"); // candidate extent & filter

// execute the query

Collection results = (Collection) q.execute ();



// iterate the 'results' collection; its elements are real AuctionItem instances 

Iterator iter = results.iterator();

while (iter.hasNext()) {

    AuctionItem item = (AuctionItem) iter.next();

    // further processing as required 

}

If you iterate the results collection, the instances contained within it are those AuctionItem instances for which the filter "location.isoCode == 'UK'" evaluates to true, i.e. those whose 'location' field references a Location instance which has its 'isoCode' field set to 'UK'. Notice that JDOQL 2.0 allows string literals to be delimited by single quotation marks for convenience.

The query is translated into the native query language of the underlying datastore which is usually, but not necessarily, SQL.

JDOQL (#2) - kodo workbench screenshot

As with prepared statements in SQL, JDOQL queries can be written to take parameters. This second example is equivalent to the first except that a parameter called code is used instead of the string literal 'UK'. This makes the query more reusable.

// define the query 

Query q = pm.newQuery (AuctionItem.class,

 "location.isoCode == code"); // candidate extent & filter

q.declareParameters("String code"); // formal parameter declaration



// execute the query with appropriate parameter values

Collection results = (Collection) q.execute ("UK");

That's enough for now. For further details about JDOQL please refer to my book, but you probably know enough already to understand the new features which are described and illustrated below.

Searching for Users - Case Insensitive Comparison

When they come to the Hammer.org auction site users type in their user name and the system locates their RegisteredUser object. The query shown below tests for case-insensitive equality between the userName field and the incoming parameter "name".

Query q = pm.newQuery (RegisteredUser.class, "userName.toLowerCase() == name");

q.declareParameters("String name");

Collection results = (Collection)q.execute("jaques");

Results

The returned collection now contains the one RegisteredUser object with the userName "Jaques" (capital 'J').

SQL

Expressed in JDOQL the query is entirely portable. For those who are interested, the SQL that this query generated against my particular database schema was:

SELECT t0.JDOID, t0.JDOCLASS, t0.JDOVERSION, t0.BIDDER_JDOID,

 t0.EMAILADDRESS, t0.SELLER_JDOID, t0.USERNAME

FROM REGISTEREDUSER t0

WHERE (LOWER(t0.USERNAME) = 'jaques')

Searching for Items - Regular Expression Matching

A great new facility for searching is the addition of regular expression support for String types through the matches() method. The parameter to matches() is a subset of regular expression grammar, restricted to case insensitivity and wildcard operators. This subset is easily translated into SQL wildcard notation.

Here is a query which looks for auction items with titles that contain the word "Connect".

JDOQL (#5) - kodo workbench screenshot

Query q = pm.newQuery (AuctionItem.class, "description.matches('.*Connect.*')");

Collection results = (Collection)q.execute ();

Results

The results collection contains AuctionItem instances which match the filter criteria. With my data the collection contained only one matching auction item, that with the description "Java Database Connectivity".

SQL

Expressed in JDOQL the query is entirely portable. For those who are interested, the SQL that this query generated against my particular database schema was:

SELECT t0.JDOID, t0.JDOCLASS, t0.JDOVERSION, t0.CATEGORY_JDOID, t0.DESCRIPTION,

 t0.LOCATION_JDOID, t0.OPENINGPRICE, t0.RESERVEPRICE, t0.SELLER_JDOID, t0.TITLE,

 t0.WINNINGBID_JDOID, t0.WINNINGPROXYBID_JDOID

FROM AUCTIONITEM t0

WHERE (t0.DESCRIPTION LIKE '%Connect%')

Exploiting Case Insensitive Regex

Let's rewrite the query above to be case insensitive in its search for the word "connect".

JDOQL (#6) - kodo workbench screenshot

Query q = pm.newQuery (AuctionItem.class, "description.matches('(?i).*Connect.*')");

Collection results = (Collection) q.execute ();

Results

Now that the query is case insensitive the results contains two auction items, with descriptions "Java Database Connectivity" and "Disconnected Data Manipulation Strategies".

SQL

The case-insensitive SQL looks like:

SELECT t0.JDOID, t0.JDOCLASS, t0.JDOVERSION, t0.CATEGORY_JDOID, t0.DESCRIPTION,

 t0.LOCATION_JDOID, t0.OPENINGPRICE, t0.RESERVEPRICE, t0.SELLER_JDOID,

 t0.TITLE, t0.WINNINGBID_JDOID, t0.WINNINGPROXYBID_JDOID

FROM AUCTIONITEM t0

WHERE (LOWER(t0.DESCRIPTION) LIKE '%connect%') 

Items Won by Proxy Bid - Runtime Type (instanceof) Operator

In JDO 1.0 a query could use the cast operator (<typename>) to perform runtime type determination, but this wasn't always easy to read.

Here's an example which uses the newly-introduced instanceof operator to select only those auction items for which the winning bid is a proxy bid.

JDOQL (#7) - kodo workbench screenshot

Query q = pm.newQuery (AuctionItem.class, "winningBid instanceof ProxyBid");

Collection results = (Collection) q.execute ();

Results

The results colletion contains AuctionItem instaneces which match the filter criteria. With my data I got only the single auction item, that for "Enterprise JavaBeans", for which the winning bid is indeed a proxy bid placed by David.

SQL

Expressed in JDOQL the query is entirely portable. For those who are interested, the SQL that this query generated against my particular database schema was:

SELECT t0.JDOID, t0.JDOCLASS, t0.JDOVERSION, t0.CATEGORY_JDOID, t0.DESCRIPTION,

 t0.LOCATION_JDOID, t0.OPENINGPRICE, t0.RESERVEPRICE, t0.SELLER_JDOID, t0.TITLE,

 t0.WINNINGBID_JDOID, t0.WINNINGPROXYBID_JDOID

FROM AUCTIONITEM t0 INNER JOIN BID t1 ON t0.WINNINGBID_JDOID = t1.JDOID

WHERE (t1.JDOCLASS = 'org.hammer.domain.ProxyBid')

Items without Bids - Collection Containment

Are there any auction items for which no bids have yet been place? This type of query is now easy with the support of the isEmpty() method for Collection types.

JDOQL (#8) - kodo workbench screenshot

Query q = pm.newQuery (AuctionItem.class, "bids.isEmpty()");

Collection results = (Collection) q.execute (); 

Results

The results collection contains instances of AuctionItem for which their bids collection is empty. With my data I got two; "Disconnected Data Manipulation Strategies" and "Speaking Javanese".

SQL

Expressed in JDOQL the query is entirely portable. For those who are interested, the SQL that this query generated against my particular database schema was:

SELECT t0.JDOID, t0.JDOCLASS, t0.JDOVERSION, t0.CATEGORY_JDOID, t0.DESCRIPTION,

 t0.LOCATION_JDOID, t0.OPENINGPRICE, t0.RESERVEPRICE, t0.SELLER_JDOID, t0.TITLE,

 t0.WINNINGBID_JDOID, t0.WINNINGPROXYBID_JDOID

FROM AUCTIONITEM t0

WHERE (0 = (SELECT COUNT(*)

FROM AUCTI_BIDS

WHERE AUCTI_BIDS.JDOID = t0.JDOID)) 

Searching on User-Defined Properties - Map Containment

It is sometimes useful to peek inside Map constructs within a query filter. To help achieve this JDOQL now supports the containsKey() and containsValue() methods.

Here is a query which selects auction items whose itemProperty map contains the key "Pages" and also, independenly, contains the value "Addison Wesley".

JDOQL (#9) - kodo workbench screenshot

String filter = "itemProperties.containsKey('Pages') &&

 itemProperties.containsValue('Addison Wesley')";

Query q = pm.newQuery (AuctionItem.class, filter);

Collection results = (Collection)q.execute ();

Results

The results colletion contains AuctionItem instances which match the filter criteria. I got just the one - "Java Data Objects".

SQL

Expressed in JDOQL the query is entirely portable. For those who are interested, the SQL that this query generated against my particular database schema was:

SELECT DISTINCT t0.JDOID, t0.JDOCLASS, t0.JDOVERSION, t0.CATEGORY_JDOID, t0.DESCRIPTION,

 t0.LOCATION_JDOID, t0.OPENINGPRICE, t0.RESERVEPRICE, t0.SELLER_JDOID, t0.TITLE,

 t0.WINNINGBID_JDOID, t0.WINNINGPROXYBID_JDOID

FROM AUCTIONITEM t0 INNER JOIN AUCTI_ITEMPROPERTIES t1 ON t0.JDOID = t1.JDOID INNER

 JOIN AUCTI_ITEMPROPERTIES t2 ON t0.JDOID = t2.JDOID

WHERE (t1.KEY0 = 'Pages' AND t2.VALUE0 = 'Addison Wesley')

Dropping the Collection Wrapper - Unique Results

In JDO 1.0 the execution of a query always returned a Collection of instances matching the filter criteria. If no instances matched the filter then the Collection would be empty (an iterator would return false to its first hasNext() invocation). If exactly one instance matched the filter the the Collection would contain exatly one instance, but you'd still have to iterate the collection to retrieve it. Often, of course, the Collection would contain multiple matching instances.

Now some queries are specifically written to retrieve at most one instance, and the requirement to iterate a returned Collection is most onerous. To address such use-cases a "unique" property has been added to the Query interface. Setting this property to true causes execution of the query not to return a Collection, but instead to return a single object reference (which is null if no instances match the filter).

JDOQL (#10) without uniqueness - kodo workbench screenshot

Here is an example query which returns the single Seller instance which has a given account number. The account number is passed in as a parameter. This example does not use the "unique" property, so iteration of the Collection is required.

Query q = pm.newQuery (Seller.class, "accountNumber == '0072'");

Collection result = (Collection) q.execute ();

Iterator iter = result.iterator();

if (iter.hasNext()) {

    Seller slr = (Seller) iter.next();

}

else {

    // no Seller exists with account number 0072

} 

JDOQL (#11) with uniqueness - kodo workbench screenshot

This next example shows how much simpler this type of query is if the "unique" property is set to true. The result returned from the from execution can merely be cast to the Seller type and then tested for a null result.

Query q = pm.newQuery (Seller.class, "accountNumber == '0072'");

q.setUnique (true);

Seller slr = (Seller) q.execute (); 

if (slr == null) {

    // no Seller exists with account number 0072

}

Uniqueness and Projection

Query execution usually returns a Collection of results. Each of the results in the Collection is an instance of the same type hierarchy. Thus a query with a candidate class of Employee, when executed, returns a Collection of Employee instances (and possibly subclasses). With uniqueness enabled the result is no longer a Collection, but just a single reference of the Employee type (or a subclass thereof), which might be null.

When you look at Projection you will see that queries can now return raw field data as Collections of Object[] (instead of Collections of persistent instances). With uniqueness enabled what you get back is just a single reference of type Object[] (which might be null).

In JDO 2.0 it is also possible to pass the descriptor of a JavaBean class to a Query, causing the result to be a Collection of instances of this JavaBean. Each projected field is mapped into a property of the bean class. With uniqueness enabled on such queries, the result of execution is a single reference of the bean class type (which might be null).

Non Unique Queries

In conclusion it must be noted that an attempt to execute, with uniqueness enabled, a query which returns more than one matching instance is illegal. A JDOUserException is thrown with an informative message. This would happen in the queries I wrote above if more than one Seller had the account number 0072. Naturally I would expect a propper database schema to be constrained in such a way as to prevent that eventuality.

Pages and Pages of Items - Query Result Paging

It's a common requirement to process query results a page at a time, but until now there has bee no portable way to do this. Perhaps the most typical scenario is that of a web site displaying pages of data.

JDO 2.0 provides a range capability on the Query interface. The range is specified as two long primitives, start and end. When the query is executed the first "start" records are discarded, and at most the next "end - start" records are returned.

For reference there is no requirement for the persistence manager to be kept open between successive page requests. This facilitates scalable applications in which the persistence manager is closed after each page of records is retrieved, allowing it or its resources to be re-allocated.

Here's an example which retrieves the third page of AuctionItems with 5 records per page.

JDOQL (#12) - kodo workbench screenshot

Query q = pm.newQuery (AuctionItem.class); // no filter, so all instances match

q.setRange (10, 15); // start from the 10th up to but excluding the 15th

q.setOrdering ("title ascending");

Collection results = (Collection) q.execute ();

Results

The returned Collection will contain at most 5 instances of AuctionItem.

SQL

Expressed in JDOQL the query is entirely portable. For those who are interested, the SQL that this query generated against my particular database schema was:

SELECT LIMIT 10 5 t0.JDOID, t0.JDOCLASS, t0.JDOVERSION, t0.CATEGORY_JDOID, t0.DESCRIPTION,

 t0.LOCATION_JDOID, t0.OPENINGPRICE, t0.RESERVEPRICE, t0.SELLER_JDOID, t0.TITLE,

 t0.WINNINGBID_JDOID, t0.WINNINGPROXYBID_JDOID

FROM AUCTIONITEM t0

ORDER BY t0.TITLE ASC 

About Query Ordering

When using the query paging capability it is advisable to set an explicit ordering clause on the query unless your datastore maintains an inherent ordering. Data in most relational databases is not guaranteed to be in any particular order without an explicit ordering clause.

About Transaction Isolation

Please note that query paging does not of itself provide guarantees of transaction isolation. If you request successive pages within a single transaction then the results will be consistent with that transaction. However if you request successive pages in different transactions, as would be the usual case for this feature, then the results you see may be affected by insertions and deletions which are being committed to the database by other clients between your page requests. This is the normal and expected behavior.

Removing Unwanted Items Efficiently - Deletion by Query

In JDO the deletion of a single persistent instance is achieved by passing it as an argument to the persistence manager's deletePersistent() method. If you have a collection/array of objects you wish to delete then you can pass that collection to deletePersistentAll(). However, each of these approaches requires the JDO client to instigate the deletion, and thus it must have knowledge of each persistent instance which is to be deleted.

How can we delegate this entire operation to the server? In JDO 1.0 it was not possible to do so in a portable manner. However, JDO 2.0 adds specific capabilities to the Query interface to facilitate this.

Deleting Selected Auction Items

Any Query which, when executed, would return one or more persistent instances to the client can instead be executed "for deletion". In this case the statements issued to the database are not to select back data for matching instances, but instead to delete matching instances.

Here is code snippet using a Query which selects all Auction Items for which the bids collection is empty and deletes each returned auction item:

JDOQL (#13) individual deletions instigated by the client - kodo workbench screenshot

Query q = pm.newQuery (AuctionItem.class, "bids.isEmpty()");

Collection results = (Collection) q.execute();

Iterator iter = results.iterator();

while (iter.hasNext) {

    pm.deletePersistent(iter.next());

}

SQL

This sequence of statements below is representative of how such actions might be mapped into SQL:

SELECT t0.JDOID, t0.JDOCLASS, t0.JDOVERSION, t0.CATEGORY_JDOID, t0.DESCRIPTION,

 t0.LOCATION_JDOID, t0.OPENINGPRICE, t0.RESERVEPRICE, t0.SELLER_JDOID, t0.TITLE,

 t0.WINNINGBID_JDOID, t0.WINNINGPROXYBID_JDOID

FROM AUCTIONITEM t0

WHERE (0 = (SELECT COUNT(*)

FROM AUCTI_BIDS

WHERE AUCTI_BIDS.JDOID = t0.JDOID))



DELETE FROM AUCTIONITEM t0 WHERE t0.JDOID = 1



DELETE FROM AUCTIONITEM t0 WHERE t0.JDOID = 2



DELETE FROM AUCTIONITEM t0 WHERE t0.JDOID = 3 etc.

Here is an equivalent query which is executed for deletion. Deletion is delegated to the database server. The object returned from Query.deletePersistentAll() is an Integer representing the number of objects that were deleted.

JDOQL (#14) deletion delegated to the database

Query q = pm.newQuery (AuctionItem.class, "bids.isEmpty()");

int deletedCount = q.deletePersistentAll();

There are two subtleties to notice here. Firstly, deleted objects are not retrieved into the JVM unless their persistence-capable class implements the DeleteCallback interface. Secondly, the deletePersistentAll() invocation is being made on the Query (to delegate deletion to the database) and not on the PersistenceManager (in which case a collection/array of already in-memory instances would have to have been passed as a parameter).

SQL

I think you'll agree that the SQL resulting from deletion-by-query is much more efficient:

DELETE FROM AUCTIONITEM

WHERE (0 = (SELECT COUNT(*)

FROM AUCTI_BIDS

WHERE AUCTI_BIDS.JDOID = AUCTIONITEM.JDOID))

Unbound Query Variables

JDOQL has supported variables since JDO 1.0. They are usually used in filters which examine the contents of one-to-many relationships (e.g. Collections), in which the variable is said to be bound to the contents of the collection. They can also be used to represent all persistent instances of their persistent class typs, in which case they are said to be unbound.

JDO 2.0 provides a new supported option constant which indicates whether an implementation supports unbound query variables. This makes explicit the portability of queries which use unbound variables.

Beyond the formalization of this optional feature there is nothing new in JDO 2.0 concerning the indispensable part that variable play in JDOQL. However I am illustrating their use below partly to raise awareness of the capabilities they deliver, and partly as a background against which I can later (in Part III of this series) demonstrate a new JDO 2.0 feature for the implicit definition of variables according to the context in which they are used.

JDOQL (#15) bound query variables - kodo workbench screenshot

Here is an example of a query which uses a bound query variable to identify all Auction Items for which the corresponding bids include at least one bid by David.

String filter = "bids.contains(bid) && bid.bidder.user.userName == 'David'";

Query q = pm.newQuery (AuctionItem.class, filter);

q.declareVariables ("Bid bid");

Collection results = (Collection)q.execute ();

SQL

SELECT DISTINCT t0.JDOID, t0.JDOCLASS, t0.JDOVERSION, t0.CATEGORY_JDOID, t0.DESCRIPTION,

 t0.LOCATION_JDOID, t0.OPENINGPRICE, t0.RESERVEPRICE, t0.SELLER_JDOID, t0.TITLE,

 t0.WINNINGBID_JDOID, t0.WINNINGPROXYBID_JDOID

FROM AUCTIONITEM t0 INNER JOIN AUCTI_BIDS t1 ON t0.JDOID = t1.JDOID INNER JOIN BID t2 ON

 t1.BIDS_JDOID = t2.JDOID INNER JOIN BIDDER t3 ON t2.BIDDER_JDOID = t3.JDOID INNER

 JOIN REGISTEREDUSER t4 ON t3.USER_JDOID = t4.JDOID

WHERE (t4.USERNAME = 'David')

Results

With my data set I got only one AuctionItem, the one with description "Enterprise JavaBeans". Apparently David had very focussed interests!

JDOQL (#16) unbound query variables - kodo workbench screenshot

Here is an example of a query which illustrates unbound variables. The query selects all AuctionItems that have a bidder who has also bid on an another item offered by the same seller.

The query is easliy expressed with a candidate extent of AuctionItem, an unbound variable (bidder of type Bidder) and a bound variable (b of type Bid). The unbound variable logically iterates over all persistent Bidder instancesfor each candidate AuctionItem instance.

String filter = "bidder.bids.contains(b) && b.item.seller == seller && b.item != this";

Query q = pm.newQuery (AuctionItem.class, filter);

q.declareVariables ("Bidder bidder; Bid b");

Collection results = (Collection)q.execute ();

Results

The candidate class of the query is AuctionItem, so the returned collection is comprised of AuctionItem instances. For each persistent AuctionItem, the filter "bidder.bids.contains(b) && b.item.seller == seller && b.item != this" is evaluated for all Bidders (remember variable "bidder" is unbound). The results collection therefore contains all AuctionItems for which there exists at least one Bidder whose collection of Bids contains at least one Bid against a different AuctionItem being sold by the same Seller. Against my data set I got all 6 items.

SQL

SELECT DISTINCT t4.JDOID, t4.JDOCLASS, t4.JDOVERSION, t4.CATEGORY_JDOID, t4.DESCRIPTION,

 t4.LOCATION_JDOID, t4.OPENINGPRICE, t4.RESERVEPRICE, t4.SELLER_JDOID, t4.TITLE,

 t4.WINNINGBID_JDOID, t4.WINNINGPROXYBID_JDOID

FROM AUCTIONITEM t3, AUCTIONITEM t4, BID t2, BIDDER t0, BIDDE_BIDS t1

WHERE (t3.SELLER_JDOID = t4.SELLER_JDOID AND t2.ITEM_JDOID <> t4.JDOID) AND t0.JDOID =

 t1.JDOID AND t1.BIDS_JDOID = t2.JDOID AND t2.ITEM_JDOID = t3.JDOID

There is nothing new about the capabilities which variables bring to JDOQL, which has existed since JDO 1.0. However the formalization of the optional feature "javax.jdo.option.UnconstrainedQueryVariables" makes explicit the portability of such queries. It is anticipated that all of the leading JDO implementations will support this feature.

About the Author

Robin Roos is the author of Java Data Objects (Addison-Wesley) and a member of the JDO 2.0 (JSR-243) expert group, and VP Professional Services at SolarMetric.