Posts filed under dev

Monitoring quotas on Google App Engine

One of my periodic chores with the Google App Engine is monitoring the quotas, particularly in my apps without billing enabled. Unfortunately Google provides no programatic way of doing this, and it doesn't look likely that it will. There is a QuotaService, but that isn't well documented and only shows quota use during a request.

However, one can report on quota exceptions that occur using the LogService. With this it's possible to find all exceptions within the last hour, say, that involved an OverQuotaException, like so:

    final LogService logService = LogServiceFactory.getLogService();

    LogQuery query = LogQuery.Builder.withDefaults();

    Calendar cal = Calendar.getInstance();
    cal.add(Calendar.MINUTE, -60);


    final Iterable<RequestLogs> requestLogsIterable = logService.fetch(query);

    int quotaFailures = 0;

    for (RequestLogs requestLog : requestLogsIterable) {;

        for (AppLogLine appLogLine : requestLog.getAppLogLines()) {

            if (appLogLine.getLogMessage().contains("OverQuotaException")) {

I can use the total number of quota exceptions within the last hour to create a healthcheck servlet, which can be queried by a an automated monitor (I use ServerMojo to ping this URL once an hour).

Of course, this doesn't warn you that you're about to go over quota, but it's given me a good handle on how the app fares over the course of a day.

One warning, LogService querying is subject to its own quota. During my early experiments I managed to get the date range wrong, and blew my LogService read quota in one hit! YMMV.

Posted on April 8, 2013 and filed under dev.


I've started collecting useful links on all manner of subjects here. I hope you find these helpful. I will keep them up to date.

Posted on March 25, 2013 and filed under links, dev.

Objectify and Google Guice

I've been working over several Google App Engine Java apps recently to introduce Google Guice and Objectify to them. Guice is a lightweight dependency injection framework, and Objectify is a superb replacement for JDO/JPA in your Java GAE projects.

Google Guice lets you bind interfaces to implementations and annotate dependencies for injection, eg:

public interface MyService...

public class ClientCode {

    private MyService myService;

    public void setMyService(MyService myService) {
        this.myService = myService.


If you're familiar with Spring then you'll find this a doddle. There's no XML in sight - Guice concentrates pretty much only on dependency injection, and the Java-based configuration classes one uses instead of XML seem perfectly adequate for this.

It also works nicely with Objectify. This is a data access API for the app engine. Take a look at the examples, they are extremely straightforward:

class Car {
    @Id String vin; // Can be Long, long, or String
    String color;

ofy().save().entity(new Car("123123", "red")).now();
Car c = ofy().load().type(Car.class).id("123123").get();

There's an Objectify servlet filter, somewhat similar in purpose open session in view filters, which can easily be set up in a couple of lines in Guice.

Moreover, now I can easily write pretty concise DAO and Service classes that are easily testable, which is something essential I've been sorely missing.

Posted on March 25, 2013 and filed under dev.

The Google Prediction API

In a previous post I explored some sample sentiment training data available from Sanders. Now let's try using it in the Google Prediction API.

The API lets you upload a set of training data. It will then create a model which you can interrogate. Training data is stored in Google Cloud Storage, and the API is accessible via REST, secured by OAuth in the usual Google style.

To get a good idea of what's involved I recommend reading the Hello Prediction! tutorial. I pretty much followed their example, except instead of detecting the language I used it to detect sentiment.

I had to refine my aforementioned training data to be in a form suitable for the API. That just means in this case that it has to be CSV file like so:

"positive","I love the whole world and everything in it"
"negative","You guys suck"
"neutral","Cheese is a kind of dairy product"

After following the steps described in the tutorial I was then in a position to query the model. Here's the prediction for an actual example taken from the positive data set:

 "kind": "prediction#output",
 "id": "my_model_id",
 "selfLink": "",
 "outputLabel": "positive",
 "outputMulti": [
   "label": "positive",
   "score": 0.666667
   "label": "negative",
   "score": 0
   "label": "neutral",
   "score": 0.333333

Note that it doesn't give a unanimous positive vote, although it clearly chooses positive as the most likely category. I suspect this is because there is a lot more neutral data in the training set than either positive or negative, so that there is always a tendency to treat things as neutral. This is a useful quality where borderline cases are involved.

The other thing worth noting is the suspicious looking 2/3 and 1/3 score values themselves. Playing around with different queries always shows this 1/3 to 2/3 split, never any other numbers. I don't know what the cause of this is.

I need to spend some more time with this model, and probably get some more training data. One thing I will say is that it's both easy to use and fast. In Java terms the google-api-java-client covers a lot of ground here. I will post some more on developing with the Prediction API, and how well it performs in future posts.

Posted on March 22, 2013 and filed under dev.


One of the great things about Google App Engine is, if you stay inside the box, so to speak, many things are a doddle. So much so that I was able to create this new app, Googomi, in a day or two, most of which involved fiddling with and learning about the Google+ API.

The Googomi app is a very simple beast with only one purpose: it will take your public Google+ stream and turn it into an RSS feed.

I've put a modicum of processing into it, so that it should correctly guess the most appropriate title for each RSS item, eg choosing the annotation, or the remote URL's title, where appropriate.

I personally had a use case for this (apart from learning about various Google APIs) whereby I wanted to export Google+ posts to other services automatically. For example, with this I can post from Google+ to Buffer and then beyond automatically.

Posted on March 18, 2013 and filed under dev.

Google App Engine and the Google+ API

I've been playing with what the Google+ API has to offer and I've found it quite easy to integrate into my Google App Engine apps using the google-api-java-client.

I initially followed the Quick start for Java tutorial with regard to creating the OAuth tokens and so forth, but the google-api-java-client has some good tutorials regarding making the actual OAuth calls. See for example this section about how to make the calls from a Google App Engine app. The library handles all the plumbing for you.

I only had to make one ammendment to their example. I found that the refresh token wasn't being returned along with the access token after it was granted. However this was simply fixed by adding a call to setApprovalPrompt("force") on the GoogleAuthorizationCodeFlow.Builder, like so:

public static GoogleAuthorizationCodeFlow newFlow() throws IOException {
    return new GoogleAuthorizationCodeFlow.Builder(HTTP_TRANSPORT, JSON_FACTORY,
            getClientCredential(),             Collections.singleton(PlusScopes.ME.getUri())).setCredentialStore(
            new AppEngineCredentialStore()).setAccessType("offline")
Posted on March 18, 2013 and filed under dev.

Twitter Sentiment Data

I've been delving into some twitter sentiment analysis and have been casting about for some useful training data. I've found various sources but few have any neutral data, which I think is important for any training as sort of control.

One useful source is Sanders Analytics, which has a source of tweet ids and a script to download the actual tweets from the ids (Twitter's terms & conditions do not allow the tweets themselves to be distributed).

This script takes a couple of days to download all the tweets because it has to honour Twitter's API limits.

I found one issue in the script which is easily fixed. It could cope with the presence of "error" in the response, but not "errors", eg:

{"errors":[{"message":"Sorry, that page does not exist","code":34}]}

The simple fix is to add this to the parse_tweet_json function, after the error check:

if 'errors' in tweet_json:
    raise RuntimeError('errors in downloaded tweet')

When the script finishes it will produce a file called full-corpus.csv. Now the final data has this format:

"apple","positive","126360398885687296","Tue Oct 18 18:14:01 +0000 2011","a tweet of some sort"

That is, the subject, the sentiment, the tweet id, the date and the tweet content.

The subject is what the tweet is about. This is important, as the sentiment refers to the subject. In other words the sentiment is about the subject (in this case "apple"), and not anything else in the tweet content.

Regardless, for my purposes I do actually need the tweet content without the subject. This can be simply achieved using grep and awk. Eg to extract the neutral tweets:

grep "\"neutral\"" full-corpus.csv | awk -F"\",\"" '{print $5}' | cut -d "\"" -f1

The output of this will just be the tweets themselves.

Posted on March 18, 2013 and filed under dev.

Updating to GAE 1.7.5

Today I updated a maven-based Google App Engine app from 1.7.4 to 1.7.5. As before, it didn't turn out as straightforward as I expected (maybe I should stop expecting this).

Once I'd installed 1.7.5 and set gae.version to 1.7.5 the build failed yet again - the issue this time boiled down to this error:

Could not find artifact net.kindleit:maven-gae-parent:pom:0.9.6-SNAPSHOT

As usual I turned to stackoverflow for help, where several others have had the same problem.

The key for me was to specify for the GAE runtime version.




I continue to use 1.7.5 for other dependencies, eg appengine-api-stubs. I have no idea about the whys and wherefores regarding this inconsistency I'm afraid.

Posted on March 13, 2013 and filed under dev.

Facebook Apps in Heroku

A couple of years ago or so Heroku and Facebook teamed up to make creating Facebook apps a doddle. Indeed one can do so with a few clicks from the app creation centre in Facebook if you already have a Heroku account.

Here are pretty comprehensive instructions from Heroku on how to do this, and I can attest that it all works well.

I've added to this setup with a staging instance for team testing purposes using the facility Heroku has for managing different environments by pushing to different remotes. See this handy guide for full details.

To create a staging branch called staging:

heroku create --remote staging

And to add Facebook app credentials for the staging version of your app just do:

heroku config:add FACEBOOK_APP_ID=123456 FACEBOOK_SECRET=789102323etc --remote staging
Posted on March 3, 2013 and filed under dev.

Managing Javascript Resources In Maven

One of the fiddly steps in setting up a web app, and maintaining it is managing all the various javascript libraries your pages use. But it's quite easy to manage resources like jQuery in Maven thanks to WebJars. Here's how to use it in Dropwizard.

If you take a look at WebJars you'll see all sorts of supported libraries. I'll use jQuery in this example.

Adding jQuery to Dropwizard

First add your jQuery dependency in your pom:


Now add an AssetBundle in your Dropwizard service class:

public void initialize(Bootstrap<StreamWebAppConfiguration> bootstrap) {
    ... other assets ...
    bootstrap.addBundle(new AssetsBundle("/META-INF/resources/webjars", "/webjars"));

This will map the path "/webjars" to that jar resource - which will contain the jQuery js files in our example.

Now you can reference them in your HTML pages:

<script src="/webjars/jquery/1.9.0/jquery.min.js"></script>

And that's that. But you can go one step further. You can remove references to the version number in your pages by using the dropwizard-webjars-resource library.

Dropwizard Webjars Resource

To do this add another maven dependency:


In your service class remove the aforementioned AssetBundle and instead add a WebJarResource to your run method:

environment.addResource(new WebJarResource());

This will handle all the asset mapping (which is why an AssetBundle is no longer required). Now in your pages you can reference:

<script src="/webjars/jquery/jquery.min.js"></script>

i.e. without the version number. Simple! If you need to update your whole site to the next version of jQuery, just update the pom.

Posted on February 19, 2013 and filed under dev.

Streaming Twitter with Twitter4J

Twitter4J is an excellent java library for all sorts of twitter work. I've been using it recently to connect to the "garden hose", ie Twitter's streaming API. Here's how to follow a particular user with it.

You can load this into your project via Maven:


Now you can construct your TwitterStream class:

ConfigurationBuilder cb = new ConfigurationBuilder();
TwitterStreamFactory twitterStreamFactory = new TwitterStreamFactory(;
TwitterStream twitterStream = twitterStreamFactory.getInstance();

Of course you'd put your own oauth tokens etc. here.

To listen to a particular user you can use a FilterQuery object:

  FilterQuery filterQuery = new FilterQuery();
  filterQuery.follow(new long[] {3473284738472384327743L});

The follow method takes an array of user's ids to follow.

To track the user you need to add a listener and attach this filter:

  twitterStream.addListener(new MyStatusListener());

Now the MyStatusListener class merely implements StatusListener. The important method we implement here is onStatus. For our purposes we just print the statuses out:

  public void onStatus(Status status) {
          System.err.println("status: " + status);

We don't need to do anything else in our StatusListener implementation for our current purpose.

If you execute this code and let it run - Twitter4J will start a thread for you - you will see the results comming in, eg:

  status: StatusJSONImpl{createdAt=Mon Feb 18 15:51:26 GMT 2013, id=303532182444584960, text='RT @stephenfry: Oh no, ...
  status: StatusJSONImpl{createdAt=Mon Feb 18 15:51:30 GMT 2013, id=303532201159557123, text='RT @stephenfry: Oh no, ...
Posted on February 18, 2013 and filed under dev.

Up the Google App Engine Backend

I've been playing with the Google App Engine Backend service recently. Google's documentation on this is clear but it could really do with a tutorial. This post isn't anything like that desired tutorial but a recap of what I discovered and ended up with trying to address my particular need, which may be helpful!

I've been using the Backend in a game engine I'm writing. I'm using Google's Channel API to send messages to the client, but the game needs a background thread to handle the game engine itself. More preciesly, each game process is likely to run longer than the 60 second limit normally imposed on GAE requests. This is where the backend comes in.

Here I create a backend called zengine:

    <backend name="zengine">

I put this configuration in a file called backends.xml. A word about the options used here:

  • dynamic: launches the backend upon request, and that backend will die when finished
  • fail-fast: give up immediately when trying to launch the backend

To start the backend from my game launching code I use a queue, via the following code:

Queue queue = QueueFactory.getDefaultQueue();

This will create a GAE queue task which will execute the path /zengine in the backend environment called zengine. The code backendService.getBackendAddress("zengine") creates an address that works both in the production and local development environments. This is crucial, because the dynamic instances have an otherwise random and unguessable address - in dev environments they are on a random port!

By passing in the Host header with the address of the backend my /zengine servlet will run in the zengine backend environment instead of the normal, 60-second-limited one.

The actual task item will in practice contain parameters that specify which game process to start, ie for what user and what game. My /zengine path is mapped to the game process itself. It merely needs to start the process. Using the supplied queue task parameters I can get a handle on the channel for communication back to the client, and start the game.

Posted on February 5, 2013 and filed under dev.

Maven GAE Plugin Version Issues

While deploying something to Google App Engine this morning I noticed this minor but annoying warning:

[INFO] Updating Google App Engine Server...
There is a new version of the SDK available.
Latest SDK:
Release: 1.7.4
Timestamp: Sun Nov 11 09:09:13 GMT 2012
API versions: [1.0]

Your SDK:
Release: 1.7.3
Timestamp: Wed Oct 24 02:01:39 BST 2012
API versions: [1.0]

Please visit for the latest SDK.

I say annoying because I'd already specified the gae.version property to be 1.7.4, as per the plugin documentation. It was also definitely pointing to a 1.7.4 SDK installation.

If I run mvn gae:version it does indeed show the unwanted version:

INFO] Plugin version: 0.9.5
[INFO] SDK directory: /Users/nick/dev/support/appengine-java-sdk-1.7.4
[INFO] SDK version:
Release: 1.7.3
Timestamp: Wed Oct 24 02:01:39 BST 2012
API versions: [1.0]

Searching about I found a similar problem in previous versions of the plugin, a couple of years ago. Unfortunately the solutions given there don't work any more, because neither the appengine-local-runtime nor appengine-tools-api artifacts are available any more, at least from the central maven repository.

However a bit of trial error gave me this solution, to include appengine-tools-sdk instead as follows:



Then when running gae:version we get the correct version shown:

[INFO] Plugin version: 0.9.5
[INFO] SDK directory: /Users/nick/dev/support/appengine-java-sdk-1.7.4
[INFO] SDK version:
Release: 1.7.4
Timestamp: Tue Dec 11 11:41:31 GMT 2012
API versions: [1.0]

and the warning no longer shows when running mvn gae:deploy.

Posted on January 29, 2013 and filed under dev.

Google App Engine & Maven Tips

I found this useful blog posting by Gunawan Deng very useful in getting me started on using Maven to create Google App Engine projects.

Here's another tip. I ran into an error when trying to run the app locally with gae:run:

[ERROR] Failed to execute goal net.kindleit:maven-gae-plugin:0.9.5:run (default-cli) on project extropymvnspike: ${gae.home} is not a directory: ${gae.home} is not a directory:
[ERROR] Please set the sdkDir configuration in your pom.xml to a valid directory. Make sure you have correctly extracted the app engine sdk.

The error was easy to fix - one can add a property for gae.home in your pom.xml called gae.home which should point to the installation directory of your app engine SDK. However, I found it more useful to add it to my settings.xml as I imagined I would want it for all my GAE projects:



Any other tips for Maven/GAE integration?

Posted on January 25, 2013 and filed under dev.

Session-based Security in Dropwizard

Dropwizard is an incredibly useful framework for creating REST apis very quickly. One thing that it doesn't come with out of the box (yet) is support for session-based security, that is, holding principal information with a plain old HttpSession.

There are excellent reasons not to do this, particularly for REST apis. Session management can be a fiddly business that isn't particularly scalable. However, sometimes you need it anyway.

Here's how to add a simple annotation-based scheme, drawing heavily on the very useful posting by Antoine Vianey.


Let's add an annotation to represent a logged in user, i.e. a user that's been set into the session as an attribute:

@Target({ElementType.PARAMETER, ElementType.METHOD, ElementType.FIELD})
public @interface SessionUser {

With this we can write an authentication provider:

public class SessionUserProvider
        implements Injectable<User>, InjectableProvider<SessionUser, Type> {

    private final HttpServletRequest request;

    public SessionUserProvider(@Context HttpServletRequest request) {
        this.request = request;

    public Injectable<User> getInjectable(ComponentContext cc, SessionUser a, Type c) {
        if (c.equals(User.class)) {
            return this;
        return null;

    public ComponentScope getScope() {
        return ComponentScope.PerRequest;

    public User getValue() {
        final User user = (User) request.getSession().getAttribute("user");
        if (user == null) {
            throw new WebApplicationException(Response.Status.UNAUTHORIZED);
        return user;


This provider will inject a user object from the session which can be used in our resource methods. If there is no such object, the Response.Status.UNAUTHORISED (i.e. 401) is returned.


To wire up our provider we need to add these lines to our Service run method:

environment.setSessionHandler(new SessionHandler());

The first line enables session support, which is not enabled by default. The second line wires up our new authentication provider in the request scope.


To use this we just add a @SessionUser annotated parameter to our resource methods, eg:

public MessageRepresentation getMessage(@SessionUser User user) {

    return new MessageRepresentation("hello");

and always have a valid user object available in our resource methods.


To set the User into the session in the first place, for example from the submission of a login form, we just need to get hold of it from the request context, eg:

public HomeView login(@FormParam("username") String username, @FormParam("password") String password,
                      @Context HttpServletRequest request) {

    // should lookup the user etc. here - let's assume the user is fine and dandy to proceed for this example

    request.getSession().setAttribute("user", new User());

    return new HomeView();


This is an easy and flexible way to add session management to your resources when you need it. As Antoine adds, one can create other annotations for different roles, eg a @SessionAdminUser, if you want to restrict different methods to different roles.

Posted on January 23, 2013 and filed under dev.

Google App Engine & Maven

Here's another example of going from zero to deployment rather speedily, this time with the Google App Engine and the maven plugin.

1. Get yourself a Google App Engine account

You can get one quite easily here. For the minimal example use described here, no fees should be incurred. You'll need a google account.

2. Create a project in GAE

Once you have an account you'll see a list of your apps and a Create Application button. Simply choose a name and a description and you're set.

Screen Shot 2013-01-21 at 16.55.18.png

3. Use maven to create an example GAE project

Just type

mvn archetype:generate

If you search for "gae" you will find several options that use the maven plugin. I chose the simple JSP-based version:

remote -> net.kindleit:gae-archetype-jsp (Archetype for creating maven-gae projects that uses jsp for the view)

4. Test the project locally


mvn gae:run

and it will launch the app - go to http://localhost:8080 and you'll see a very simple guestbook application.

5. Upload it to GAE

Now type

mvn gae:deploy

It will ask you to login with your google credentials, and begin uploading:

[INFO] Updating Google App Engine Server...
20% Scanning files on local disk.
25% Initiating update.
28% Cloning 1 static files.
31% Cloning 35 application files.
40% Uploading 6 files.
80% Uploaded 6 files.
82% Initializing precompilation...
84% Precompiling... 6 file(s) left.
87% Sending batch containing 6 file(s) totaling 22KB.
90% Deploying new version.
95% Will check again in 1 seconds.
98% Will check again in 2 seconds.
99% Will check again in 4 seconds.
99% Will check again in 8 seconds.
99% Closing update: new version is ready to start serving.
99% Uploading index definitions.

Update completed successfully.

You should then be able to see your app in the cloud. Assuming you named your app myapp it will be visible at

Posted on January 21, 2013 and filed under dev.

Delightful Code

I've been trawling about looking for old Inform (Z-Machine) games to play, and I came across this rather pleasant example of Inform 7 code. This produces an actual running game, of a sort:

  Will's Study is a room. The desk is here.
  A hastily handwritten note is on it.
  Description is "It's from your friend Shakespeare:
  'I've gone to lunch. You'll have to write the sonnet.'"
  Composing is an action applying to nothing.
  The quill is a thing that is in the study.
  Understand "write sonnet" as composing.
  Description of the quill is "Old and cruddy".
  Instead of composing when the player
  has no quill, say "You have not got the quill."
  Instead of composing, say "And... done. 'Heya',
  says Will, returning. You say, 'Hello, Will!'
  Says Shakespeare, 'Thank you for the time you've taken!
  You really are a pal, Sir Francis Bacon.'"

I might have to give this a go myself. Maybe some interactive system documentation...

Posted on January 21, 2013 and filed under dev.

From CLI to the Cloud with Github, Heroku and Dropwizard

I've been experimenting with various ways of narrowing the gap between development and production, particularly for the lone developer or very small team. Here's an overview of how to get up and running with a maven Dropwizard project in Github and Heroku. I'm assuming a basic familiarity with git and maven.

I am indebted to Gary Rowe's blog entry for getting me started on this. He goes into deeper detail on several points than I do here.

Create a local Dropwizard project in maven

You can create the maven project itself quite easily from the command line:

mvn archetype:generate

Choose the default option. Let's say you call your project myproject.

cd myproject
git init
git add .
git commit -m "my new project"

You've now created a new (empty) project in your local git reposity. All standard stuff.

Now you can write the code itself. I would suggest using Dropwizard's own tutorial. This will create a good old "hello world" example.

Create a github project

Now this step is optional. You could of course just push your changes to Heroku directly, but I like to keep my shared repository more general purpose - hence Github.

Once you're happy with the project and committed all the changes, you can push the changes to Github. There you can create your project (let's say it's called myproject there too), and sync your local git and github. Github actually provides instructions on how to do this after you've created the project:

git remote add origin
git push -u origin master

From now own you can push locally committed git changes to github as follows:

git push origin master

Create a heroku project

Now you can push your changes to Heroku. Let's assume you have a Heroku account and have installed the Heroku toolbelt.

Inside your project you need to add a Procfile. Heroku uses this to determine how to run your application once it has been built. For a Dropwizard project this is quite simple:

web:    java $JAVA_OPTS -Ddw.http.port=$PORT -Ddw.http.adminPort=$PORT -jar myproject/target/myproject-1.0-SNAPSHOT.jar server myproject/config.yaml

Now, once you've committed this you can push this project up to Heroku.

heroku create

This will create a new app in Heroku. It will create it with a randomly generated name which you'll probably want to change. You can change this easily enough in the Settings section of the app on the Heroku website. Once you've done this you need to perform an extra step to update your local git remote configuration to reflect the new name. Let's say you changed it to myproject.

git remote rm heroku
heroku git:remote -a myproject

Now, finally, you can deploy:

git push heroku master

You should see the build taking place, and finally something like:

-----> Discovering process types
       Procfile declares types -> web
-----> Compiled slug size: 57.7MB
-----> Launching... done, v9 deployed to Heroku

   df0d0b4..06a12b7  master -> master

If you now type

heroku open

a browser will open pointing at your app. Each time you push to Heroku, you'll update the app running there.


Hopefully this is enough to get you started. I haven't covered some tidbits like creating your Heroku keys and other set up shenanigans, nor tuning your app Heroku settings. I refer the reader to the Heroku Java tutorial and Gary's aforementioned article for further details.

If you have any suggestions or corrections please let me know in the comments!

Posted on January 18, 2013 and filed under dev.

Installing leiningen on OSX

I ran into this while trying to install leiningen 2 on a mac. In fact installing works, but if you try to do anything, this happens:

    Could not find artifact lein-newnew:lein-newnew:jar:0.3.7

It turns out there’s a dependency issue that won’t be fixed until RC2.

To fix it is quite easy - first install leiningen 1.7 (which does work - you probably need to uninstall lein 2 first) to create a new project:

    lein new flounder

Then uninstall lein 1.7 and install 2 again.

Then within that project just type:

    lein deps

and all the required dependencies will be installed, and lein will now work.

Posted on January 12, 2013 and filed under dev.