Welcome to Episode 12 of the series. This episode is an extension of the earlier episode where we saw how to write Simple Google Wave Robots using the WadRobotFramework. I strongly recommend that  you have completed the earlier episode and have got comfortable with the WadRobotFramework since this episode builds on the earlier one.

To recap, the WadRobotFramework distinguishes between 2 kinds of Robots and I summarize it again over here.

Simple Robots : These robots we covered in the earlier episode and saw how you can write a simple robot to react to a blip by appending a new blip (BlipAppenderRobot) or even modify the Blip Text (BlipModifierRobot).

Advanced Robots : These are of main focus in this article and I reproduce from the earlier episode the text so that you understand what Advanced Robots are first. The definition of Advanced Robots is per the WadRobotFramework and it is not meant to indicate this is the final definition of it.

Advanced Robots are those that can react to instructions ( or commands) in the Blips. Here are some samples of Advanced Robots and how they would react to commands from Blips:

1. A character Counting Advanced Robot:

Your submitted Blip Text contains : “Here is some text in the blip. Please count the length of this message for me.” {mycountingrobot:count}.

You can write an advanced Robot (mycountingrobot) that knows how to count the length of the message. So it gets notified when it there is a instruction (count) in the Blip. Your Advanced Robot can then count out the characters and then either append or modify a blip as needed.

2. A Tweeting Robot:

Your submitted Blip Text contains the following text : “{mytweetingrobot:tweet} Some text to tweet”

You can write an advanced Robot (mytweetingrobot) that knows how to tweet the message to Twitter. So it gets notified when it there is a instruction (tweet) in the Blip. Your Advanced Robot can then append a blip or modify a blip saying that the message has been tweeted.

The best part of it all is that you could combine all of this into a single Robot that can respond to one or more commands. For example, take a look at Today’s Special Robot (see http://ppandhi.wordpress.com/2009/11/08/todays-special-robot/) that can respond to more than one command. It can give you the quotes, day in history, word of the day, cricket score, your daily horoscope by simply responding to the command that you type in.

So for example, you could write a robot and give it commands like this:

1. {myrobot:doCommand1}

2. {myrobot:doCommand2}

3. {myrobot:doCommandN} and so on.

In this article, we are going to see exactly how to achieve the above command Robot that will delegate its work to different workers who are responsible for executing the command i.e. doing the work.

Let us get a few definitions in place first:

1. The Command Robot: This is the main class of your Robot and you need to extend the org.wadael.waverobotfrmwrk.advanced.WithWorkersRobot. You need to have an identifier for your robot, which is a unique ID for your Robot. Let us call it GAEJRobot.

2. Each Command Robot is capable of following instructions or commands. These instructions are executed by the Workers.

3. A Worker is a Command implementation that performs a certain logic. For e.g. fetching a stock quote, getting a word of a day, sending a Tweet, sending an email, etc. Each worker will indicate what instruction or command it obeys.

As an example, say you want to write an Advanced Robot class (WithWorkersRobot) whose identifier is named GAEJRobot that can responds to the following two commands:

a. SendTweet
b. GiveWordOfTheDay

So, you will implement two Workers and register (add) them to the GAEJRobot class. The two worker classes will be :

• SendTweetWorker which says that listens to an instruction named tweet and it will implement its core logic in the doWork() method.
• GiveWordOfTheDayWorker which says that it listens to an instruction named wotd and it will implement its core logic in the doWork() method.

Now, in your Wave Conversation, you can give the following text in the Blip (of course after adding the Robot as a participant).

1. {GAEJRobot:tweet}

2. {GAEJRobot:wotd}

Voila! The WadRobotFramework will then do the heavy lifting for you. It roughly works as follows:

• When you submit a Blip, it will scan the Blip Text for the identifier and see if it matches itself.
• If yes, it will scan out the instructions and invoke the doWork() method of the Worker Robot that implements the instruction.

This is not all. The WadRobotFramework has thought about parameters that you may need to pass to your Robot. For e.g. consider the following fictitious instruction that you need to give to a Stock Quote Robot.

{StockQuoteRobot:getquote GOOG} or {StockQuoteRobot:getquote GOOG,MSFT,ADBE,ORCL,IBM}

In short the format is as follows:

{RobotIdentifier:Instruction<space>[parameters]}

So in the above two examples the parameter GOOG and the parameter string “GOOG,MSFT,ADBE,ORCL,IBM” will be passed to the doWork() method of your RobotWorker that has registered with the Advanced Robot and who implements the getquote instruction. Please read this last statement clearly with the highlighted words as the key pieces required to build out an Advanced Robot.

Simple yet powerful and it opens up a wide range of Robots that you can start writing today. So let me get started and demonstrate the basic flow to get an Advanced Robot up and running which can accept 1 or more instructions. The Robot does not do anything specific except for simply demonstrating the flow. Readers are expected to extend it with their ideas brimming inside their minds.

To understand what we will build, it helps to take a look at the running robot as shown below:

You will notice that I have added my robot called MyAdvancedRobot. The identifier for the Robot is GAEJRobot and the Robot has two workers (Worker1 and Worker2)  registered with it, which implement the instructions  command1 and command2 respectively.

Now when I submit the text {GAEJRobot:command1} , the doWork() method of the Worker1 is invoked. It simply accepts the command and prints out that it received the command with no parameters passed to it.

Similarly, look at the wave conversation below:

Here I give the command2 to the GAEJRobot and I am also passing a string of parameters. When I click the Done button, the doWork() method of the Worker2 is invoked. It simply accepts the command and prints out that it received the command with the parameter string. Now, it could have processed the parameters and performed its logic accordingly. This demonstrates how the wiring is done by WadRobotFramework to make your life easier in writing Advanced Google Wave Robots.

Let us start writing code now. I will assume that you are fairly conversant now with the typical project structure and requirements of writing a Google Wave Robot. If not, I suggest that you first go through these tutorials in the following order:

• Episode 7 : Writing your First Google Wave Robot
• Episode 11: Develop Simple Google Wave Robots using the WadRobotFramework

Project Setup

Create a New Project or use the same project MyGoogleWaveRobot that we used in the earlier Episode 11.

If you are creating a New project, then please make sure that you download all the JARs from the following two locations:

http://code.google.com/p/wave-robot-java-client/downloads/list

The web page when you navigate to the above URL is shown below:

Download all the above files to your machine.

The WadRobotFramework JAR file is available at the following location :

http://code.google.com/p/wadrobotframework/downloads/list

The web page when you navigate to the above URL is shown below:

Download the JAR file to your machine. Once you have downloaded the 5 JAR files, make sure that they are copied to the WEB-INFlib folder of your project and that the Project Build Path is also setup with the JAR files as shown below:

Writing the Advanced Robot : MyAdvancedRobot.java

The first step is to create a Java class. Call it MyAdvancedRobot.java. The source code is listed below:

package com.gaejexperiments.waverobot;

import org.wadael.waverobotfrmwrk.advanced.WithWorkersRobot;

public class MyAdvancedRobot extends WithWorkersRobot {

public MyAdvancedRobot() {
super();
//This will process 'command1'
addRobotWorker(new Worker1());

//This will process 'command2'
addRobotWorker(new Worker2());
}

@Override
public String getRobotIdentifier() {
return "GAEJRobot";
}

@Override
protected String getUsage() {
return "Advanced Robot commands : command1 and command2";
}

@Override
protected String getRobotSelfIntroduction() {
return "I am an Advanced Robot";
}

}

Let us dissect the code now:

1. Our Advanced Robot class MyAdvancedRobot extends the WithWorkersRobot class, which is required for creating the Advanced Robots with the WadRobotFramework.

2. The WithWorkersRobot class constructor uses a method called addRobotWorker(…) to add one or more workers to it. Remember each worker will execute or do its job as per the instruction that it obeys. So we are going to have two workers : Worker1 and Worker2 which we are adding to our AdvancedRobot. We will get to the listing of the Worker1 and Worker2 later but it is sufficient to keep in mind, that Worker1 class will perform the work required with command1 instruction is given in the Blip and Worker2 class will perform the work required when command2 instruction is given in the Blip. To recap, as you add more workers, you will need to add them here in the constructor using the addRobotWorker method.

3. The getRobotIdentifier() method is used to return the string that is the unique identifier of your Robot. This is used to distinguish Robots in the same Wave. The identifier if you recollect is going to be used by the other participants in the wave to give instructions to your Robot. As mentioned, the format in the Blip to give an instruction to your robot will be like this:

{RobotIdentifier:Instruction<space>[parameters]}

Each Instruction is implemented by a Worker. For e.g. command1 will be implemented by Worker1 and command2 will be implemented by Worker2.

So to invoke Worker1, we have to say {GAEJRobot:command1} in the Blip and submit it. Hence we return GAEJRobot in the getRobotIdentifier() method and this will be the unique way in which we identify this Robot

4. The getUsage() method is used to return a string that will be displayed to the participant when they type /help in the start of the Blip. This is useful to give a list of possible instructions that your Advanced Robot implements. In our case here, we are implementing two instructions and hence we have returned some string. But you can give an elaborate help string stating detailed command text, sample parameters, etc.

5. Finally, we have the getRobotSelfIntroduction() method. This is not mandatory but it is nice to announce to the particpants when you (Robot) gets added to the wave as a participant. Simply return a String that you would like to announce to the other (existing) participants in the Wave.

Implementing the Workers

We are now going to implement the Workers i.e. Worker1 and Worker2. The code is identical for both of them and it is listed below:

Worker1.java

package com.gaejexperiments.waverobot;

import org.wadael.waverobotfrmwrk.advanced.RobotWorker;

import com.google.wave.api.Blip;
import com.google.wave.api.Event;
import com.google.wave.api.RobotMessageBundle;

public class Worker1 implements RobotWorker {

public String getInstruction() {

return "command1";
}

public boolean doWork(RobotMessageBundle bundle, Blip blip, Event evt, String params) {
blip.getDocument().append("Robot Worker 1 got the command with parameter string : " + params);
return true;
}

public String getDescription() {
return "Robot Worker 1";
}
}

Worker2.java

package com.gaejexperiments.waverobot;

import org.wadael.waverobotfrmwrk.advanced.RobotWorker;

import com.google.wave.api.Blip;
import com.google.wave.api.Event;
import com.google.wave.api.RobotMessageBundle;

public class Worker2 implements RobotWorker {

public boolean doWork(RobotMessageBundle bundle, Blip blip, Event evt, String params) {
blip.getDocument().append("Robot Worker 2 got the command with parameter string : " + params);
return true;
}

public String getDescription() {
return "Robot Worker 2";
}

public String getInstruction() {

return "command2";
}

}

Let us go through the code of one of them and you will be able to understand it:

1. To recap, a Worker implements an instruction or a single command. Each Worker class needs to implement the RobotWorker interface in the WadRobotFramework.

2. It needs to implement the getInstruction() method which returns a String. This is the instruction that the Worker will obey or perform. In our case, the command1 is being done by Worker1 class and the command2 is being done by Worker2 class respectively.  So when someone submits {GAEJRobot:command1} in the Blip, the doWork() implementation of the Worker1 class will be invoked and if they submit {GAEJRobot:command2} in the Blip, the doWork() implementation of the Worker2 class will be invoked.

3. It needs to implement the doWork() method. This method is the heart or the main implementation of the Worker. Here you will place all your processing logic. Notice that since this is an Advanced Robot, it is assumed that you would even like to make use of the Google Wave API classes directly. So you are passed in instances of RobotMessageBundle, Blip and Event classes. The last parameter passed is params and it represents any parameters passed to the robot.

You will notice in the implementation that we have done for the Worker, that we simply Append to the Blip Text saying that the Worker got its command and notice that we also print out the Parameter String. So if you logic depends on the values of the parameters passed, you can parse out the parameters here itself and perform your logic.

That is all we need to do as far as writing Java code is concerned. Of course we have the other mandatory files that we need to create, which we will cover quickly now:

Configuring the MyAppenderRobot in web.xml

We need to add the MyAdvancedRobot in the <servlet/> and <servlet-mapping/> entry to the web.xml file. This file is present in the WEB-INF folder of the project. The necessary fragment to be added to your web.xml file are shown below.

<servlet>
<servlet-name>MyAdvancedRobot</servlet-name>
<servlet-class>com.gaejexperiments.waverobot.MyAdvancedRobot</servlet-class>
</servlet>
<servlet>
<servlet-name>MyRobotProfileServlet</servlet-name>
<servlet-class>com.gaejexperiments.waverobot.MyRobotProfileServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>MyAdvancedRobot</servlet-name>
<url-pattern>/_wave/robot/jsonrpc</url-pattern>
</servlet-mapping>
<servlet-mapping>
<servlet-name>MyRobotProfileServlet</servlet-name>
<url-pattern>/_wave/robot/profile</url-pattern>
</servlet-mapping>

Notice that we also have the ProfileServlet configured here, which is a good and recommended thing to have for your Robot. The ProfileServlet class is covered in the next section.

ProfileServlet.java

The implementation is straightforward and contains the ApplicationId that I have used for my AdvancedRobot. You can replace it with your id.

package com.gaejexperiments.waverobot;

import com.google.wave.api.ProfileServlet;

public class MyRobotProfileServlet extends ProfileServlet {

@Override
public String getRobotAvatarUrl() {
return "http://myinfoagent.appspot.com/_wave/myimage.jpg";
}

@Override
public String getRobotName() {
return "MyAdvancedRobot";
}

@Override
public String getRobotProfilePageUrl() {
return "http://myinfoagent.appspot.com";
}

}

Creating the MyAdvancedRobot capabilities.xml files

We need an additional file to describe the capabilities of the Robot that we have written. This file is called the capabilities.xml and it needs to reside in a certain location. You need to create a _wave directory inside of the war directory of your project. The location of this file is  war/_wave directory.

You will need to create the _wave directory and create the capabilities.xml file over there. The capabilities file shown below is pretty straightforward and is shown below:

<?xml version="1.0" encoding="utf-8"?>
<w:robot xmlns:w="http://wave.google.com/extensions/robots/1.0">
<w:capabilities>
<w:capability name="BLIP_SUBMITTED" content="true" />
</w:capabilities>
<w:version>1</w:version>
</w:robot>

Deploying the Application

To deploy the application, you will need to first create your Application ID. The Application Identifier can be created by logging in at http://appengine.google.com with your Google Account. You will see a list of application identifiers already registered under your account (or none if you are just getting started). To create a new Application, click on the Create Application button and provide the Application Identifier as requested. Please note this name down since you will be using it for deployment.

For e.g. I have registered an application identifier named myinfoagent.

To deploy the application, follow these steps (they should be familiar to you now):

1. Click on the Deploy Icon in the Toolbar.
2. In the Deploy dialog, provide your Email and Password. Do not click on Deploy button yet.
3. Click on the App Engine Project settings link. This will lead you to a dialog, where you need to enter your Application ID [For e.g. my Application Identifier myinfoagent]
4. Click on OK. You will be lead back to the previous screen, where you can click on the Deploy button. This will start deploying your application to the GAEJ cloud. You should see several messages in the Console window as the application is being deployed.
5. Finally, you should see the message “Deployment completed successfully”.

MyAdvancedRobot in Action

Your application is going to be available at the http://yourapplicationid.appspot.com. In my case, the application is available at http://myinfoagent.appspot.com.

You can test for the presence of your robot capabilities file by simply typing in the following:

http://yourapplicationid.appspot.com/_wave/capabilities.xml [Replace yourapplicationid with the Application ID that you have] and you should see the capabilities.xml file being served up.

To test out the Robot, you need to launch the Google Wave client and login in with your account by going to http://wave.google.com. On successful login, you will be inside the Wave client from where you can create a new wave by clicking on the New Wave link. I then add the myinfoagent@appspot.com, which is our AdvancedRobot to the Wave as a participant as shown below:

On addition, the AdvancedRobot will announce itself to the Participants. This is the method getRobotSelfIntroduction() that we implemented in the MyAdvancedRobot.java class. The output is shown below:

Now, we type in the message /help in the beginning of the Blip and submit it as shown below:

When submitted, the method getUsage() of the MyAdvancedRobot class is invoked and it displays the commands and any help instruction that you provided in there.

The next thing we do is give a command to the first Worker as shown below and click on the Done button.

This will invoke the doWork() method on the Worker1 class since this class has indicated that it implements command1 as mentioned in the getInstruction() method of the class. The response of the Worker is shown below:

As you can see we did not pass any parameters to it, so it printed out null.

Now, let us invoke the second Worker as shown below and notice that we are passing parameters to it now:

When we click on Done, the doWork() method on the Worker2 class since this class has indicated that it implements command2 as mentioned in the getInstruction() method of the class. The response of the Worker is shown below:

Notice that the parameter string was now passed and successfully displayed. You can easily parse out the String and perform your logic.

Conclusion

This concludes the 2-part tutorial on creating Simple and Advanced Robots using the WadRobotFramework. The framework takes away much of the API that you need to know to get going with the Wave API and instead lets you focus on the core of our Robot logic. The framework is still is its early stages and would do with some feedback from you.

Till the next episode, just Smile and Wave!

Welcome to Episode 10. In this episode, we shall cover the experimental Task Queue Service in Google App Engine. This is an experimental service, which means that it could undergo change in its core functionality from all respects like methods, package names, etc. In any case, we can safely expect that it shall get confirmed in some form or the other in a future release of the Google App Engine.

This episode is sort of a natural progression on our last episode, which covered the Cron Service. To reiterate, the Cron Service was used for background jobs that you wish to perform outside of a user request. They are not typically initiated by a user but instead configured by your application to perform certain periodic tasks like summary reports at end of the day, daily backups, etc. The Task Queue Service provides a facility to process tasks in the background which are :

• Typically initiated by the user.
• They can be created internally by the application also to break down a background job into smaller tasks.

Some of the examples of performing a Task in the background include:

1. A user signs up at the website and submits his/her information. On receiving the request, a simple message is displayed back to the user that his/her subscription request has been received and they will hear soon about the status. At the same time, a task can be created that can process this users request, perform some verification and then send out an email to the user if the subscription is setup successfully. So you could have done either of the following:

• Receive the user details and create a Task. This single task will take care of creating the required records in the database and send out an email.
• Receive the user details and create a Task. This task will simply create a record in the database and then create another task to send out an email.
• And so on.

2. An online shopping site could accept the order from a buyer. A task is then created in the system that will process the order. As part of the order processing, once shipping is done, another task is told to update the shipping status. Similarly when the logistics provider gives updates on the shipment locations, a task could be launched to update the shipment route of the package.

To summarize, any background (asynchronous) task can be launched when a user initiates the event or when an application or business event occurs. All these tasks are put into one or more user defined or default queues and are executed by the Google App Engine infrastructure on behalf of your application.

What does a Task constitute? What is a Queue ? Who executes it ?

The official documentation of Tasks is excellent and I suggest reading that in detail. In this episode I will cover just about enough for you to get started on Tasks and then dig deeper into them depending on your needs. So let us first understand what is the basic information that I need let the Google App Engine know about my task. I will take the liberty here to describe all the key concepts via an example that we will build in this application.

We wish to implement the following flow in our application:

1. A user wishes to sign up for your newsletter by providing an email id in a web form provided at our site.
2. The user enters the email id and clicks on sign up (a button).
3. The request is sent to a Servlet that accepts the email id and creates a Task. The response is sent back to the user thanking them for their interest.
4. The Task is then executed independently by the Google App Engine infrastructure and our code inside the Task i.e. checking if the email id is used, etc is verified and then an email is sent.

So, as you can see we have decoupled the background task (in step 4) from the sign up process (step 1, step 2 & step3).

It should now be straightforward to define the key elements:

1. Task : This is the unit of work that we wish to perform. In fact, the actor that will be performing this task is Google App Engine. So when we create a Task, we need to provide a standard way for the Google App Engine to invoke tasks and pass them their payload. It should now be straightforward to see that when we create a task, all we need to tell GAEJ is the URL (where to invoke the task) and the parameterized payload (data). This can be done in a standard fashion that you know. The URL is nothing but the servlet URL that will invoke the servlet that implements the Task. And the parameterized data is nothing but request parameters passed into your servlet so that it can execute accordingly. The data in this case will be nothing but the email id of the user who wants to sign up for your newsletter.

2. Queue : All Tasks when they are created are placed in a queue. They are then executed by the Google App Engine. To help us manage and categorize our tasks, you can define your queues by giving them appropriate names. For e.g. myqueue, emailqueue, etc. What this helps you to do is to place your tasks in the appropriate queue. Few points to note about queues (refer to the documentation for finer details):

• All queues in your application are defined in a file named queue.xml that is present in the WEB-INF folder of your application.
• Each queue has a unique name and you can control the rate at which tasks are executed in this queue by the Google App Engine. If you do not specify a rate, then the default rate is 5 tasks/second.
• There is a default queue for your application named ‘default’ and if you can chose to add your tasks to the default queue. Alternately, you can add them to your application defined queue.

I believe this should be sufficient for us to begin developing a simple flow that will demonstrate how to define a task queue, create a task and then see it getting executed by the Google App Engine.

Task Queue in Action

The diagram below shows what we will implement in the next few sections. The use case is that of an user that wishes to subscribe to our newsletter and how we shall break up the process into the request processing and then the task processing.

Let us break down the above flow into the steps given below and what we shall be developing at each step:

1. In Step 1, the user visits a web page and enters his/her email id to sign up for the newsletter. We shall not be developing a web page over here to keep things simple. Instead we shall be testing it out by directly invoking a servlet that accepts the email id of the user. This is the same thing that you would have normally done  by hooking up the action of the HTML form to that of this servlet.

2. In Step 2, we will be looking at a Servlet whose existence is to do some verification and then create a Task for background processing. This servlet (GAEJCreateTaskServlet) will create a Task in a queue called subscription-queue. As we covered earlier, to create a Task, we need to provide two pieces of information to the Google App Engine so that it can execute it. They are the URL and the Data. The URL (/gaejsignupsubscriber) is going to be that of a Servlet (GAEJSignupSubscriberServlet) that shall be doing the core task. And the data will be what the servlet needs to complete the task. In our case, the data is the emailid request parameter.

3. In Step 3, Google App Engine automatically scans the queues for any tasks that are queued up and picks them up for execution. In our case, it will find a Task instance and execute it by invoking the URL (/gaejsignupsubscriber) and passing it the relevant data i.e. emailid

4. Finally in Step 4, our Servlet (GAEJSignupSubscriberServlet) is invoked and it will complete its task. To keep things simple in our example, it will currently only print out a message. But it should be obvious that core logic associated with the task would have gone into the Servlet here. For our specific case, it would have involved checking if the user has not signed up already, creating a database record and then sending off a welcome email.

Implementing the above flow

To summarize the above steps in terms of what we have to code, here is the list:

1. Code the GAEJCreateTaskServlet that will accept the request parameter and create a Task instance in the subscription-queue.

2. Code the GAEJSignupSubscriberServlet that will be invoked by Google App Engine automatically. We will currently only print out a log statement because the intent is to demonstrate the whole sequence.

3. Configure our queue (subscription-queue) in a file named queue.xml. This file needs to be placed in the WEB-INF folder of your application.

4. Configure our GAEJCreateTaskServlet and GAEJSignupSubscriberServlet in the web.xml file.

Finally, we can execute our application and use the local development server to see the application in action. Users can optionally even deploy it to the Google App Engine cloud if they wish.

So let us get started.

GAEJCreateTaskServlet.java

This servlet accepts our request for subscription. We shall invoke it via the following url : http://appurl/gaejcreatetask?emailid=XYZ.

package com.gaejexperiments.taskqueue;

import java.io.IOException;

import javax.servlet.ServletException;
import javax.servlet.http.*;

import com.google.appengine.api.labs.taskqueue.Queue;
import com.google.appengine.api.labs.taskqueue.QueueFactory;
import com.google.appengine.api.labs.taskqueue.TaskOptions;

@SuppressWarnings("serial")
public class GAEJCreateTaskServlet extends HttpServlet {
public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException {

String strCallResult = "";
resp.setContentType("text/plain");
try {
//Extract out the To, Subject and Body of the Email to be sent
String strEmailId = req.getParameter("emailid");

//Do validations here. Only basic ones i.e. cannot be null/empty

if (strEmailId == null) throw new Exception("Email Id field cannot be empty.");

//Trim the stuff
strEmailId = strEmailId.trim();
if (strEmailId.length() == 0) throw new Exception("Email Id field cannot be empty.");
//Queue queue = QueueFactory.getDefaultQueue();
Queue queue = QueueFactory.getQueue("subscription-queue");
queue.add(TaskOptions.Builder.url("/gaejsignupsubscriber").param("emailid",strEmailId));
strCallResult = "Successfully created a Task in the Queue";
resp.getWriter().println(strCallResult);
}
catch (Exception ex) {
strCallResult = "Fail: " + ex.getMessage();
resp.getWriter().println(strCallResult);
}
}

@Override
public void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
doGet(req, resp);
}
}

The code listed below is straightforward to understand. It does the following:

1. Extracts out the request parameter (emailid) and does some basic validation on it.

2. It gets a handle to the subscription-queue through the following statement:

Queue queue = QueueFactory.getQueue("subscription-queue");

3. It adds a Task to the above queue by providing a Task URL (/gaejsignupsubscriber) and Data (emailid parameter). It uses a helper class TaskOptions.Builder to help create the instance of the task. As you can see it provides a url and then the param. The task is created by invoking the add method on the queue.

queue.add(TaskOptions.Builder.url("/gaejsignupsubscriber").param("emailid",strEmailId));

4. For the readers information, I have shown a commented out line

//Queue queue = QueueFactory.getDefaultQueue();

which shows how to get the handle to the default queue in case you wish to place your tasks in the default queue itself.

5. Do not that all the Task Queue classes are experimental and are present in the com.google.appengine.api.labs.taskqueue package. This could change in the future.

GAEJSignupSubscriberServlet.java

This servlet contains the core task logic. This will be invoked by Google App engine if it finds any tasks present in the subscription-queue. If any tasks are there, it will pick them up and invoke the URL mentioned in the Task and pass to it the data present in the Task instance.  The code shown below is straightforward, it simply logs a statement saying that it got invoked and it also logs the email id.

package com.gaejexperiments.taskqueue;

import java.io.IOException;
import java.util.logging.Logger;

import javax.servlet.ServletException;
import javax.servlet.http.*;

@SuppressWarnings("serial")
public class GAEJSignupSubscriberServlet extends HttpServlet {
private static final Logger _logger = Logger.getLogger(GAEJSignupSubscriberServlet.class.getName());
public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException {

String strCallResult = "";
resp.setContentType("text/plain");
try {
String strEmailId = req.getParameter("emailid");
_logger.info("Got a Signup Subscriber Request for Email ID : " + strEmailId);
//
// PUT YOUR TASK CODE HERE
//
strCallResult = "SUCCESS: Subscriber Signup";
_logger.info(strCallResult);
resp.getWriter().println(strCallResult);
}
catch (Exception ex) {
strCallResult = "FAIL: Subscriber Signup : " + ex.getMessage();
_logger.info(strCallResult);
resp.getWriter().println(strCallResult);
}
}

@Override
public void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
doGet(req, resp);
}

}

queue.xml

All queues are configured in a file named queue.xml. Google App Engine provides a default queue. This queue is aptly named “default“. But in case you need to define your own queues, which is what we are going to do, we need to define them in a file called queue.xml. This file is placed in the WEB-INF directory of your application. You can also override settings of the default queue by defining it in the file and providing your own values.

Take a look at the queue.xml shown below:

<?xml version="1.0" encoding="UTF-8"?>
<queue-entries>
<queue>
<name>default</name>
<rate>5/s</rate>
</queue>
<queue>
<name>subscription-queue</name>
<rate>5/s</rate>
</queue>
</queue-entries>

In the above configuration, you will find that we have defined our own queue named “subscription-queue”. There is also another element that we have defined for the <queue> called <rate>. This element determines the rate at which you tell Google App Engine to execute tasks. If you do not specify a rate, then the default execution rate is 5 tasks per second. In the above file, we have provided the expression as “5/s”, which reads as 5 per second. Other examples of <rate> expressions are 1000/d (One thousand per day), etc. I suggest to read up the documentation for more examples.

You will also find that we have defined the default queue and we can change the rate if we want. But I have left it as is.

Please make sure that the above file (queue.xml) is present in the WEB-INF folder at the time of deploying the application.

Configuring the Servlets (web.xml)

We need to add the <servlet/> and <servlet-mapping/> entry to the web.xml file. This file is present in the WEB-INF folder of the project. The necessary fragment to be added to your web.xml file are shown below. Please note that you can use your own namespace and servlet class. Just modify it accordingly if you do so. We are defining here both our servlets.

<servlet>
<servlet-name>GAEJCreateTaskServlet</servlet-name>
<servlet-class>com.gaejexperiments.taskqueue.GAEJCreateTaskServlet</servlet-class>
</servlet>
<servlet>
<servlet-name>GAEJSignupSubscriberServlet</servlet-name>
<servlet-class>com.gaejexperiments.taskqueue.GAEJSignupSubscriberServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>GAEJSignupSubscriberServlet</servlet-name>
<url-pattern>/gaejsignupsubscriber</url-pattern>
</servlet-mapping>
<servlet-mapping>
<servlet-name>GAEJCreateTaskServlet</servlet-name>
<url-pattern>/gaejcreatetask</url-pattern>
</servlet-mapping>

Task Execution in Action

I am assuming that you have already created a new Google Web Application Project and have created the above Servlets, web.xml and queue.xml respectively. For a change, we shall be running this episode within our local development server only.

So assuming that all is well, we will run our application, by right-clicking on the project and selecting Run As –> Web Application. Once you see the following message shown below, then the local server is ready to accept requests.

Nov 24, 2009 5:11:33 AM com.google.apphosting.utils.jetty.JettyLogger info
INFO: jetty-6.1.x
Nov 24, 2009 5:11:40 AM com.google.apphosting.utils.jetty.JettyLogger info
INFO: Started SelectChannelConnector@127.0.0.1:8080
The server is running at http://localhost:8080/

Follow the steps given below:

1. Launch the browser on your local machine and navigate to http://localhost:8080/_ah/admin. This is the administrative console of the local development server.

2. Click on Task Queues to view the current task queues that are configured in your application. You should see the screen shown below, which shows that we have configured two task queues: default and subscription-queue.

You will notice that both of the queues currently do not have any tasks since we have not created any.

3. The next step is to create a task in the subscription-queue. To do that, all we need to do is invoke the following url :

http://localhost:8080/gaejcreatetask?emailid=romin@rocketmail.com

This invokes our GAEJCreateTaskServlet that we have configured. It will create the sample task in the subscription-queue and we will see a message as shown below in the browser:

“Successfully created a Task in the Queue”

4. Click on the Task Queues link again, you will find that there will now be 1 task listed in the subscription-queue as shown below:

5. Click on the subscription-queue link. This will display the task instance as shown below:

6. Since this is the development server, no automatic execution of tasks (this will occur in Google App Engine cloud) will take place and you have to run them manually, by clicking on the Run button. Click that. It will display that there are no more tasks present if the task executes successfully.

7. To check that our task has executed successfully, we can visit the Console tab in Eclipse and you will see the success log as shown below:

Moving on

In this episode, we saw how to split up a process into individual tasks and assign them to the Google App Engine for execution. What we have demonstrated here is a simple flow and the specifics of the configuration. I encourage you to try it out in your applications and as an exercise deploy it to the Google App Engine and monitor it there via the Administration Console.

Do keep in mind that this API is experimental and is likely to change drastically. At the same time, if you have any feedback, it would be nice to pass it along to the Google App Engine team.

Till the next episode, Happy Multitasking!

Welcome to Episode 8. In this episode, we shall cover an important service that is provided in the Google App Engine. The service is called Memcache and is used allow applications to manage their data in a cache.

What is a Cache and why do we need one?

As per the Wikipedia definition, a cache is a temporary storage area where frequently accessed data can be stored for rapid access. Once the data is stored in the cache, it can be used in the future by accessing the cached copy rather than re-fetching or recomputing the original data.

The decision to use a cache in your application should come after carefully determining which operations would be benefit from it. Look at the following scenarios:

1. If you have stored information in a database and which is not updated frequently, then it might make sense to put some of that information in a cache so that if multiple requests come for the same data then you can simply look up the memory cache and retrieve it from there, rather than make repeated database calls that are expensive (and which may return the same data).

2. If you invoke external web services and you determine that the same data could be requested, then a cache would help here too to avoid expensive network calls.

There is a lot of material available on the use of a cache in software applications and I suggest reading from those sources to arrive at a good caching design in your applications. We shall keep our discussion here limited to a simple use case of using Memcache, which is a Caching Service provided by Google App Engine and how to implement it quickly in your application.

Before we begin (Important!)

We will be introducing MemCache in an existing application that we have written. This was the Dictionary Service Application that we implemented in Episode 4. I strongly urge you to read up that episode and understand what the application was about and have that project ready to make changes accordingly.

To recap, our GAEJ Dictionary application is shown below:

1. Navigate to http://gaejexperiments.appspot.com/dictionary.html. This will show a page as shown below:

2. Enter the word that you wish to lookup the definition for in a dictionary and it will return you the result as shown below:

Introducing a Cache

The request/response flow for the above Dictionary application is explained below via the diagram shown:

The steps were as follows:

1. The user makes a request to the GAEJ Experiments application by sending a request for the word.

2. The GAEJ Dictionary Service Servlet receives the Request and makes a call to an External Dictionary Service hosted at (http://services.aonaware.com/DictService/DictService.asmx).

3. The Response from the Dictionary Service i.e. the definition of the word is received by the GAEJ Dictionary Service Servlet.

4. The Response is sent back to the user application.

The flow is straightforward and typically found in most applications. Now, what happens if several users make a request for the same word? As per the application flow, each request from the user will result in a call to an external service for the definition, irrespective of whether it was the same word or not. This is wasteful in terms of network resources and the application response too. Coupled with the fact that the definition of a word is not going to change overnight , it would be nice to return back the definition from the GAEJExperiments application itself, if the word definition had been looked up already. Enter a cache!

So, what we are introducing in our application flow now is a cache called GAEJ Dictionary Cache and the modified application flow is shown below:

The steps now would be as follows:

1. The user makes a request to the GAEJ Experiments application by sending a request for the word.

2. The GAEJ Dictionary Service Servlet receives the Request and checks if the word and its definition is already present in the Cache. If it is present in the Cache, then we short circuit and go to Step 6.

Optional Steps (3,4,5)
3. If the Servlet does not find the definition in the Cache, then it makes a call to the External Dictionary Service hosted at (http://services.aonaware.com/DictService/DictService.asmx).

4. The Response from the Dictionary Service i.e. the definition of the word is received by the GAEJ Dictionary Service Servlet.

5. The Servlet puts the word and its definition in the Cache, so that future requests to determine if the word/definition is present in the Cache are fulfilled.

6. The Response is sent back to the user application.

To summarize, we introduced the Cache that functions as follows:

• All word definitions looked up from the External Service are placed in the Cache.
• If a word is already present in the Cache, then it is returned from the Cache itself and an external network call is saved.

The Memcache Service API

A cache is typically implemented as a Map. A Map is a generic data structure that contains a key and its value. You look up the Cache by specifying the key and if it is found, then the value associated with that key is returned. What I am describing here is an over simplification of what a Cache is. There is a lot more to a Cache implementation then just putting a value and getting a value. But we shall keep it simple here.

The Memcache Service API is simple enough to use and there is good documentation available on it here. The Memcache Service implements JSR-107 (JCache Interface). The JCache classes are present in the javax.cache package and that it what you will use.

At a high level, all you need to do is follow these steps:

1. Get a handle to the Cache implementation:

The snippet of code to do that (reproduced from the Documentation) is shown here:

import java.util.Collections;
import javax.cache.Cache;
import javax.cache.CacheException;
import javax.cache.CacheFactory;
import javax.cache.CacheManager;

Cache cache;

try
{
CacheFactory cacheFactory = CacheManager.getInstance().getCacheFactory();
cache = cacheFactory.createCache(Collections.emptyMap());
}
catch (CacheException e)
{
// ...
}

The code is simple. We get a handle to the CacheFactory instance and then create a Cache. Notice that we are creating a Map i.e. an empty Map. Once we have the Map, then all we need to do is play around with the (key,value) pairs.

2. Put a value or get a value from the Cache

Shown below is how we would use the cache in a simple manner. We invoke the put(key,value) method on the javax.cache.Cache instance. Similarly, to extract a value, we need to invoke the cache.get(key) value. It will return us the value if found, which we can then typecast to the appropriate class.

String key;      // The Word
String value;    // The Definition of the Word

// Put the value into the cache.
cache.put(key, value);

// Get the value from the cache.
value = (String) cache.get(key);

GAEJDictionaryCache.java

Let us first discuss a utility class that I have written that encapsulates the Caching API. I have made this class a singleton and it is called GAEJDictionaryCache. The source code is shown below:

package com.gaejexperiments.networking;

import java.util.Collections;
import java.util.logging.Level;
import java.util.logging.Logger;

import javax.cache.Cache;
import javax.cache.CacheException;
import javax.cache.CacheFactory;
import javax.cache.CacheManager;

public class GAEJDictionaryCache {
public static final Logger _log = Logger.getLogger(GAEJDictionaryCache.class.getName());

private static GAEJDictionaryCache _instance;
private Cache cache;

private GAEJDictionaryCache() {
try {
CacheFactory cacheFactory = CacheManager.getInstance().getCacheFactory();
cache = cacheFactory.createCache(Collections.emptyMap());
}
catch (CacheException e) {
//Log stuff
_log.log(Level.WARNING, "Error in creating the Cache");
}
}

public static synchronized GAEJDictionaryCache getInstance() {
if (_instance==null) {
_instance = new GAEJDictionaryCache();
}
return _instance;
}

public String findInCache(String word) {
if (cache.containsKey(word)) {
return (String)cache.get(word);
}
else {
return null;
}
}

public void putInCache(String word, String definition) {
cache.put(word,definition);
}
}

Let us discuss the key parts of the code:

1. The Singleton design pattern should be obvious over here and the application needs to use the getInstance() method to obtain a handle to this singleton.
2. The constructor of this class is private and called only once. In that an instance of the Cache is created.
3. There are two utility methods written : findInCache and putInCache.
4. The application invokes findInCache by providing a key value. If the key is found, the value is written via the cache.get(key) method. For our dictionary application, the key is the word that you wish to look up the definition for.
5. If the application wants to put a key,value record into the Cache, then it invokes the putInCache(…) method that takes the key and the value. For our dictionary application, the key is the word and the value is the definition of the word.

By encapsulating the MemCache Service API in this fashion, you can create a reusable class that takes care of Cache API details. You could then improve upon this class and provide advanced Cache features and APIs and reuse it in all your future GAEJ applications.

Modifying the Original GAEJDictionaryService.java class

All that remains now is for us to modify the existing GAEJDictionaryService.java class by introducing the appropriate cache usage. The modified code is shown below and I suggest to look at the comment //MEMCACHE to make it easier for you to see what I have changed.

package com.gaejexperiments.networking;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.StringReader;
import java.net.URL;
import java.util.logging.Logger;

import javax.servlet.ServletException;
import javax.servlet.http.*;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathExpression;
import javax.xml.xpath.XPathFactory;

import org.w3c.dom.Document;
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;

@SuppressWarnings("serial")
public class GAEJDictionaryService extends HttpServlet {
public static final Logger _log = Logger.getLogger(GAEJDictionaryService.class.getName());
public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException {

String strCallResult = "";
resp.setContentType("text/plain");
try {

//Extract out the word that needs to be looked up in the Dictionary Service
String strWord = req.getParameter("word");

//Do validations here. Only basic ones i.e. cannot be null/empty
if (strWord == null) throw new Exception("Word field cannot be empty.");

//Trim the stuff
strWord = strWord.trim();
if (strWord.length() == 0) throw new Exception("Word field cannot be empty.");

//MEMCACHE
//First get a handle to the Cache
GAEJDictionaryCache _cache = GAEJDictionaryCache.getInstance();

//Determine if the value is present in the Cache
String strWordDefinition = _cache.findInCache(strWord);

//If the word/definition is present in the Cache, return that straightaway, no need for external network call
if (strWordDefinition != null) {
//Return the definition
_log.info("Returning the Definition for ["+strWord+"]"+" from Cache.");
strCallResult = strWordDefinition;
}
else {
_log.info("Invoking the External Dictionary Service to get Definition for ["+strWord+"]");
//Make the Network Call
String strDictionaryServiceCall = "http://services.aonaware.com/DictService/DictService.asmx/Define?word=";
strDictionaryServiceCall += strWord;
URL url = new URL(strDictionaryServiceCall);
BufferedReader reader = new BufferedReader(new InputStreamReader(url.openStream()));
StringBuffer response = new StringBuffer();
String line;

while ((line = reader.readLine()) != null) {
response.append(line);
}
reader.close();

strCallResult = response.toString();

DocumentBuilderFactory builderFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder builder = builderFactory.newDocumentBuilder();
Document doc = builder.parse(new InputSource(new StringReader(strCallResult.toString())));

XPathFactory factory = XPathFactory.newInstance();
XPath xpath = factory.newXPath();
XPathExpression expr = xpath.compile("//Definition[Dictionary[Id='wn']]/WordDefinition/text()");

Object result = expr.evaluate(doc, XPathConstants.NODESET);
NodeList nodes = (NodeList) result;
for (int i = 0; i < nodes.getLength(); i++) {
strCallResult = nodes.item(i).getNodeValue();
}

//MEMCACHE
//Need to check depending on your logic if the values are good
//Currently we will assume they are and put it in the cache
//For e.g. if the word is not found, the Dictionary Service gets the word back.
//So you could use that logic if you want.
_cache.putInCache(strWord, strCallResult);
}

resp.getWriter().println(strCallResult);

}
catch (Exception ex) {
strCallResult = "Fail: " + ex.getMessage();
resp.getWriter().println(strCallResult);
}
}

@Override
public void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
doGet(req, resp);
}

}

Let us discuss the modified flow in brief:

1. We first extract out the word request parameter and do some basic validation to make sure that it is not empty.
2. We get the handle to the Cache by calling our utility class that we just wrote.

   //First get a handle to the Cache
   GAEJDictionaryCache _cache = GAEJDictionaryCache.getInstance();

3. We check if the word is present in the cache. The findInCache method will return null if not found.

   //Determine if the value is present in the Cache
   String strWordDefinition = _cache.findInCache(strWord);

4. If the definition is returned from the cache, then we simply return that and no network call is made.
5. If the definition is not found, then the network call is made, the response stream is parsed out for the definition. And the most important step is to put this definition in the cache, so that the findInCache(…) method will get the definition, the next time another request for the same word is made.
   

   _cache.putInCache(strWord, strCallResult);

Try it out

You can try out the modified Dictionary Service application at: http://gaejexperiments.appspot.com/dictionary.html. The first time that you search for a word, it might take some time to get back the definition but once present in the cache, it is returned much faster. Do note, that if the same word has been looked up by another user, then the word and its definition will be present in the cache.

Cache Design considerations

A cache if implemented well will save your application from repeated resource intensive operations and also improve application response times significantly. Of course, simply putting in a cache comes with risks. Some of the factors, you need to take into consideration are:

1. Determine precisely which operations in your application would benefit from a cache.

2. Analyse if the data that you are putting in a cache changes frequently or not. If it changes frequently, then it might negate the benefits of a cache.

3. Control the size of the cache and prevent it from becoming too large or unmanageable. The way we have implemented it, the cache will keep growing and we are leaving it to the App Engine run time to truncate our cache due to memory limitations if any.

4. Take into consideration several other factors like what is an interval across which you want to refresh the cache, cache expiration policies, etc.

In short, monitoring the cache and tuning its parameters is required from any production worthy application.

Moving on

This brings Episode 8 to an end. Hope you enjoyed reading it.

Welcome to Episode 6 of this series. In this episode, we will learn how your Google App Engine Java (GAEJ) Application can receive incoming email.

In an earlier episode of this series, we have covered how to use the Email Service of GAEJ to send out emails from your application. At that point in time, we were using version 1.2.5 of the AppEngine SDK. That version did not provide support for handling incoming email. Since then a newer version of the AppEngine SDK for Java 1.2.6 has been released. And one of the nice features of this new release is support for incoming email in your application. What it means is that anyone can send an email to your GAEJ hosted application, it can receive an email and then you can process and perform some business logic with that as required.

Prerequisites

Before we move on, it is important that you have a working setup of the environment and are comfortable with developing a new project using the Google Eclipse plugin. If not, please go through earlier episodes that contained detailed setup information along with a few development episodes like using Email Service, using the URL Fetch service, etc.

The most important prerequisite is to make sure that you have upgraded your Eclipse environment to the latest AppEngine SDK i.e. 1.2.6. Please go through the following : Episode 5: Upgrading to Google App Engine 1.2.6

A quick check: To make sure that you  have version 1.2.6 of the SDK installed, do the following:

1. In your Eclipse IDE, go to Window –> Preferences.
2. Navigate to Google –> App Engine
3. You should see version 1.2.6 of the SDK installed. And make sure that it is the default one by selecting it. By selecting it, it will be added to the Build Path of your Google Web Application projects. And we need the latest SDK since it will have support for the Incoming Email feature.

Receiving Email Feature

App Engine now supports incoming email in your applications. Read the official documentation here. Your applications can now receive email and you can parse out the email and determine any business logic that needs to be processed. This opens up a whole new range of applications where you can fulfill requests and send information from your application by allowing users to simply send an email from their favourite email client. Think of it as a Instant Message itself that your application can receive and react to it. We had seen in an earlier episode how through XMPP Support, we can write our own Agent that can receive XMPP messages directly and respond to them. Now with version 1.2.6 of the SDK, the same functionality has got extended to email too. And the best part about it is the consistency with which Google has implemented it.

The steps to follow to receive an email is identical to the ones required to receive XMPP messages:

1. Configure your application to receive incoming email by configuring the Mail Service

2. Write and configure a servlet to receive email

3. Once the application is deployed, anyone can send an email to SomeID@YourApplicationId.appspotmail.com. SomeID is any id like test, admin, support,etc. It is just an id. And YourApplicationId is the application id of your hosted Google App Engine application.

Let us look at each of the above points in detail now. But before we begin, create a New Google Web Application Project (If you wish you can continue to use an existing project to add the incoming Email support, which is what I have done personally, but the choice is yours) . Follow these steps to create a new project:

1. Either click on File –> New –> Other or press Ctrl-N to create a new project. Select Google and then Web Application project. Alternately you could also click on the New Web Application Project Toolbar icon as part of the Google Eclipse plugin.
2. In the New Web Application Project dialog, deselect the Use Google Web Toolkit and give a name to your project. I have named mine GAEJExperiments. I suggest you go with the same name so that things are consistent with the rest of the article, but I leave that to you.
3. Click on Finish

This will generate the project and also create a sample Hello World Servlet for you. But we will be writing our own Servlet.

Configuring the incoming Email Service

This is straightforward and all you need to do is add the following element to the appengine-web.xml file. The appengine-web.xml file as you know is specific to the Google Java Web Application project and is used for configuring certain services. You need to configure the Incoming Email Service so that your application is enabled to receive it, being one of them. It is found in the warWEB-INF folder of your Web Application Project. The XML fragment to add at the end but before the </appengine-web-app> element

<inbound-services>
<service>mail</service>
</inbound-services>

Configure and code a Java Servlet that will receive the incoming Message

All Email messages to your application are delivered via POST to following URL path in your application: /_ah/mail/ as per the Google AppEngine documentation. So you will need to configure the servlet like the following snippet in the web.xml file, present in the warWEB-INF folder of your Web Application Project.

We need to add the <servlet/> and <servlet-mapping/> entry to the web.xml file. This file is present in the WEB-INF folder of the project. The necessary fragment to be added to your web.xml file are shown below. Please note that you can use your own namespace and servlet class. Just modify it accordingly if you do so.

<servlet>
<servlet-name>emailhandler</servlet-name>
<servlet-class>com.gaejexperiments.email.GAEJReceiveEmailServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>emailhandler</servlet-name>
<url-pattern>/_ah/mail/*</url-pattern>
</servlet-mapping>

<security-constraint>
<web-resource-collection>
<url-pattern>/_ah/mail/*</url-pattern>
</web-resource-collection>
<auth-constraint>
<role-name>admin</role-name>
</auth-constraint>
</security-constraint>

In the above snippet, you will find the fixed URL path /_ah/mail/* configured as the <url-pattern/>. And then I have a Java Servlet class com.gaejexperiments.email.GAEJReceiveEmailServlet as the <servlet-class>. The security constraint has been added so that in case anyone invokes your url directly, then only Google Account authenticated users will be able to do that.

Now, all we have to do is write our Servlet. As mentioned, the incoming Email messages will be POSTed to our Servlet, so we need a simple doPost(…) implemented in our Servlet. The code is shown below:

package com.gaejexperiments.email;

import java.io.IOException;
import java.io.InputStream;
import java.util.Properties;
import java.util.logging.Level;
import java.util.logging.Logger;

import javax.mail.MessagingException;
import javax.mail.Multipart;
import javax.mail.Part;
import javax.mail.Session;
import javax.mail.internet.MimeMessage;
import javax.servlet.ServletException;
import javax.servlet.http.*;

@SuppressWarnings("serial")
public class GAEJReceiveEmailServlet extends HttpServlet {
public static final Logger _log = Logger.getLogger(GAEJReceiveEmailServlet.class.getName());

@Override
public void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {

try {

Properties props = new Properties();
Session session = Session.getDefaultInstance(props, null);
MimeMessage message = new MimeMessage(session, req.getInputStream());

//Extract out the important fields from the Mime Message
String subject = message.getSubject();

_log.info("Got an email. Subject = " + subject);

String contentType = message.getContentType();
_log.info("Email Content Type : " + contentType);

printParts(message);
//Parse out the Multiparts
//Perform business logic based on the email
}
catch (Exception ex) {
_log.log(Level.WARNING, "Failure in receiving email : " + ex.getMessage());
}
}

private static void printParts(Part p) throws IOException, MessagingException {
Object o = p.getContent();

if (o instanceof String) {
System.out.println("This is a String");
System.out.println((String)o);
}
else if (o instanceof Multipart) {
System.out.println("This is a Multipart");
Multipart mp = (Multipart)o;

int count = mp.getCount();
for (int i = 0; i < count; i++) {
printParts(mp.getBodyPart(i));
}
}
else if (o instanceof InputStream) {
System.out.println("This is just an input stream");
InputStream is = (InputStream)o;
int c;
while ((c = is.read()) != -1)
System.out.write(c);
}
}

}

Let us discuss the main parts of the code:

1. We have a doPost() method that gets invoked by the Google App Engine when an email is received.

2. In the doPost() method, we build out the email message (a instance of class MimeMessage) using the javax.mail.* classes as shown below:

Properties props = new Properties();
Session session = Session.getDefaultInstance(props, null);
MimeMessage message = new MimeMessage(session, req.getInputStream());

3. We extract out key attributes from the Message like subject, content type, etc.

//Extract out the important fields from the Mime Message
String subject = message.getSubject();
_log.info("Got an email. Subject = " + subject);

String contentType = message.getContentType();
_log.info("Email Content Type : " + contentType);

4. We have a utility method printParts(), that helps simply print out the contents of the message. But you could explore the Java Mail API to parse out the multiparts as required and then incorporate your business logic.
5. To help debug our Servlet, we have put in some log statements along with System Out statements, which we shall look for to verify that the Application did receive email.

Finally, we have used the INFO level to log if the message was sent out successfully or not, so we will have the change the logging level by modified the logging.properties file present in the warWEB-INF folder. The necessary line after modification is shown below:

# Set the default logging level for all loggers to INFO
.level = INFO

Deploying our application

To deploy the application, you will need to first create your Application ID. The Application Identifier can be created by logging in at http://appengine.google.com with your Google Account. You will see a list of application identifiers already registered under your account (or none if you are just getting started). To create a new Application, click on the Create Application button and provide the Application Identifier as requested. Please note this name down since you will be using it for deployment.

For e.g. I have registered an application identifier named gaejexperiments.

To deploy the application, follow these steps (they should be familiar to you now):

1. Click on the Deploy Icon in the Toolbar.
2. In the Deploy dialog, provide your Email and Password. Do not click on Deploy button yet.
3. Click on the App Engine Project settings link. This will lead you to a dialog, where you need to enter your Application ID [For e.g. my Application Identifier gaejexperiments]
4. Click on OK. You will be lead back to the previous screen, where you can click on the Deploy button. This will start deploying your application to the GAEJ cloud. You should see several messages in the Console window as the application is being deployed.
5. Finally, you should see the message “Deployment completed successfully”.

Testing our Application

To test our application, send an email from any mail client to an address within your application. As explained below, the email address to be used is shown below:

SomeID@YourApplicationId.appspotmail.com

For e.g. my Application Id is gaejexperiments, so I can send email to any of the following:

• test@gaejexperiments.appspotmail.com
• user1@gaejexperiments.appspotmail.com
• and so on…

Once the email has been sent successfully from a Google Mail Account or Yahoo Mail or Outlook/Thunderbird, etc – you can use the App Engine console to verify if your application received the email or not. To do that, perform the following steps:

1. Go to http://appengine.google.com and log in with your account.
2. You will see a list of applications registered. Click on the application that you just deployed. In my case, it is gaejexperiments.
3. When you click on a particular application, you will be taken to the Dashboard for that application, which contains a wealth of information around the requests, quotas, logs, versions, etc. This will be a subject of a future episode but for now, it is sufficient to say that you can come over here to monitor the health of your application and also to analyze what is going on.
4. Click on the Logs link as shown in the screenshot below:
5. This will display the application log. And all your application log statements that you code using the Logger class can be visible here.
6. By default, the severity level is set at ERROR and we can change that to DEBUG, the lowest level and you should be able your log statements that had the log level of INFO. This was the log level at which we had logged statements like Received an Email, etc in our Java Servlet, so that is what we need to check for.
7. If you see the statements, it means that the message has been received. Shown below is a screen shot of the log for a message that I sent.

Once this mechanism is in place, it means that your application has been correctly setup and deployed for receiving email messages. You can now build in business logic on what you need to do when an email is received.

Hope you had a good time reading this episode. The next episode I plan to extend an existing episode in which we built an XMPP Bot and deployed it on the Google App Engine. I plan to extend the bot to support the collaboration tool Google Wave, so that your bot can participate in a Wave conversation too. Till then, happy GAEJ’ing!

Welcome to Episode 4 of GAEJ Experiments. In this episode, we will learn how your GAEJ Application can invoke external URLs and get data back from them. In the process we will build a fully capable Dictionary Application. While it may not be a complete application, it will demonstrate just about enough how you can integrate external services into your application.

This episode is particularly important given the fact that there are hundreds of public APIs now available out there on the web, which your application can integrate. I strongly recommend visiting The Programmable Web that catalogs hundreds of Web APIs that are available today and which you can integrate into your application.

Dictionary Application in Action

In order to maintain a consistent style across the episodes so far, let us first watch the application in action to better understand what we will be building over here. Follow these easy steps:

1. Navigate to http://gaejexperiments.appspot.com. This will result in a page as shown below:
2. Click on Dictionary Service link. This will lead you to a page shown below:
3. Enter any word that you wish to lookup in the dictionary and click on Lookup Dictionary. For e.g. engine. This will display the meaning of the word as shown below:

Behind the Scenes

Let us understand how the above application is working. The key points to note in the flow are given below:

1. The dictionary page shown in step 2 above is a simple HTML form page. The only user input needed in this step is the word that you wish to look up in the dictionary.
2. On click of the Lookup Dictionary button, a Servlet is invoked in our GAEJ Application. The Servlet will utilize the GAEJ URL Fetch Service to invoke an external Dictionary Service.
3. The Dictionary Service is hosted at a site called http://services.aonaware.com/DictService/DictService.asmx . One of the service operations is Define and you can try it out here. It supports a REST style interface (GET/POST) for invoking it, so that makes our task simple. Give it a try here to better understand what we are integrating or simply punch in the following in your browser: [code language="java"]

   http://services.aonaware.com/DictService/DictService.asmx/Define?word=trial

   [/code]

4. This service takes in a word and returns us a XML Response which contains the meaning of the word in several dictionaries. We parse out the dictionary meaning and return that back to the client (HTML form).
5. The HTML form that renders the meaning as shown in the screen above.

URL Fetch Service

The core piece that we will be utilizing in our application here is an ability to invoke external URLs. This is provided by the URL Fetch service of the Google App Engine. It allows us to invoke any external URL and receive the data from the URL. At a high level, usage of the URL Fetch service is pretty simple and a boiler plate code is shown below: [Most details have been omitted]

try {
URL url = new URL(PUT_EXTERNAL_URL_HERE);
BufferedReader reader = new BufferedReader(new InputStreamReader(url.openStream()));

String line;
StringBuffer responseData = new StringBuffer();

//Read the entire response in this loop
while ((line = reader.readLine()) != null) {
responseData.append(line);
}
reader.close();

//Process the data from the responseData StringBuffer
//Might involve Parsing the XML or JSON Response format, etc.

}
catch (Exception e) {
//...
}

This is pretty much standard Java code and it involves opening an inputstream to the external URL and collecting the response in a local variable. You can then decide what to do with the response data received. For e.g. most external services provide a XML or JSON formatted response, which you may have to parse out and then proceed with your application logic.

In our Dictionary Application here, we will be receiving a XML response, which we will simply parse out and return back to the HTML client.

Developing our Application

The first thing to do is to create a New Google Web Application Project. Follow these steps:

1. Either click on File –> New –> Other or press Ctrl-N to create a new project. Select Google and then Web Application project. Alternately you could also click on the New Web Application Project Toolbar icon as part of the Google Eclipse plugin.
2. In the New Web Application Project dialog, deselect the Use Google Web Toolkit and give a name to your project. I have named mine GAEJExperiments. I suggest you go with the same name so that things are consistent with the rest of the article, but I leave that to you. In case you are following the series, you could simply use the same project and skip all these steps altogether. You can go straight to the Servlet Development section.
3. Click on Finish

This will generate the project and also create a sample Hello World Servlet for you. But we will be writing our own Servlet.

The Front end HTML form [dictionary.html]

Create a new HTML file in the war directory of your application. In that directory, you will find an index.html that is generated for you. So you can either use index.html or generate a new HTML file as I have done. Name this file as dictionary.html and code it as shown below:

The first thing that we will do is to code out a simple HTML form that takes a single parameter as an input. This parameter is the word that we need to lookup in the dictionary. We will use a bit of AJAX here to dynamically call our servlet and display the response that we get.

<html>
<head>
<script type="text/javascript">
var xmlhttp;
function lookupDictionary(word)
{
xmlhttp=null;
if (window.XMLHttpRequest)
{// code for IE7, Firefox, Opera, etc.
xmlhttp=new XMLHttpRequest();
}
else if (window.ActiveXObject)
{// code for IE6, IE5
xmlhttp=new ActiveXObject("Microsoft.XMLHTTP");
}
if (xmlhttp!=null)
{
xmlhttp.onreadystatechange=state_Change;
var url = "/gaejdictionaryservice?word="+word;
xmlhttp.open("GET",url,true);
xmlhttp.send(null);
}
else
{
alert("Your browser does not support XMLHTTP.");
}
}

function state_Change()
{
if (xmlhttp.readyState==4)
{// 4 = "loaded"
if (xmlhttp.status==200)
{
// 200 = "OK"
document.getElementById('DictionaryServiceResponse').innerHTML=xmlhttp.responseText;
}
else
{
alert("Problem looking up Dictionary Service :" + xmlhttp.statusText);
}
}
}
</script>
</head>

<body>
<h2>Dictionary Lookup </h2>
<hr/>
<h3>(Powered by Aonaware <a href="http://services.aonaware.com/DictService/">Dictionary Service</a>)</h3>
<hr/>
<p>
<b>Lookup meaning of word:</b><input type="text" id="word"></input>
</p>
<p><b>Meaning :</b>
<br /><span id="DictionaryServiceResponse"></span>
</p>

<button onclick="lookupDictionary(word.value)">Lookup Dictionary</button>

</body>

Here are some key points from the code above. For most web programmers it is pretty much standard stuff:

1. We have a single input field HTML element with an id named word. There is a button with a label Lookup Dictionary, on click on which, we invoke a Javascript function called lookupDictionary, passing in the word value.
2. The lookupDictionary method builds up the standard XMLHttpRequest object that we shall use to send our request across. The request url is /gaejdictionaryservice with a single parameter named word, which contains the value of the word that we need to lookup in the dictionary.
3. Note that /gaejdictionaryservice is our servlet endpoint that we will be seeing in the next section. This servlet will take it the word parameter, and use the URL Fetch Service to invoke the external Dictionary API and return us back the response.
4. The response when returned is inserted into the span element named DictionaryServiceResponse in the above HTML form to display the response received.

Next we shall code the heart of our application a servlet named GAEJDictionaryService.

Coding the GAEJDictionaryService Servlet [GAEJDictionaryService.java]

Create a new Servlet in your Project as shown below. I have created the GAEJDictionaryService.java in the package com.gaejexperiments.networking. You can choose a package of your choice.  The code is straightforward and is listed below:

package com.gaejexperiments.networking;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.StringReader;
import java.net.URL;
import javax.servlet.ServletException;
import javax.servlet.http.*;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathExpression;
import javax.xml.xpath.XPathFactory;

import org.w3c.dom.Document;
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;

@SuppressWarnings("serial")
public class GAEJDictionaryService extends HttpServlet {
public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException {

String strCallResult = "";
resp.setContentType("text/plain");
try {
//Extract out the word that needs to be looked up in the Dictionary Service
String strWord = req.getParameter("word");

//Do validations here. Only basic ones i.e. cannot be null/empty
if (strWord == null) throw new Exception("Word field cannot be empty.");

//Trim the stuff
strWord = strWord.trim();
if (strWord.length() == 0) throw new Exception("Word field cannot be empty.");

String strDictionaryServiceCall = "http://services.aonaware.com/DictService/DictService.asmx/Define?word=";
strDictionaryServiceCall += strWord;
URL url = new URL(strDictionaryServiceCall);
BufferedReader reader = new BufferedReader(new InputStreamReader(url.openStream()));
StringBuffer response = new StringBuffer();
String line;

while ((line = reader.readLine()) != null) {
response.append(line);
}
reader.close();

strCallResult = response.toString();

DocumentBuilderFactory builderFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder builder = builderFactory.newDocumentBuilder();
Document doc = builder.parse(new InputSource(new StringReader(strCallResult.toString())));

XPathFactory factory = XPathFactory.newInstance();
XPath xpath = factory.newXPath();
XPathExpression expr = xpath.compile("//Definition[Dictionary[Id='wn']]/WordDefinition/text()");

Object result = expr.evaluate(doc, XPathConstants.NODESET);
NodeList nodes = (NodeList) result;
for (int i = 0; i < nodes.getLength(); i++) {
strCallResult = nodes.item(i).getNodeValue();
}

resp.getWriter().println(strCallResult);

}
catch (Exception ex) {
strCallResult = "Fail: " + ex.getMessage();
resp.getWriter().println(strCallResult);
}
}

@Override
public void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
doGet(req, resp);
}

}

Let us go over the key points in the code above:

1. I have provided both GET and POST handlers in the servlet and the POST handler simply invokes the GET handler here.
2. Then we parse the request parameters for the word parameter that we need to look up in the dictionary (word) and we do some basic validation to make sure that it is not empty.
3. The Dictionary Service that we plan to use is available at the following URL: [code language="java"]

   http://services.aonaware.com/DictService/DictService.asmx/Define?word=[YOUR_WORD_HERE]

   [/code]

4. In the above URL, you need to provide the word. So what we do in the code is to simply append the word request parameter that was passed.
5. Next, we use the URL Fetch Service as discussed earlier to collect the entire response.
6. The response returned to use is in XML format and the service returns us the meaning of the word based on 6 dictionaries. We will be using just one of those dictionaries WordNet 2.0 which is the 3rd definition in the XML. I suggest that you punch in the following url to understand what we will be parsing out here. I have used the word ‘engine’ here.

   http://services.aonaware.com/DictService/DictService.asmx/Define?word=engine

7. Finally we use XPath. I intentionally used this to demonstrate how easy it is to use XPath to extract out the element text that we are interested in. You are free to choose an alternative way of extracting out the text value. You could use standard SAX/DOM parsing too if you wish. Whatever you are comfortable with will suffice for the example.
8. We first build the Document object by using the standard DocumentBuilderFactory and DocumentBuilder classes.
9. Then on the Document object doc, we evaluate the XPath expression. The XPath expression is //Definition[Dictionary[Id='wn']]/WordDefinition/text().
10. The XPath expression given above can be read as following. First consider the //Definition[DictionaryId ='wn']] which means that find all definitions anywhere in the document which have a child element named DictionaryId whose value is ‘wn’. This is the Definition that we are interested in extracting.
11. Once that is found, comes the rest of the XPath expression, which says that for that Definition element found, get a child element named WordDefinition and extract out its text() value. This is returned as a collection of Text nodes.
12. Finally, we iterate through the Text Nodes and get the value which we then send back as a response.
13. I suggest that if you are still having a problem following the code, try out the URL as mentioned in step 6, study the XML and then the XPath expression will become clear. The rest of the code is standard XML/XPath code from the Java SDK.

Servlet Configuration

To complete our Servlet development, we will also need to add the <servlet/> and <servlet-mapping/> entry to the web.xml file. This file is present in the WEB-INF folder of the project. The necessary fragment to be added to your web.xml file are shown below. Please note that you can use your own namespace and servlet class. Just modify it accordingly if you do so.

<servlet>
<servlet-name>GAEJDictionaryService</servlet-name>
<servlet-class>com.gaejexperiments.networking.GAEJDictionaryService</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>GAEJDictionaryService</servlet-name>
<url-pattern>/gaejdictionaryservice</url-pattern>
</servlet-mapping>

Deploying and running your application

To deploy the application, you will need to first create your Application ID. The Application Identifier can be created by logging in at http://appengine.google.com with your Google Account. You will see a list of application identifiers already registered under your account (or none if you are just getting started). To create a new Application, click on the Create Application button and provide the Application Identifier as requested. Please note this name down since you will be using it for deployment.

For e.g. I have registered an application identifier named gaejexperiments.

To deploy the application, follow these steps (they should be familiar to you now):

1. Click on the Deploy Icon in the Toolbar.
2. In the Deploy dialog, provide your Email and Password. Do not click on Deploy button yet.
3. Click on the App Engine Project settings link. This will lead you to a dialog, where you need to enter your Application ID [For e.g. my Application Identifier gaejexperiments]
4. Click on OK. You will be lead back to the previous screen, where you can click on the Deploy button. This will start deploying your application to the GAEJ cloud. You should see several messages in the Console window as the application is being deployed.
5. Finally, you should see the message “Deployment completed successfully”.

This means that you application is ready to serve. Depending on whether you used index.html or dictionary.html, you should be able to access your application at the following url:

http://<YourAppId>.appspot.com/dictionary.html

or

http://<YourAppId>.appspot.com/index.html

Type in a word and check if the meaning is being returned.

Moving forward

In this episode, we have seen how to utilize the URL Fetch Service provided by the Google App Engine. This service provides us with a convenient way to access external services that can be integrated into our application. Creating Mashup applications would be straightforward with the URL Fetch Service.

You can integrate other external services, RSS feeds in this fashion. Just give it a try.

Till the next episode, good bye and happy coding!