Scalability Unlimited

A GigaSpaces solution to the Poison Message problem

2009-08-17T08:30:00.000-07:00

GigaSpaces uses the notion of Event-containers for matching and handling data in the Space as events. This can be done in either Polling mode (the container polls the Space for data matching the criteria), or Notify (the Space pushes matching data as events to the container).

In case of a polling-container, the object should be changed and returned to the Space, to ensure it no longer matches the criteria. Otherwise the event would be re-triggered.

Now, what would happen in case the event-handler encounters a (permanent or temporary) failure during processing, such as a down-stream system is not available?

Well, clearly, the object is not handled, transactions will be rolled-back, etc, but most importantly: the object is returned to the Space in its original state. This means the object will still match the polling-criteria. This can cause endless loops in your polling container, unnecessarily consuming resources, and the problem will only disappear when the underlying problem is resolved.

This problem is known as the Poison Message problem (see for example this link).

A solution is to add try-catch blocks in your event-handler, and handle the problematic events appropriately to avoid loops. Simply catching and handling the exception has an issue though: It does not take in account that the problem may only be temporary. Therefore adding a retry-mechanism may make sense in your application.

However, adding a retryCount and maxRetries to the problematic object will not work: We cannot modify the erroneous object and write it back to the space, as we must revert the transaction under which it failed - there might have been very viable reasons for the exception in the first place!

For this reason we should use a separate object, which I call a Hospital object. This Hospital object holds a uid-reference to the erroneous object, and can include fields like retryCount and maxRetries.

Here's a snippet of what this Hospital object might possibly look like:

public class Hospital implements Serializable {
    private String id;
    private String poisonObjectId;
    private Integer retryCount;
    private Integer maxRetries;
    private Integer routingId;

    public Hospital() {
    }

    // Getters and setters beyond this line
}

Although a try-catch block in your event-containers will work, I have used an Aspect that intercepts any exception from a (polling) event-container:

 
@Pointcut("@annotation(org.openspaces.events.adapter.SpaceDataEvent)")
private void spaceDataEvent() { }

@Around("spaceDataEvent()")
public Object exceptionHandling(ProceedingJoinPoint pjp)
   throws EventContainerRetryException {
 Object o = null ;
 try {
           o = pjp.proceed();
 } catch (Throwable e) {
           eventContainerExceptionHandler.handleException(
               (BaseObject)pjp.getArgs()[0],
               e
           );
 }
  
 return o ;
}

In the aspect we use an exception handler, that is invoked whenever any exception occurs. This handler takes care of finding and writing the hospital object, incrementing the retry-count, etc:

   
public void handleException(BaseObject baseObject, Throwable e)
   throws EventContainerRetryException
{
 Hospital template = new Hospital();
 template.setPoisonObjectId(baseObject.getId());
 Hospital hospital = hospitalGigaSpace.read(template);
  
 if ( hospital == null ) {
     hospital = new Hospital();
     hospital.setPoisonObjectId(baseObject.getId());
     // Set other fields
 }
  
 if ( hospital.getMaxRetries().equals(hospital.getRetryCount()) ) {
     // Maybe you want to do something generic here to avoid the event
     // from being triggered again, for example setting a status flag.
 } else {
     // Increase the retry count on the hospital object
     hospital.setRetryCount(hospital.getRetryCount() + 1) ;
     // Write the hospital object back, if you want with a lease-time
     hospitalGigaSpace.write(hospital, 120000);
     // Rethrow the initial exception caused in the event container
     // This will cause its transaction to be rolled back.
     throw new EventContainerRetryException(e) ;
 }
}

What is important to realize is that the operations on the hospitalGigaSpace bean should work under a different transactional context from the event container: the polling container transaction must be rolled back while the operations on the hospital object must succeed.

BTW, in a next release of GigaSpaces a generic exception handler will be implemented, which can be used instead of try-catch blocks or AOP.

Busy days...

2009-06-20T00:55:00.000-07:00

It's been extremely busy lately, travelling around a lot, here are some highlights of the recent two weeks...

A week ago I spent a full week in New York, giving a 5-day training (Basic and Advanced) at the GigaSpaces offices just opposite Grand Central Station. A great location, and a great team, that travelled down from Connecticut, New York, San Fransisco, even somewhere in Georgia ;). As I've encountered before, this kind of training is very exhausting for the trainer (and students!), but it gives great energy too when the concepts start to 'live'.

Yesterday I gave a one-hour presentation at the Profict Summer Camp to explain the vision of GigaSpaces with respect to Cloud Computing. We finished with good weather and a BBQ. The other speakers were Jerome Barnard, who is working on an open-source project named ElasticGrid, and Benoît Chesnau who is member of the Apache CouchDB project. Both two very interesting guys that work on challenging projects that share the GigaSpaces vision with respect to more dynamic application design that is not influenced by the 'magic' dimensions 'scalability', 'performance' and 'high availability/reliability'. Thanks Profict for inviting me to this well-organized session at your really great offices!

Coming Monday I will host a GigaSpaces-workshop for 18 people of The Future Group, where I will show them the highlights of the GigaSpaces product in-4-hours. Apparently it's a mixture of Java and .NET people, so this is going to be a lot of fun.

Only a few more days to go before GigaSpaces 7.0 is released. We're working hard on our side to complete the first release of the Eclipse Plugin, so it can be announced with the GA of GigaSpaces. We've completed the roadmap of the plugin, and it's going to be nicely stuffed with great features, such as managing (Deploy and Undeploy) Processing Units, managing (starting, stopping) Service Grid Components, etc, etc. All from your Eclipse IDE. We'll make sure we'll host an update-site, and will provide a way to give feedback, to make it even better.

Till next time!
Jeroen

Eclipse Plugin for GigaSpaces

2009-04-23T13:11:00.000-07:00

One of my colleagues - Robin van Breukelen - is a graduation student at the Hogeschool Arnhem Nijmegen (www.han.nl). For his graduation project, he is currently working on the implementation of a plugin for Eclipse, that will allow you to manage the GigaSpaces runtime from your IDE. The goal is to give developers an even friendlier developer-experience to GigaSpaces.

Some of the key-functionality that we already delivered is the following:

View the Grid Service Agents (GSA's), Managers (GSM's), Containers (GSC's) and Lookup Services.
Starting and stopping the above components, either embedded in the IDE, or externally as separate processes.
View the Processing Units and Spaces.
Deploying and undeploying Processing Units.
Launching Processing Units embedded in the IDE, through dedicated launch configurations for this.
Managing more than one GigaSpaces installation through the Eclipse-preferences.

In essence it means we're rebuilding part of the GigaSpaces Management UI, however, since we're using the new Admin API of GigaSpaces 7.0, this is quite trivial. This API allows you to retrieve information on running components, and administer these. Building the Eclipse Widgets actually turns out to be the hard part.

We've abstracted the GigaSpaces installation from the view that is being rendered, so support for new versions of the product will be very easy to incorporate.

Here is an image of the Eclipse view:

This is what the Preferences window for GigaSpaces installations looks like:

And a specialized Run Configuration for Processing Units:

I am very enthusiastic about these new developments - and although we're only in the fourth Sprint of the development cycle Robin has already created some impressive work. Once the projects has reached a mature stage, we will start releasing the components to the public.

In fact, there is more stuff coming that will be even more compelling - stay tuned!

Jeroen

Training in Pune, India

2009-01-11T06:28:00.001-08:00

I just returned from a short trip to India, where I gave a 3-day GigaSpaces training for a financial company with a development branch in Pune.

Having worked in India before, it surprised me how easily memories can slip away. It was a good refresher, and I made an oath for 2009 to not complain about the traffic jams in the Netherlands for the next few months, and also to eat more Indian food :).

In my class there were 7 bright and nice people. All of them had some or a little more exposure to GigaSpaces as the company is already a customer. During the three days we ploughed through a hectic schedule based around the e-Auction Use Case. The e-Auction use case is a fictitious, but very common example on how success can turn into failure by using a traditional tier-based architecture. By nature such architectures are very hard to scale, especially if you want to combine scalability with aspects such as performance and fault-tolerance. At some point (and sometimes this can happen very early!) these aspects start to bite eachother, making it ever-more difficult to walk the fine line of balancing throughput with reliability and scalability aspects, and at the same time maintain ease of implementation and deployment.

In the use-case we demonstrate how to create a architecture using GigaSpaces that can scale beyond the regular life-expectancy of the application or beyond any foreseen or unforeseen success- and growth-rate, while at the same time being extremely fault-tolerant and performant. To explain this in a matter of three days we normally cover the following subjects:

GigaSpaces Basics, such as product walkthrough, topologies.
Basic API's for storing and retrieving data into the GigaSpaces In Memory Data Grid.
Processing Units, which is the beating heart of GigaSpaces.
Concepts and semantics for creating a Messaging Grid.
Task Execution and Remoting, allowing for Grid Computing paradigms such as Map/Reduce, Master/Worker.
SLA's which will let GigaSpaces scale on demand, or recover from failures.
Web Application Support.
Mirror-Service Support, which allows for reliable, but asynchronous persistency to a disk.
Since GigaSpaces is very reliant on Spring, we usually also include a short topic on Spring.

All of the above is lab-driven, lots of coding, and very hands-on. The training was a lot of fun to do, and my overall stay has been perfect: well-accomodated, great great food and some wonderful people. Here's a nice snap:

I believe the team in Pune is ready to take on action! Good luck guys!