An Analyst's Development Environment

Here in the land of academic research we're working with a "new" take on mashups. It seems like a no-brainer to me but a lot of people have expressed interest and surprise when I explain to them what we're doing. For now let's call it an analyst's development environment (ADE).

One thing that mashups are really, really good at is taking disparate data sources and allowing "momentary" relationships in the sources to be created. This in effect creates a new data source that is a fusion of the inputs. As is often the case in fusions, this new source tends to be more than just the sum of the parts. You often come up with new views on the data as you add extra sources.

Most people stop here at the fusion stage. Once they have the new view onto the data they rely on other tools outside the scope of a mashup to do interesting things. They might pipe that data into a tool such as Fusion Charts in order to visualize it or they might pipe it into an analysis tool such as a model or sim. But, why do they need to leave the scope of the mashup to do this? What if that analysis or the creation of the Fusion Charts XML was an automated part of the mashup itself?

Mashups deal with web services primarily (though there are some nifty products out there that allow you to mash more than just web services). A web service is usually considered to be a data source. But, in practice they are much more than that. Consider all of the specialized web services provided by Google for geolocation or Amazon for looking up aspects of books. The simplest example I can give you is Google's web service which converts an address to a lat and long pair (called geocoding). With these in mind let's take a different look at web services. Let's look at them as processing units.

A processing unit has 3 criteria: it takes input; does something interesting with that input; and provides output. Processing units are the basis of modern programming. They're known as methods, functions, procedures, etc. depending on context. We can most often build bigger processing units from simpler units.

Web services fit these 3 criteria handily. You can easily provide input, they can easily do something interesting with that input and then just as easily provide output. All communication is done in a standardized protocol driven environment.

The interesting thing about web services is that we can string them together (with the right tools) rather easily into processes. That's exactly what we're doing here. Each web service is either a data source or a processing unit. Given the ability to ferry data from one web service to the next (in an easy way) it is possible to create mashups that do more than just mash data. They actually do some form of processing.

Consider what it would be like if you had a web service endpoint attached to a model? You could pre-mash your data from various sources then run it all through the model and create a new output that would be very interesting. It would be so easy.

Using Presto we recently put together a demo which worked along these lines. It made our demo come together in several weeks rather than over several months. We used Presto to access databases then ferried that data (in XML format) into a custom built web service that took said data and ran XSL transforms on it. That produced Fusion Charts XML which we then piped into our presentation layer for visualization. It was easy.

Here is a diagram of what the actual flow of the mashup was.

Here is a screen shot of the actual chart produced by the generated Fusion Charts XML.


An ADE would work in a similar way. Using provided tools which allow for ferrying of data from one endpoint to another and given a grab-bag of analysis and transformation web services an analyst could create some amazing things with little effort or technical know-how. The only developer support would be in the creation of any custom web services. It could be a very powerful tool.

It's a Transforming Process!

So, right now I'm working on a gadget that takes in generic info and sends out Fusion Charts XML. It's a SOAP service and there will be many service endpoints, but right now there are only 2, one for a simple, single series bar chart, and one for a multi-series "drag node chart" (think network diagram with drag able nodes).

I chose to go about it in a different manner than I've seen a lot of people use for Fusion Charts, though. The prevailing way that I've seen people create Fusion Charts XML is to take the data in on the JavaScript side and create the XML, in string format, in the JavaScript. For this approach, I have only one thing to say: Building XML in JavaScript is less than optimal (translation: it sucks).

So, I decided to go about it in the web service itself. My web service is written in Java and once you get the question into Java a few, more palatable, alternatives suggest themselves. In my web service there are 3 distinct transformations: request object to traditional object; traditional object to simplified XML; simplified XML to Fusion Charts XML.

The request object to traditional object transformation is really the beast. The inputs for the endpoints are comma-delimited strings. A lot of work goes into parsing those strings and putting them into the more traditional object. I have my inputs be comma-delimited strings so that the Presto JUMP requests can invoke them effectively. I could just as easily have one of my endpoints be a direct invocation of the more traditional object, but as I understand it, that's a bit of a bad practice.

Once I have my traditional object the easiest step occurs. In this step I use XStream to serialize the traditional object into a simplified XML. If you've never used XStream, it's very simple, very powerful and I recommend it highly.

The last step is where the real magic happens, though. Here is where I transform the simplified XML into Fusion Charts XML. I use the Saxonica XSLT engine to do the transformation and it's a matter of using the right tool for the right job (with regards to using XSLT to transform XML).

XSLT is designed to transform XML, whether it be from XML to XML or XML to some other language. You write a transformation wherein you process the source XML and then create a document in the desired format. It's really not all that hard to take the simplified XML and transform it into the Fusion Charts XML.

Once I have the Fusion Charts XML document I send it back out of the service in a special response that contains the document in string format and the name of the Fusion Chart swf file that will correctly process that document. When my response arrives at its destination all that needs to be done is input the Fusion Charts XML document into the Flash engine with the correct swf file pulled up and voila, it's all done.

I like this approach in that it moves all of the heavy lifting out of the display side (the mashlet, in my case) and into a much more suitable environment, that being a Java web service. I don't have to do endless string concatenation that is hard to debug inside of the JavaScript presentation layer. As a matter of fact, I can write all of the pieces independently of each other and then put them all together in the end. It's a nice break up of all of the work.

An acknowledgment needs to go to @angleofsight for his help in getting this whole process set up. Without his paving of the way I wouldn't be anywhere near as far along as I am right now.