Pages

    Monday, March 15, 2010

    What Should be Next for Enterprise Mashups?

    My friend over at JackBe, Chris Warner, recently sent me an email asking me what the next 2 to 3 years should hold for Enterprise Mashups. JackBe, in case you don't know, is the creator of the Enterprise Mashup software Presto.  We've used Presto on a project and it's a wonderfully crafted piece of software with many, many possibilities in front of it.  Presto has, at its heart, the creation of mashups, those elusive but so useful beasts which give you all kinds of new-fangled views into your data.

    This is a topic I've spent a little bit of time thinking about, fancying myself to be at least a passable user and creator of mashups.  I'm going to change up the question a bit, though, and delve into territory that is slightly afield.  So, I propose these related questions:  Just what the heck are Enterprise Mashups?  And, what's next for them?

    Any Ol' Programmer
    Let's be honest, any programmer can get out there and create a mashup with a little elbow grease.  The technology is there such that, with a little bit of web service lifting and a little bit of UI know-how you can create fantastical mashups that will make everyone sit up and take notice.  What's more, if you do it right, your own mashup can, in turn, become a data source for someone else to consume.  It's a bit like Sir Elton John says, that whole Circle of Life thing.

    But, that's just the problem, isn't it?  We're not all programmers, now, are we?  Not all of us even have access to programmers (or, as we like to style ourselves now-a-days, "software engineers"), so it's not like we can ask someone else to do it for us.  And, most important of all, programmer time is expensive.  If you do have access to a programmer and task them with spending hours creating your pet mashup, you're likely to get a whack on the wrists from the company dumby stick.  So, what use are mashups to the common-day man?  They're nice to look at, but not really practical beyond that.

    Enter the Enterprise
    No, not the ship, though that would be cool.  The Enterprise is (or should be) all about putting a useful tool into the hands of someone other than an engineer.  To a certain extent it's about lowering the technical barrier of entry.  Any programmer should be able to put a mashup together, but that doesn't mean that they are easy.  There is a lot of wiring together and hacking things up that takes place.

    Enter the folks at JackBe, handing us Presto.  With software like Presto it's possible for someone of a less technical bent, someone that's not a programmer, to put together a mashup that's useful.  This Enterprise tool takes some of the difficulty out of the creation.  And that, in a nutshell, is what the Enterprise should be about.

    What's Next?
    Even though products like Presto lower the barrier of entry, there is still a long way to go.  Presto shines at doing the actual mashing, wiring the data together to get the new view.  But, one area where Enterprise Mashup software needs to tread next is into more robust presentation of that new view.

    One thing that mashups excel at is the creation of data.  One thing that data desperately wants to do is be visualized.  There are so many tools out there for visualizing data, but most of them have a similar dilemma to non-enterprise mashups:  it takes specialized skills to use them.  Enterprise Mashups need to tread here, next, lowering the barrier of entry into rich presentation of the data that mashups are so good at generating.

    To Infinity...
    From there, well, who knows.  I'd like to see tools which are good at creating not just a good presentation but actual interaction, all done so that the common layman can create them with minimal know-how.  I've blogged before about how it's important for people like analysts to be able to do at least some kind of rudimentary software development.  I see that Enterprise Mashups can fill this role once they come to full maturity.  It's a big area, one that needs more exploration, and I'm quite happy that I get a peek into it at the ground floor.

    Wednesday, December 30, 2009

    Social Media: Capturing Knowledge in a Self-Organized Way

    It's the early 80s and you sit at your terminal with a stack of papers, a document holder and a keyboard. Your mission: Enter as many of the paper forms into the terminal as possible. Exciting work, isn't it?

    The problem is that this is not an inaccurate way to view data entry today. Granted, a lot of the brute force work has been done and legacy systems exist from which to pull data. Further, the forms are now entered directly into the system as opposed to copied from paper, but as regards entering novel data the situation has changed little.

    The other major problem is that this type of raw entry, which is generally entering data into a form, only captures defined phenomena. The data that is being entered, especially into a form, is often classified and defined in advance. There is no elasticity to what can be captured.

    This is problematic in that you must have a clear picture of what you are capturing in advance. For hard problems and complex situations you very rarely know much, if anything, in advance. If your only valid form of capturing data is via traditional predefined methods, such as forms, then your ability to capture data, and eventually knowledge, is vastly compromised.

    This revelation is nothing new, of course. People have been trying to innovate data entry and knowledge capture for several decades. But, what other types of data can be captured and how?

    The Army is asking this exact question, if indirectly. In reading several SBIRs the concept of capturing the knowledge inherent in soldiers heads is coming to the forefront. It is being recognized that not only do experts have valid perspectives and answers, the boots on the ground do, as well (keep in mind, this is probably not a new perspective in the military, but is one that I have seen in several SBIRs recently). Beyond that, though, they are starting to explore how to bring that knowledge into existing systems.

    The how of this is a serious question. Computer systems today are clearly defined and generally purposeful to a single end. Human thought, on the other hand, is often multi-purposed and the field of understanding human thinking (philosophy) has been around for as long as humans and has yet to reach one shared conclusion on how we think. Even if we could get some Matrix-like data jack implanted into soldiers heads could we really transfer the knowledge as it is represented inside their brains into a computer system?

    Perhaps we need another way of gathering data since it would seem that direct access to the human mind would avail us little. There is a new(-ish) movement which is providing an answer: social media.

    Probably one of the most useful aspects of social media is how important and interesting knowledge percolates to the top. This is done in various ways. For instance, if I see an interesting tweet on Twitter, I will retweet it. If I read something worthwhile on Facebook, I may comment on it or repost it. It's this interaction with the content that causes the interesting bits to rise to the top.

    The exciting thing, at least from a systems perspective, is that this is self-organizing behavior. It is through the interaction of the components of the system (here, the components are the people) that the interesting bits are being obtained. While it may be difficult to capture human thought and knowledge in its native form, it's not as difficult to capture the important pieces as they are being defined by the social system already.

    Further, the nature of social media, in that it tends to interact in bite-sized, discrete pieces, means that the computer system needs not have much understanding of what it is capturing at all. The knowledge is already distilled into its core component, often with attribution, and the computer system merely need remember it. It can be stored without pre-defined labels and fields.

    The thing which the computer system must crucially provide is a robust search capability. Whether this search capability is enacted after the fact, or whether there is a component of the system which searches as knowledge comes in is immaterial. As long as the system can search through the knowledge is what's important.

    Eventually, this captured knowledge can be used and reused as more people interact with it. Each interaction would in essence refine the knowledge, making it more useful to the computer system and the people in general.

    Tuesday, November 10, 2009

    Git vs. SVN (I know, right, another one?!?!)

    I've been endeavoring to set up a code repository (or even a document repository, if that need should arise) and have been weighing the merits of both Git and SVN.

    At the very heart of the comparisons lies the manner in which Git and SVN operate. SVN is a central repository. When you "checkout" a file in SVN, you get only the most recent version. Should you need to do backwards comparisons you must communicate with the server for this. You get no history, either. SVN relies on the availability of the central repository to operate.

    Git, on the other hand, is fully distributed. There is generally a "blessed repository" from which everyone will start and ultimately commit to, but when you "clone" that repository you get a full copy of it. Backing up a Git repository with many contributors is actually trivial as there are countless copies of that repository floating around.

    Branching

    Another major difference revolves around branching. In Git, branching is a way of life (as is the subsequent merging of branches). You want to develop a new feature? Branch on your local box and work on it there, then merge it back into your local main repository before committing back to the blessed repository.

    This is not so in SVN. Branching is not done as often (nor as easily). Branching must occur in the central repository and is not a way of life. In this area Git outshines SVN.

    Client Tools

    One area where Git does not outshine SVN is in the client tools. SVN has been around forever (in digital terms). There are very elegant clients for SVN (such as TortoiseSVN) which allow for an incredible ease of use when working with repositories. Further, most modern IDEs have SVN repository manipulation as a core capability. There are several options for working with SVN in Eclipse, for instance, one of which is core to Eclipse itself.

    Git, on the other hand, is young. The tools out there are not nearly as elegant nor are they as wide-spread. What's worse, Git is incredibly Linux centered. There are two Windows clients for Git (with the advent of JGit, that will climb to three), all of which require one to work with the command line. Some GUI projects, such as TortoiseGit, are in the works but will not be ready for prime-time for a while. The last issue here is that there is only limited integration with IDEs. With time, these situations will change, but for now it is a major draw-back to adoption by those other than the most determined.

    Ease of Setup

    To the end that I would like to work with both systems I decided to set up both on our Windows Server 2003 server. I chose to use Cygwin and OpenSSH, along with Gitosis (a Perl mod for Git), for Git. I used Shannon Cornish's tutorial to set things up (along with a little help from scie.nti.st on matters Gitosis). This turned out to be a rather easy and relatively painless way to go about things.

    The basic gist is that you install Git when you install Cygwin then install and setup OpenSSH (by far the most difficult part). At this point you can connect to the server using SSH and clone any repository you would like. Installing Gitosis on top of things (recursively using Git, no less, which is so cool in my book) allows you to use public/private key pairs to authenticate users. You can then use Git to clone the control repository of Gitosis and admin the system remotely. Very elegant and one which doesn't require the anticipated user to have to input a password or create an account on the server.

    Setting up SVN was more difficult. The differences, though, are myriad. While the above Git scheme works on SSH the method I chose to use for SVN works over HTTP/HTTPS, which has advantages all of its own. I worked off of several tutorials, but the most significant was this tutorial.

    The real difficult part here is that you have to rely on Apache. It seems a bit overkill to have to install Apache and get it running in order to serve up your repository, but this is the accepted way of doing things. Once you have it running you must still log into the server to create a username/password combo for any user that wants to use the system, and you must also log in to the server in order to administer the repository.

    The Best of Both Worlds

    I have to say that the thing which gets me most excited about Git is the notion of branching it carries with it. I really like the thought of creating a local branch for every new feature. It seems natural to me.

    On the other hand, I don't think that I want to saddle everyone else around me with command line tools and vi if they want to work with our repositories. So, can a compromise be made?

    In fact, it can! Git has the wonderful ability to clone and commit to SVN repositories. The real details are outlined here by Clinton R. Nixon. In this way, I can take the pain of the command line on myself without foisting it on anyone else, but I also get all of the wonderful features Git brings with it.

    Conclusion

    In light of all of this, we will be hosting our repositories using SVN. However, I will be keeping an eye towards the maturity of the Git clients. If they should ever advance to the level where any "power user" can attain them, then we very well might switch.

    Wednesday, October 7, 2009

    A Thought on Environments: Portability

    My friend asked me yesterday what I thought of the Kindle. My response was that I was a fan of actually holding a book, feeling the paper, reading in a full fidelity mode. I spoke of how tired my eyes could become from reading on a screen all day. I made a decent case for not adopting the Kindle.

    Then, on a whim, I checked out the Kindle app for the iPhone and immediately found myself sucked in.

    The first thing that did it was the free availability of a book that we have all been discussing here at the office, Bertrand Russell's "The Problems of Philosophy". Turns out that it is a "classic" and Amazon offers many of the classics for free. I have now downloaded 10 free classics for my iPhone Kindle app and am well on my way to finding book reading Nirvana.

    Free content put the hook in my mouth, but what set it was the concept of WhisperSync. WhisperSync is a service that Amazon offers which will sync your content between devices. Now, this is not just the raw content, this is the detailed content, the state content.

    For instance, let's say that I am on page 50 of "The Problems of Philosophy" on my iPhone. Further, let us say that I have a Kindle at home on which I do the bulk of my reading. As I read on the iPhone, the Kindle app updates the state for that book. When I get home and fire up my (physical) Kindle my place in the book comes right up (in computer terms, the state is restored). No futzing around with finding my place, everything is just magically the same.

    This got me thinking about the synchronicity of environments. In our research the environment you work in is of paramount importance. That environment can be unique to you, or you can share it with others. Everyone can have their own, if necessary. The environment is at least partially reflected in software.

    One thing Dr. Sousa-Poza and I talk about a lot is the ability to "save" environments. Environments should be transportable and shareable. If you need to see what I see then you should be able to load up a copy of my environment, see things just as I see them with the data I've been using. What's more, your environment should be able to subsume my environment! Environments should be nestable yet discrete.

    WhisperSync brings an interesting possibility to my mind. Shouldn't the environmental changes that I enact on one device translate to another device? What if I access my environment from my iPhone and then switch to my laptop or a web browser? Shouldn't the environment be exactly the same?

    This necessitates two things: A place of storage (centralized or decentralized, makes no difference) that all environments have in common and the ability to capture state.

    Ideally, the system would operate like this (from the 10,000 foot view):

    1. I access my environment and make some change to its state.
    2. That change is transmitted to the "server" which keeps track of environment state. ("server" is in quotations as a means to capture an idea. It need not be an actual server. It might be better to think of it as a state oracle)
    3. I switch to a different device.
    4. As I start up my environment on the new device the state is restored from the "server".
    5. I continue my work.
    Such a system would be truly powerful from a portability standpoint. I could do my work wherever I needed to be and have that work mirrored wherever I go. I could work from multiple devices and not lose a thing.

    Monday, August 10, 2009

    An Analyst's Development Environment


    Here in the land of academic research we're working with a "new" take on mashups. It seems like a no-brainer to me but a lot of people have expressed interest and surprise when I explain to them what we're doing. For now let's call it an analyst's development environment (ADE).

    One thing that mashups are really, really good at is taking disparate data sources and allowing "momentary" relationships in the sources to be created. This in effect creates a new data source that is a fusion of the inputs. As is often the case in fusions, this new source tends to be more than just the sum of the parts. You often come up with new views on the data as you add extra sources.

    Most people stop here at the fusion stage. Once they have the new view onto the data they rely on other tools outside the scope of a mashup to do interesting things. They might pipe that data into a tool such as Fusion Charts in order to visualize it or they might pipe it into an analysis tool such as a model or sim. But, why do they need to leave the scope of the mashup to do this? What if that analysis or the creation of the Fusion Charts XML was an automated part of the mashup itself?

    Mashups deal with web services primarily (though there are some nifty products out there that allow you to mash more than just web services). A web service is usually considered to be a data source. But, in practice they are much more than that. Consider all of the specialized web services provided by Google for geolocation or Amazon for looking up aspects of books. The simplest example I can give you is Google's web service which converts an address to a lat and long pair (called geocoding). With these in mind let's take a different look at web services. Let's look at them as processing units.

    A processing unit has 3 criteria: it takes input; does something interesting with that input; and provides output. Processing units are the basis of modern programming. They're known as methods, functions, procedures, etc. depending on context. We can most often build bigger processing units from simpler units.

    Web services fit these 3 criteria handily. You can easily provide input, they can easily do something interesting with that input and then just as easily provide output. All communication is done in a standardized protocol driven environment.

    The interesting thing about web services is that we can string them together (with the right tools) rather easily into processes. That's exactly what we're doing here. Each web service is either a data source or a processing unit. Given the ability to ferry data from one web service to the next (in an easy way) it is possible to create mashups that do more than just mash data. They actually do some form of processing.

    Consider what it would be like if you had a web service endpoint attached to a model? You could pre-mash your data from various sources then run it all through the model and create a new output that would be very interesting. It would be so easy.

    Using Presto we recently put together a demo which worked along these lines. It made our demo come together in several weeks rather than over several months. We used Presto to access databases then ferried that data (in XML format) into a custom built web service that took said data and ran XSL transforms on it. That produced Fusion Charts XML which we then piped into our presentation layer for visualization. It was easy.

    Here is a diagram of what the actual flow of the mashup was.

    Here is a screen shot of the actual chart produced by the generated Fusion Charts XML.


    An ADE would work in a similar way. Using provided tools which allow for ferrying of data from one endpoint to another and given a grab-bag of analysis and transformation web services an analyst could create some amazing things with little effort or technical know-how. The only developer support would be in the creation of any custom web services. It could be a very powerful tool.

    Wednesday, July 22, 2009

    The Walled Garden

    Let me hereby declare that I love my iPhone. It is useful and wonderful and keeps me connected all the time. I have been using it in lieu of my computer at home for quite some time now. I write emails on it, craft witty 140 character tweets on a regular basis, listen to books on iPod and even play extremely enjoyable games. It is a great experience.

    I have, however, begun to chafe under the strictures placed on my iPhone by both AT&T and Apple (often in conjunction with each other).

    My gripes against AT&T are especially aggravating. I pay them enough money as it is for the privilege of using my iPhone, why do I need to pay them even more in order to use my iPhone as a modem? It does not seem fair that I will have to shell out an additional $30/month to do what is freely available on other, older and less capable smart phones.

    What's worse, if AT&T sees an app as competitive to it's business model, it will limit that app, or flat out deny it! Consider Skype: Skype offers free calls over the Internet to other Skype users, yet AT&T will not allow Skype to make calls over its 3G or Edge networks. They pull the undue competition card.

    On the Apple front, a nifty app came to my attention recently that I thought was a truly innovative and awesome use of the iPhone. Given an iPhone 3GS (with its video capabilities, compass and GPS) an "Augmented Reality" app has been developed called TwitARound.

    TwitARound looks at the tweets from Twitter in your area and plots them on a map. The AR part, though, comes when you hold your phone up. The app takes your GPS position and your bearing from the compass and lays the tweet on the screen. So, as you move in a circle with your iPhone in front of your face, you can see the actual locations on your iPhone of the tweets as they would appear if the tweets were layered over real life. It's quite awesome and I would like to see more apps like this.

    However, because TwitARound accesses APIs which Apple has not, but should have, made public, it cannot be published in the iTunes store.

    Apple plays the non-public API card too much. For instance, they did not make their "find my phone" APIs public so that they could charge you a monthly fee through mobileMe. There are already jailbroken apps which can do this, but since they didn't make the APIs public, you won't see legitimate apps show up in the app store.

    Call me naive or non-business-savvy, but all of this seems like bad business to me. As a consumer, I want freedom. It's my device, I should be able to do with it as I choose.

    So, while I love my iPhone, I chafe. Yes, I chafe.

    Update: (on 7/29/09)

    First off, it turns out that Apple will release the video camera APIs with iPhone OS 3.1 (per Ars here). Yay for Apple on this one. It's good to see that some of the "hidden" functionality is being exposed. Now, let's see if they expose the "find my phone" API or if they milk it for more money.

    Secondly, the app denial shenanigans continue. In a story here (also on Ars) it appears that all apps relating to Google Voice are being pulled and any apps which feature Google Voice are being denied. The scuttlebutt is that AT&T is pulling the strings here. Some disagree, but my vote goes towards AT&T.

    The Dark Side of Twitter

    I've seen a very interesting phenomenon going on in the Twitter-verse recently. It has brought to my attention that Twitter (and micro-blogging in general) can be used for reasons that are not above-board. What, pray-tell, is this dark and nefarious phenomenon?

    I keep getting followed by prostitutes.

    The first time it happened I just thought it was some random individual with a sick sense of self. However, the next day, another woman of the same ilk followed me, and the next day another. That's when I started getting curious (not about what the women offered, but about what was really going on).

    Invariably, they all posted a provocative picture of a woman with at least one post which was anywhere from lewd to slightly suggestive. That post would have a link attached. The link takes you to some triple-X "dating" service. Within a couple of days the account is shut down (you get the "Nothing to see here, move along" message when you try to visit the account).

    No doubt, for some reason I am not aware of my twitter user name has been picked up by this "dating" service and they keep following me with fake accounts, all in vain hopes of promoting their "service". It's all at least partly automated, it has to be, and there's probably one person sitting behind a desk creating profiles then running those profiles through some tool they had custom made to follow a few thousand people.

    The practice, though, really brings questions to my mind about what twitter can't be used for. If it can be used for prostitute marketing, why not black-market marketing or subversive political marketing? Why even marketing at all? I once had the privilege of speaking with an individual that detailed how an anarchist group used Twitter to attempt to disrupt the RNC in Colorado.

    Of course, far from being upset by all of this I tend to think of this as rather ingenious. What uses can Twitter serve? What's the most creative use any of you have seen?