The Agile Stack

I spent most of this week at the Mark Logic User Conference. One of their slogans is "Agile Content." It's an appropriate phrase. Mark Logic provides an XML repository with an XQuery interface and blazing fast indexes. The beauty of developing applications with Mark Logic is that you can load it up with any content, with or without a schema, and go. Add new types of content, modify existing structures and keep going. Easy to work with, easy to modify, agile.

This was a constant theme from presenters, people experienced building complex applications with Mark Logic. They raved about the flexibility, about iterating, about developing production applications rapidly. They were raving about a technology that naturally enables an agile development process.

Taking this a step further, Dan McCreary is promoting XRX, XForms - REST - XQuery. Dan explains how XRX removes the data impedance mismatch of classic architectures. Forget about OR mapping, in XRX everything is XML. I believe this type of data alignment creates an agile technology stack. But wait, there is more!

What if you had process, tools, and technology that were naturally supportive of an agile development process? Take one part eXtreme Programming, one part wiki documentation, and one part XRX. Blend into Agile Stack. Would there be noticeable benefits? I haven't seen this full stack in action but I have a hunch...

Fragmented Technology

I've been knee deep in my own world of fragmented technology: web servers, XML, iPhones, widgets, treos... the list just keeps growing. And these are just the technologies I use. If you take all of the core technologies used to develop networked applications and all of the applications and devices used to access *those* applications it gets complex really quickly.

In technology there is an idea related to growth and complexity that I consider somewhat classic: As any given layer of technology becomes overly complex to manage, a new layer will be created to manage that complexity. When the new layer is too complex, another layer emerges. Running a business with spreadsheets, then ERP systems, and then EAI is a great example. As our technology becomes saturated, we invent technology to manage or replace our existing technology.

We are seeing this same type of phenomenon on the consumer side with applications and devices. We have more services that we are trying to connect to with more devices. An individual case is not that complex but taken in the aggregate, across all users, devices, and services, we have a lot of technology that inter-operates barely sort of.

Three trends highlight what is happening. Network based service applications are competing to open their API's, internet connectivity from mobile devices has increased 25% annually over the last three years (PEW), and users are consuming streams of data rather than discrete documents updated daily. More applications, increased mobile connections, more data.

Users have more services that are open to access. They can be integrated with other applications and accessed from specialty UI's. Users are increasingly accessing their online services from mobile devices. PEW also noted that mobile access corresponded to a dramatic increase in overall time online. And there are more frequent informational updates to the services they are checking, creating a vicious 'always on' circle. Sounds complex.

In my own circle of technology I have medium access to services from my different devices. My corporate sponsored Treo can access my Outlook contacts, but not my personal contacts. Likewise with email and calendar. One of my personal email accounts has a great mobile interface, the other doesn't. The only way I can 'see' my wife's calendar is to call her on the phone or email her. SMS and IM isn't managed although increasingly I need it to be. I have no way of sharing or managing my Flickr photos or LinkedIn account from my phone unless I am willing to deal with a mobile browser. even on the iPhone, no thanks.

We have fragmented our technology. We use what we can wherever we can. But so far our technology does not appear to be 'Fragment Ready'. Sort of like 'Intel Inside' but externalized. Let me use my technology wherever I want to on whatever device I want to. Then I can stop wading through all of this technology and just use what I need when I need it... until the next wave of inventions.

Devices for Widgets for Devices

I spent the last two weeks on the Pacific coast of Mexico on a beach nestled alongside the jungle. The only "device" available where I was staying was an XM radio. Aside from the inherent fact that some folks consider radio and music the most important, most necessary, form of communication, I was intrigued by the XM radio itself.

It was about the size of a Palm Treo with a small black and white screen. It also had a remote control with even more buttons. The radio physically plugged into an amplifier. So there I was surfing radio stations on a small computer that was communicating with a satellite. The form factor, another device that fits widget sizing, was not lost on me amidst the relaxing environment. In fact, it seemed more pronounced as it fit seamlessly into that world. I didn't *feel* like I was communicating with satellites. I felt like I was using a radio widget on a "device" that was plugged into a stereo system.

When I got back a few days ago I saw a Widgify post that talks about physical widgets. Ricavision has announced a wireless device with a 3.5 inch screen that supports Windows SideShow (widgets). You can attach it to your refrigerator, take notes, and use widgets! Marketing silliness aside, this device makes a lot of sense. When there are devices that are portable, networked, and have applications that I use good things happen.

• the line between "desktop applications" and "widgets" blurs. Would I like a device that travels with me that works equally well as a touch point to my social networks, works with the latest communication mediums (phone, email, IM), a note taker on the road, a contact list manager, a calendar, remote control for my home media center?

• Would I like to be able to access these same applications and associated information in desktop appropriate versions from any computer using a full monitor and keyboard or other enhanced hardware available only on a desktop?

• Would I like to ignore calling plan minutes and have generic data access, using my IP telephony application of choice? Note to phone companies: drop packets not calls.

• Would I like to extend the physical functions of the device through plug-ins and then upgrade the entire device when the same functionality is built in? (satellite radio, cell, keyboard etc...)

There is an argument that there are some devices that are pretty close, and I agree. The iPhone is a great example, supporting a widget environment and smooth wireless capabilities. But it is the vision of one incredible company, still tethered to a closed cell infrastructure and other proprietary standards. Not to mention the fact that it is not seamlessly tied to my desktop applications.

What if we had wireless data connectivity, enough standards to make widgets platform and device compatible, and well thought out widgets and services that bridged that gap between quick portable device based interactions and longer desktop oriented interactions. Looking at the XM radios, Ricavision's device, and the iPhone I see the puzzle coming together albeit under proprietary silos. Here is one person's vote for open hardware and infrastructure, supportable software standards, and well thought out services.

Widget Standards

Over the last couple of weeks there has been a lot of news and blog discussion regarding widget standards. This was due to a requirements draft document being published on July 5th . After reading the blog posts I was a bit confused because I found references to two different but related documents . I found the source material, took a look, and am reporting back with top level information.

There are actually two documents that interested folks should take a look at.

• The first is the widget 1.0 standard. This is fairly slim document, with a lot of known gaps, and a lot of requests for comments. People should comment. I posted to the mailing list about using namespaces. The list is home to other standards discussions, widgets are not the prime focus, be forewarned. This document is from November 2006.
  
• The second document is more interesting because it is packed with the potential of the standard. It is the widget 1.0 standard requirements document. This is the document that was most recently released on July 5th, 2007. These are the requirements for the specification. While it is not close to complete, it is a robust outline identifying areas that need requirements. Reading through this document will get any widget developer thinking in a deeper way about widgets and widget engines, what they need to support, how they will evolve, as well as their current state. Post if you want to help shape your future!
  

A key take away from these documents is that they are focused on desktop widget platforms. This is a mistake. Desktop and webtop widgets share so much in common. In many cases they share the same technology infrastructure down to the scripting language. While desktop platforms can offer greater access to both local and remote functionality, webtop widgets appear to be a viable, and healthy, subset of desktop widgets. Why not make the convergence official and write a specification that will cover all of the widgets from the start?

Another important point is that the technology stack is headed towards web based technology. This appears to be a natural direction of all widget development. However, it does strike me as odd that a potential specification would reference requirements of other specifications that have not yet resulted in effective standardization. I don't have a great answer but it's not hard to foresee a point in time where a widget developer is creating a single widget with a lot of crufty code that is dedicated to allowing it to run in different 'standard' widget engines with differing 'standard' implementations of JavaScript ECMAScript. This is hard enough with the current browser situation, let alone widget platforms and application platforms. Not the fault of specification authors, but a major productivity problem no less.

Don't get me wrong. I am all for a widget specification and will continue to support it and provide comments. In the long run it will help. And I don't offer any specific answers (define a subset of ECMAScript that must be/is already supported?!?). But I am not holding my breath for write once run anywhere. I've seen that movie before, it didn't end so well... or rather it hasn't ended, it just keeps on playing and playing and playing, never quite delivering an ending. I hope the widget standard will bring the environments closer together, so developers can deliver more, and better, products to people who love to use widgets.

widget standard

Widget Prime(r)

The widget conversation continues. More activity around the 'what' and 'how' of teh modern widget. Just as the widget specification was released several people began posting retrospectives or primers on widgets. There are three posts that I feel are worth reading for anyone tracking this space.

The first two are by Niall Kennedy. Take a look through all of his posts, well worth it. He appears to put a fair amount of time and energy into each entry. They are substantial and filled with specifics. He has one post on widget formats and another on updating the data in widgets. As you might imagine these are on the technical side.

The last one is from R/WW. This post looks at the short history of widgets. It can certainly be argued as to the who/what/where/when of the origination of widgets, but this post starts at a practical place that is highly relevant and appropriate for the current conversation.

Bubbletop Beta and c0ding c4mp

The Bubbletop Widget Platform is about to launch their beta site and open it up to developers according to an invitation they have sent out.

From their email:

"Have you ever felt like your RSS feed aggregator was missing a social component? Want to see what your friends are reading? Then BubbleTop is for you"

Sounds fun. Note that the site still says alpha.

They are also holding several events, including a c0ding c4mp (c00l!), to generate widget developer interest for Bubbletop.

This is the same group that is behind Pikeo. It all looks promising, well thought out, and nicely designed. I am interested to find out to what degree they are offering a full fledged widget environment and how it differs from what is out there already. They did mention a social component...

Clearing the API Air

There has been a lot of talk regarding web site API's recently. Whether it revolves around the hand to hand combat that LinkedIn and Facebook are about to engage in or the ever present beat of the Googleplex with their mapplets API, the API is the cool kid in school. API is a serious kid though: a cornerstone of emerging web technology, driving mash-ups, wadgets, and generally connecting the dots between walled gardens. Lost in this talk though, dare I say hype, is what an API actually is. There are two parts of the prevalent API usage that are actually quite different.

Ye Olde Screen Scraper
The original web application API is the screen scraper. If you saw something on an application page that you wanted you could write a client to access that page, grab the specific piece of information you were after, and then use it. This has been used for everything from short term dynamically generated pages to long term storage in a database to enterprise integrations. There are many reasons *not* to screen scrape such as changing HTML and licensing, but it is a simple, powerful, and time honored way of accessing data and processes.

The REST style API changed everything...well, sort of. What a REST API offers is often the same data and process access capability via an authorized interface. The data access typically goes far beyond what is presented on a page and processes are much more convenient to kickoff. There are concrete methods, data structures, and security. From a high level though not a lot has changed. We have essentially had the ability to pull information from other web applications since the first HTML page was presented. As Yoda, I am certain, once said "An open API an IPO does not make."

Along Came the Plug-in... and Facebook
Now mixed in to this API discussion are another set of API's. Rather than let developers pull data or kick off processes, plug-in API's allow developers to embed their functionality into in existing application... as if their functionality was native to that application. This changes the game. And that is what Facebook has done to much publicity recently, but the plug-in has been around longer than the web page. It is great that they have finally become friends although it is not an easy friendship. API's are challenging to get just right, plug-in API's are even harder. Not to mention standards.

An API that includes a plug-in component enables developers to not only build integrations, but to extend applications in a direction they deem worthy. Take a look at all of the widget platforms. They are essentially dedicated plug-in architectures. Together they have generated thousands of small applications.

So the next time someone says they are going to offer an API, ask them what kind. You'll be able to tell if they are just trying to get one more man on the moon or looking to orbit a distant star. And we should all be wary of any promises 9 months out, that's just long enough to forget.