How to go about building a Platform - Techniques and Concepts (Part 2 of 3)

16 Nov, 2007

How to go about building a Platform - Techniques and Concepts (Part 2 of 3)

This is part 2 of a 3 part article on Building Platforms vs Building Applications. Refer the below index to review the other two parts. It is recommended that you read them in order - Part 1, Part 2 and Part 3.

Index

PART I - Advantages of building a Platform
PART II - How to go about building a Platform - Techniques and Concepts
PART III - Principles to keep in mind when designing a Platform

This second part will cover techniques that one can use to build a Platform.

So … how does one go about creating a Platform
The first step is to begin thinking of your application as a platform, early on in your development lifecycle, preferably in the very beginning. Applications need to be designed as a platform from inception, since the cost of post-conversion is high and in some cases results in ugly patchwork.

Secondly, you really need to understand the meaning of creating plugin frameworks, providing APIs etc. It always helps to closely investigate various existing applications that can be classified as a platform. For instance applications like facebook, flickr etc. One of my favorite application development company is Atlassian. They have done an amazing job with JIRA and Confluence, their flagship products. Both applications have an extensive array of plugins contributed by a large number of users worldwide - the result of a very mature, easy-to-use, flexible, well thought out extensible platform. I would strongly recommend any application developer to study existing APIs, mashups, plugin frameworks of existing popular applications to broaden their horizons. It helps identify potential opportunities in terms of providing extensibility, and provides examples of tech implementations of creating a extensible application, cuz believe-you-me good platforms are thought of, and built, ground-up as “platforms”.

Lets investigate the opportunities that any given application has in terms of extensibility and customization -

Theming / Skinning

Your application can allow the user to change colour schemes, fonts, layouts and styles. This can be achieved by creating the application in such a manner that the entire user interface is encoded in separate files which form the skin. You can supply a basic set of skins. If your application becomes popular you may even have third party companies developing and distributing skins for your application. With regards to Theming/Skinning, I will broadly divide the need into two categories -

The first - applications which are single-user - and their interfaces are experienced only by that user. Most desktop applications fit into this category. For eg Winamp. These applications do not really require to allow skin customization. The only advantage derived would be decorative. Users may like to decorate the application in their own way. However since the application is for their eyes only, it may not make much of a difference
The second - applications that involve publishing - where a user can use the application to publish and share content with other users. For eg Blogs, Social Networks etc. Here individualism does become important. People want their blogs to look different from their friend’s blogs. Users want the ability to customize the look and feel of their published content. For eg, I may want my facebook wall to look different from other facebook walls. Publishing is kind of like inviting people to your home. If all homes looked the same one would not be excited about showing them off. Each house reflects the personality and tastes of each home owner. For any application that provides a publishing opportunity it would likely make sense to offer some sort of skinning ability

In order to provide skinning ability, in a clean manner, the user interface of your application must be extracted out into separate files which can be plugged back into the application. The files should essentially define the layout of the data / components, style, fonts, colors, style-associated images. For instance, a clean separation of style using CSS / images for any HTML based application would achieve this. Users can be provided the ability to upload and apply an entirely new theme, or modify components of the existing theme.

If the separation is clean, one can encourage third party companies to build templates and themes, and even theme modification tools for your application. For instance one can expose an API or an upload interface which enables 3rd-party vendors to download the current active themes of a user, and provide an easy interface to manipulate them and re-upload them, or create their own.

Customizing / Moving Widgets

Most applications consist of several independent widgets and controls. Applications may allow a user to customize the position, size, prominence, accessibility etc of these widgets. Typically this functionality is only provided for the main dashboard of the application, where a user maybe allowed to add or remove widgets, move them around etc. This type of functionality is useful since different users have different usage patterns in any application.

If a user is allowed to reorganize his/her application dashboard they can move the widgets that they commonly use to areas of the screen that are easily accessible to them. This is sort of like how Windows users manipulate their desktop, taskbar position, systray position etc. Or how facebook allows users to close or move widgets in the facebook profile page of a user.

In order to achieve this it is important to design the UI in a manner where each widget is cleanly separated from the others in its UI definition. Each widget should be a wholly contained object and cleanly defined with no dependencies on other widgets on the page

Content Embedding

Content Embedding refers to embedding content and widgets of a source application in other applications. Generally there are two ways in which an application can show data / information from another application. One is via API calls where a caller application requests data from the callee application using an API call, and the actual formatting and display of the same is handled by the caller application. Content Embedding however provides an easy way to integrate data from one application into another without any complex programming, API integration or UI creation. Generally both the applications must use the same display markup language (in most cases HTML). Content embedding can be done using protocols such as RSS / Atom, or using IFRAMES, javascript blocks etc. Examples of Content Embedding are - youtube allowing you to embed an uploaded video on your own blog, website etc. This is achieved by using a simple HTML code block which can be referenced anywhere on a web page. The advantage to youtube in this case is that it promotes their brand and promotes awareness.

Another example could be a chat widget embedded in a page which shows the presence of one or more users as online or offline OR a facebook widget which shows the mini-feed of a user on his blog. It is important to make it easy for people to integrate your widgets into their websites. The easiest way is by providing clean html code blocks that can be easily inserted into their website. Note however that some blogs and content editing applications may not support insertion of html very easily.
If your application has a large number of widgets it is also important to organize them and enable the user to be able to easily find them. They should be sprinkled at the right places. For instance, in youtube, when I am watching a video, there is a link below that says “embed video”. This is the perfect place I would expect it, and also the perfect place to promote this content embedding widget.

Another interesting way to gain embed-share is to get application developers to reverse integrate your content linking and embedding widgets into their applications. For instance - a wordpress plugin, which adds a link in the “write post” section that says “Click to embed Youtube video”. On clicking it the user can simply enter the url of the video they want to embed. Wordpress can make a call to that URL passing it a special parameter which provides it the required embedding code. Content embedding can also be achieved through RSS / Atom / XML / REST / custom feed protocols. These approaches can be classified as “in-between API and direct content embedding”. While in the true sense they represent an API since the resulting data needs to be parsed and formatted by the caller application, however the returned data is not in the form of objects or standard data types. Also typically simplistic formatting of the received XML can be handled using XSLT or simple javascript and would not require a lot of effort. The call could be an AJAX call and the recvd XML can be displayed in a designated format.

Content Embedding is an important technique for achieving extensibility. It is easy to use and easy to develop. It also has an additional advantage of being able to maintain greater control over the display format of the content. It does have a few disadvantages. Content embedding widgets do not provide as much power and flexibility as a native API would for a container 3rd party application. Also, since part or all of the formatting maybe defined by you (the source application), it represents additional effort on your part as a developer, as opposed to simply exposing a direct API function. Lastly, pre-formatted content blocks when embedded into user applications or websites, may not necessarily gel with the layout and format of the container content. It therefore is important for the widget to provide certain flexibility by parameterizing layout, color, style, font etc. Offcourse this would not be required if the Content embedding is done through an XML / RSS type protocol.

When designing an application for Content Embedding you need to first determine what content do you wish to allow users to embed within other applications / websites. You also need to decide whether the container 3rd party application will be a desktop application / a website / both. If the content that requires to be embedded within another application, requires user authentication, you will need to determine a security model to ensure that the application / website within which the content is being embedded cannot compromise the security of the user in any manner. Lastly you need to determine whether you will provide formatted content blocks, an XML API or both.

UI Mashups

UI Mashups, is another technique for building extensible applications, and involves allowing the combination of multiple web services to create an entirely new service. For instance lets say a chat application returns the current online status with the IP Address of a user. The IP Address can then be fed into a geo-mapping application to figure out the location of the user which can be fed to a map application to show that location as a green dot on a map for an online user. This example uses three separate services to generate a rich mashup showing both the location and presence status of a user.

Mashups can get fairly creative. For instance, another mashup could pull your friends list from your social network / chat application, and create a world clock showing the current time where they are.

Mashups maybe generated using services of your application in combination with other applications. If you want users to be able to create mashups out of data from your application, you need to provide data that developers / users can use to create these types of mashups.

If you want users to create mashups from your own applications you should design your application as a large number of tiny loosely coupled individual apps rather than one monolithic application. For instance take the above example. If you were running a social network, you could hardcode a feature into your application which shows a local time clock for a particular friend. Alternatively you can build multiple tiny services - one which provides a list of online buddies, another which accepts a list of buddies and provides their ip addresses, another which accepts a list of ip addresses and provides a list of countries, and a final one which accepts a list of countries and creates a timeclock interface. How granular you go is entirely a business call on your part.

Mashups are a relatively new phenomenon, but the possibilities are limitless. The output of any service can be piped in as an input to another service, as long as the interchange format is understood, creating a long pipeline of intelligent services that result in useful output. Mashups also divide processing requirements across multiple server nodes, thus allowing greater scaling of the backend web services.

APIs

This is the battle-tested age-old method of making your application extensible. APIs enable developers to directly interact with the core of the application. They are easy to provide on the part of a source application and with open standards such as SOAP, XML RPC, REST etc, developing client applications to an API is quite easy. APIs offer the maximum flexibility in terms of extensibility to a potential developer. APIs can serve different purposes. Provisioning APIs are APIs which enable developers to provision a service on a platform. For instance a mail service may expose an API to facilitate direct creation of email accounts on the mail server.

Data access APIs facilitate access to data, such as profile information, statistics etc. Data modification APIs allow developers to modify data in an application.

Provisioning APIs and Data Modification APIs sometimes require significant processing at the backend, and may be batched. This means that calls to those functions maybe accepted and queued by the server to be processed subsequently. These are termed as asynchronous calls since the actual processing of the call takes place asynchronously. The outcome of the call maybe communicated back to the caller through a callback mechanism OR via an out-of-band method such as email. Even Data access APIs may at times be batched if the result to be returned contains a large amount of data. SOAP / XML-RPC based APIs return XML strings which are directly serializable into objects in most platforms. Custom APIs which return custom XML packets require custom serializers or parsers at the client end. If the API is non-standard, it typically helps to provide multi-platform client side kits, which are essentially wrappers around the API calls in different languages, that serialize the data returned into objects / native types.

Alongwith an API an application must provide the following for ease of development -

Up-to-date documentation of all the methods, their parameters, the validation routines for those parameters, return values, format of return values, as well as all error conditions
A sandbox environment where a developer can try out the API
A web interface from where a developer can make calls and inspect the XML packets sent and received (incase the API is an XML API)
Sample code which shows the implementation of a few of the API functions

Additionally, you should also setup forums or mailing lists where API users can collaborate and assist one another. The forums also serve as a repository of queries and solutions for newer developers.

APIs require careful planning from a backward compatibility standpoint. Once an API is launched, and widely adopted, it becomes very difficult to change its method signature. If there are already hundreds of applications written around the API, one cannot simply modify the interface since it would break these applications and in turn the user experience. Therefore one must plan the interface of the methods that one chooses to expose with utmost care. Always keep in mind what the API needs to look like in the future, and ensure the current architecture takes into account future updates without breaking the interface.

An ideal approach that you can use to create an API, is to ensure that you build your application as a multi-tier application from ground up. Maintain a clear separation between the user interface and the core api. In a sense your user interface becomes a client on top of your own API. A clean multi-tier design can go a long way in terms of building an application with an easy to use API

Event driven Architecture

Another technique for developing extensible applications is to create an event driven architecture. Model your application in the form of events and actions. For instance lets take facebook. Some of the events that take place within facebook are posts on your wall, private message, photo tags etc. Each event can trigger certain actions which in turn can lead to other events. An event driven architecture consists of subroutines which register for specific events. Then when that event occurs, the subroutines which had registered for the event are fired. This type of an architecture makes it easy for external applications to register for, or subscribe to events within your application, and provide unique functionality. For instance a mail server may allow applications to register for events of type “incoming mail”, “disk quota full” and so on. Client applications can register for these events. The code on the client application would be invoked when the event it registered for occurs. The client could display a toaster popup to denote a new incoming mail etc. Typically event driven architectures work using callbacks, wherein a client application registers for an event by providing the URI of a function to be called when the event takes place. The server application triggers a callback to this URI / service when the event occurs. The server may process the callbacks as a batch by queueing them and processing them using a threadpool.

Applications built using an event driven architecture can have hundreds of different events. In this case it maybe useful to classify the events into different types. This way a client can register for a set of events by registering for all events of a particular type.

Another name for an event-driven architecture is - publisher-subscriber model. Event driven architectures are push based models, wherein, a server proactively pushes a notification or message to the URI provided by a client upon an event taking place, instead of a client polling the server regularly to figure out whether an event has occurred. This results in higher efficiency, lower bandwidth utilization, low latency and a better user experience.

Protocol Translation / Protocol Bridging or Multi-Protocol Support

Protocol translation or protocol bridging is a great way to expand your client base and provide multiple ways of accessing your service. Invariably different users and different use case scenarios require different ways of accessing a service or content. For eg take a content management system. The purpose of a content management system is to store different types of douments and enable users to access them. A typical content management system may make the stored documents available through a web interface which users can use through a browser. However an extensible application would allow users to access the same content through various other protocols such as WebDAV, FTP, File share, etc.

Infact a well designed application would provide clear abstraction between the data layer and the access layer. In this manner, different access protocols can be layered on top of the core interface which accesses the data. These access plugins can be developed by anyone, including the user or a third party vendor.

Another example can be a messaging system, for eg an instant messaging system. If designed in a manner that allows third party vendors to create protocol bridges, plugins could be developed that bridge over to other instant messaging protocols or messaging systems such as mail, SMS and so on.

An application typically provides a core service. This core service is then accessed through a standard protocol such as HTTP. If the protocol and the core service are loosely coupled, one can easily layer other protocols in to access the same service.

Outbound API integration or Reverse API Integration

Reverse API integration another technique for providing extensibility through callbacks. Basically Reverse API calls or Outbound API Calls, require you, as an application vendor, to provide an interface definition for writing functions / methods, to which your application can make outbound calls to. In that sense it can be thought of as a subset of an event driven Architecture, but it does deserve its own independent mention.

For eg, lets take a financial transaction management platform, such as a payment gateway. A payment gateway facilitates B2B and B2C transactions between entities. Such a Payment Gateway platform can define an outbound call interface which can be implemented by varous accounting software, where the payment platform can make a call to an accounting system that implements such interface, to record a transaction in the books of accounts as soon as it takes place within the gateway. Since there are so many different accounting systems (eg Quicken, Oracle Financials, Tally, SAP etc) they can all implement the standard interface published by the payment gateway allowing the payment platform to connect to them. Similarily the payment system can also define a separate interface for provisioning software, whereby, when a payment is received by a vendor from a customer, then the payment platform can make a outbound call to the vendors provisioning platform with predefined parameters. The vendor provisioning system can then decide the appropriate next action such as delivering the service, creating the customer account etc. Most platforms prefer using HTTP as a protocol of choice for such outbound calls. This is because it is very easy to implement, works through firewalls, is easy to understand and easy to troubleshoot.

Plugins

A more tightly coupled form of extensibility is plugin support. A plugin is a tightly coupled extension to an application. It runs in the same process space as the application itself. It typically consists of UI widgets and associated services. It accesses the same core data and likely some of the APIs as the primary application. It extends the functionality and interface of the existing application. Some examples are - toolbars for browsers, toolbars for email clients, plugins for confluence and jira, facebook applications, Yahoo widgets, Zimbra Zimlets etc.

Plugins are generally tightly integrated into the primary application. The application usually provides a proprietary plugin development process / guide. The application provides hooks and APIs and file formats and processes that need to be used to create a workable plugin. Usually creating an elaborate plugin is relatively more complex than the other forms of extensibility we have already spoken about. However, the application vendor may make the process very simple (eg Facebook has done an amazing job of creating a point and click process to build your own integrated plugin). A plugin likely has its own UI, which blends in with the UI of the host application. Styles and standards for the UI maybe defined by the host application.

Depending on the application vendor and the functionality / flexibility desired, plugin development maybe as simple as point and click (trivial Facebook plugins) or simple XML (Zimbra zimlets) or involve complex coding (Confluence and JIRA plugins).

Creating a robust plugin framework requires considerably more preparation, thinking and planning than most other forms of extensibility. Typically a plugin development framework will provide APIs, callbacks, abillity to manipulate UI, granular data access, a security model, standards and guidelines for design, proprietary styling and markup and more.
For eg - the Facebook Development platform provides a Facebook API, no callbacks yet (to my knowledge), ability to add UI components and load external pages within the Facebook application, access to users data, a security framework where the user defines access policies of a plugin, a markup language called FBML for formatting pages, guidelines for designing the application etc. Though facebook’s development platform allows UI customization, a majority of the UI is not hosted within facebook, but rather on the plugin developers servers. This is slightly more loosely coupled than a conventional plugin model would be. This type of loose coupling has its advantages and disadvantages for Facebook inasmuch as plugin developers have to invest in infrastructural resources, but at the same time get greater flexibility and control over the application. Another robust example is the APIs, callbacks, event listeners, ability to add UI components, access to the data, etc provided by Atlassian for development of Confluence and JIRA plugins.

This sums up a short list of techniques one can use to think of their application as a Platform. Click on any of the links below to read the other two parts of this 3 part post -

PART I - Advantages of building a Platform »
PART III - Principles to keep in mind when designing a Platform »