Web services should be both federated and extensible

One of the most important developments of the web 2.0 era is the proliferation of full featured, bidirectional APIs. APIs provide a way to “federate” web services from a single website to a distributed network of 3rd party sites. Another important web 2.0 development is the proliferation of web Apps (e.g. Facebook Apps). Apps provide a way to make websites “extensible.”

The next step in this evolution is to create web services that are both federated (APIs) and extensible (Apps).

In my ideal world, the social graph would not be controlled by a private company. That said, Facebook, to its credit, has aggressively promoted a fairly open API through Facebook Connect. Facebook has also been a leader in promoting Apps. For Facebook, creating extensible, federated services would mean providing a framework for Facebook Connect Apps – apps that extend Facebook functionality but reside on non-Facebook.com websites.

Consider the following scenario. Imagine that in the future a geolocation data/algorithm provider like SimpleGeo takes Facebook Places check-in data and, using algorithms and non-Facebook data, produces new data sets, for example: map directions, venue recommendations, and location-based coupons. The combination of Facebook’s data (social graph and check-ins) and SimpleGeo data/algorithms would create much more advanced feature possibilities than either service acting alone.

With today’s APIs, if, say, Gowalla wanted to integrate Facebook plus SimpleGeo into their app*, they would basically have 3 choices:

1) Embed Facebook widgets in Gowalla. These are simple iframes (effectively separate little websites) that don’t interact with SimpleGeo. Gowalla would just have to sit and wait and hope that Facebook decided to bake in SimpleGeo-like functionality.

2) Pre-import SimpleGeo data. This significantly limits the size and dynamism of the SimpleGeo data sets and doesn’t incorporate SimpleGeo algorithms, thus severely limiting functionality.

3) Host an instance of SimpleGeo’s servers internally. This requires heavy technical integration, undermining the main benefit of APIs.

In a world of extensible APIs (or “API Apps”), Gowalla could instead send Facebook data back to SimpleGeo. The data flow would look something like this:

(Note how there are three parties involved – @peretti calls this a “data threesome”). This configuration is much simpler to integrate – and potentially much more powerful and dynamic – than the other configurations listed above. You could implement this today, but it would create user experience challenges. For example, Gowalla would be sending Facebook data to a 3rd party (step 3), which might (depending on the data sent) require explicit user opt-in. Things become more onerous if SimpleGeo wanted to share its own user data with Gowalla. That would require an additional oAuth to SimpleGeo (authorizing step 4).

Allowing websites to be federated and extensible will open up a whole new wave of innovation. Ideally some spec like oAuth could include the multiple authorizations in a single authorization screen. Facebook could also do this by allowing 3rd parties to be part of the Facebook Connect authorization process. Inasmuch as Facebook’s seems to be trying to embed their social graph as deeply as possible into the core experiences of other websites, allowing extensible APIs would seem to be a smart move.

* I have no connection to any of these companies (Facebook, Gowalla, SimpleGeo) and have no knowledge of their product plans beyond their public websites. I am imagining functionality that Gowalla and SimpleGeo might include someday but for all I know they have no interest in these features – I just picked them somewhat arbitrarily as examples.

http://harryh.org harryh

I don’t fully understand why the data flow you describe couldn’t be done today. You can already write a client of the facebook api that doesn’t live on facebook.com.
http://www.ouvre-boite.com Julien

Chris, I feel we didn’t spend enough time together this week!
PubSubHubbub is basically what you need to have this working well!

And, to be fair, we built a very simple example with Gowalla’s public checkins via PubSubHubbub and pushed to a map http://julien51.github.com/socket-sub/maps
And I believe SimpleGeo is working on PubSubHubbub support too, for data in and data out, so you can “subscribe” to layers!

If you give a look at stuff like Cliqset or OStatus, you’ll see that this isn’t just whishful thinking. I can post pictures to Flickr, that you could “consume” in my OStatus timeline! And if you comment there, I could read your comment on Cliqset!
http://www.ouvre-boite.com Julien

Harry… it could be done, but you would have to build something specific to Gowalla and Facebook and SimpleGeo. If i want to replace Gowalla with Foursquare, I’d have to rewrite some of my code, if I want to replace Facebook with Twitter, it’s the same….

PubSubHubbub is an Open Protocol (nobody actually owns it, like APIs), which means that if these services supported it, I could build something for either without fearing the day where I’ll want to switch and replace any component. Loose Coupling FTW!
tetris

yes, but that would reside as a widget on (in my example) gowalla? How would you then include advanced API’s services from a 3rd party without the challenges I mentioned.
Pingback: Tweets that mention Web services should be both federated and extensible cdixon.org – chris dixon's blog -- Topsy.com
http://www.cdixon.org chris dixon

yes, but that would reside as a widget on (in my example) gowalla? How would you then include advanced API’s services from a 3rd party without the challenges I mentioned.
http://www.cdixon.org chris dixon

hey julien – interesting. i guess it doesn’t surprise me smarter people have thought of this first. I think the important question now is whether important services like Facebook will support it.
http://www.ouvre-boite.com Julien

Agreed, but, like always, change never comes from market leaders (why would they take the risks?). So we need all “smaller” players to adopt these open protocols to push the bigger ones toward it. AOL wasn’t really pushing toward interoperable email networks! Until everyone else was doing so.
http://sheynkman.tumblr.com Kirill Sheynkman

I am no expert on federated identity (so, I encouraged a friend of mine who is to chime in here), but aren’t we talking about the notion of “trust” between the authorization tokens between different services. One description is here:
http://www.soapatterns.org/federated_identity.php
If the three parties above had a “trust” relationship between them, at a pre-agreed level of granularity, they could all share data (read/write) to the extent that their policies and agreements permitted.
http://harryh.org harryh

Let’s dig into the “replace facebook with twitter” example. Facebook & twitter don’t have the same concept of friend relationships as facebook links are always bidirectional and twitter links aren’t. Representing both of their friend graphs with an identical protocol (so some mythical app could swap one for the other with zero code changes) wouldn’t work cleanly.
http://harryh.org harryh

It doesn’t have to be a widget on (in your example) gowalla. I can access facebook’s REST APIs from anywhere I want.

Right? I feel like I might still be misunderstanding you.
http://www.ouvre-boite.com Julien

Unless you leave the relationships themselves up to the services themselves and only “sync” user data.
This has several advantages :
- the social graph is not unique, I’m not following/friending/subscribing to the same people on 4sq, Twitter, LinkedIn, or Facebook
- It doesn’t allow me to share data about my “friends”. If I want to be public, that doesn’t mean that my gf wants to be public too! Allowing the export of my “friends/relationship” is stealing from them.
- Much simpler to express the concept of “persona/user” than relationship with people.
http://technbiz.blogspot.com paramendra

What you are suggesting makes a lot of sense. The question is, why will they not go for it? Maybe you should take some guesses as to why in a future post.
http://twitter.com/sco Scott Raymond

I’m
trizk

I dont think that we will, in the future, see a few big companies working together to share data to provide a federated service, because in some way they all compete with one another and are too big not to want it all. However, I do think we will either find many smaller companies working together via an open API or many smaller companies working with one larger company providing an API. Btw, there is a service that enables the creation of federated location based apps called Presence ( http://presence.co )
http://twitter.com/sco Scott Raymond

Hm, looks like my comment got cut off…

I’m also a little confused about what you’re proposing. We (Gowalla) already use Facebook’s API and SimpleGeo’s API (a little). Maybe you’re talking more about the licensing/privacy issues than the technical hurdles?

Today, we use Facebook as an authentication option, for its social graph data, and to publish check-ins to Facebook Places. It’s possible that in the future we’ll also read check-ins from Places.

At the moment, our only integration with SimpleGeo is to support a “moveable spot” experiment for the Threadless Everywhere Tour. But we’ve also worked with them to get real-time checkin streams during SXSW, and it’s easy to imagine lots of other threesome-style integrations we could do.

Of course, Julian’s also right that open standards make all of this stuff easier. We’ve always tried to provide Gowalla data in standard formats/protocols (like PubSubHubbub, activitystrea.ms, GeoRSS), in addition to our bespoke REST API. Facebook has been headed in the right direction recently (with OAuth 2 and the graph API), but I hope to see even more standard formats from them.
http://how2startup.com/ Roy Rodenstein

I ran into an example of this in building an app that finds your LinkedIn contacts on Twitter, which required asking the user to OAuth into both services which is a bit annoying UI-wise.

I agree there’s increasing openness but we’re very much in the early days and as you say the flexibility and architecture of integrations has a long ways to go.

I think there are some challenges in that it’s already hard to trust 3rd parties with data, e.g. very little is stopping a Chrome extension or an FB Connect app from doing extraneous or nefarious things with your data behind the scenes.

Perhaps users should be able to not just Disconnect/Remove an app’s OAuth permissions, but have an explicit “Ok, now Delete My Data” operation.
http://www.5o9inc.com/ Peter Cranstone

What about Mobile? How does this all work on say an Android browser?
http://www.cdixon.org chris dixon

I think it applies to any web context equally including mobile browsers like
android.
Blaine Cook

Hi Chris,

the missing bit in your conception here is that the relationships are still determined by Facebook. I have a post from a few months ago here: http://blog.romeda.org/2010/05/facebook-is-my-new-boatcar.html that gets into it a little bit.

I won’t try to get too deep into it in a blog comment – there’s plenty of writing on this by many smart people – here’s the most recent of a few blog posts that Evan Prodromou wrote around the Federated Social Web summit that was held in Portland earlier this summer: http://status.net/2010/07/21/federated-social-web-summit-wrapup

Also, I haven’t blogged about this, but it’s maybe worth a peek – here’s a state diagram flow that describes how to use webfinger and PubSubHubbub to allow two users, using two separate sites, to interact. The example is a text update, but any data could be exchanged (pending your call for interoperable APIs):

http://dl.dropbox.com/u/8318014/wf-auth-rev-lookup.png
http://www.cdixon.org chris dixon

good reading suggestions – thanks. i figured there was a lot of other
people thinking about this. consider me another advocate to the cause.
http://earnedmedia.wordpress.com/ Christian Brucculeri

Posts with flowcharts always get extra points in my book.

The ability for these more powerful relationships to flourish will depend on the economic flowchart. How do the dollars flow, who pays whom and how?

I have trouble with the idea that Facebook would give up the transactional “coupon” side of this experience. Why wouldn’t Facebook just serve up a coupon on check-in w/ a Facebook app and keep whole pie?
http://www.cdixon.org chris dixon

Agreed that as the most powerful player in the diagram (and likely in most
diagrams about data on the web in the future) FB will figure out a way to
capture the lions share of the profits. And coupons might be part of that.
I guess the money will flow depending on who brings what to the data
threesome (or foursome etc).
Pingback: Havas Media Lab » Disruption Landscape - 09/07/2010
Pingback: Publicis Falls Behind On Media With Microsoft; News Publishers Using Data – Really;
http://www.victusspiritus.com/ Mark Essel

If we can invent a way to finance distributed (not just federated) development patterns, companies could grow depending on the value they add to user/object streams as opposed to trying to repeatedly copy the network. Funding in almost all cases supports businesses which control data & the network that creates it. For financial reasons shared data & wealth models have trouble taking hold. Maybe in the next few years this will change with micropayments or more fluid data portability and currency exchange (Facebook or Google bucks).

What freaks me out about huge networks is not the amazing technology stacks, it’s that their value is a function of how much human attention they have. We vote for company valuations with our eyes (ads).
http://qwang.net qwang

I’m confused about your definition of “federated”. To me, “federated” means it works the way email does: Each user’s experience may be hosted by a different provider. APIs provide read/write access to data, but ultimately a single entity such as Facebook or Twitter hosts and controls the central experience.
http://www.cdixon.org chris dixon

with fb connect, the “publisher” (yelp, huffpo) is like the email client in
your example. They use FB functionality but can use it for different
experiences.
http://twitter.com/micrypt Seyi Ogunyemi

FluidDB
Justin Meltzer

I’m a little confused as to why Gowalla would have to send Fb data to SimpleGeo in your example. Why couldn’t Gowalla just import data from FB and SimpleGeo simultaneously to create these new data sets? I think you state the problem with this option here: “2) Pre-import SimpleGeo data. This significantly limits the size and dynamism of the SimpleGeo data sets and doesn’t incorporate SimpleGeo algorithms, thus severely limiting functionality.” Why couldn’t a SimpleGeo API permit this kind of in-depth functionality that seems to only be available in an “extensible” sense?
Pingback: Offline/Online Convergence, Mobile Commerce, and Life After Check-ins | The Good NET Guide
Pingback: Offline/Online Convergence, Mobile Commerce, and Life After Check-ins | BJD Productions Blog
Pingback: Offline/Online Convergence, Mobile Commerce, and Life After Check-ins |
Pingback: Offline/Online Convergence, Mobile Commerce, and Life After Check-ins » Tech Reviews
Pingback: Offline/Online Convergence, Mobile Commerce, and Life After Check-ins | High Tech Toys
Pingback: Offline/Online Convergence, Mobile Commerce, and Life After Check-ins | Let's Discuss Technology here!!!
Pingback: Offline/Online Convergence, Mobile Commerce, and Life After Check-ins « I am a Bridge (Hugues Rey Blog)
Anonymous

Chris, I would really like to get your thoughts on how the adoption of federated web services would impact the value of the layers in the “geo stack” from one of your older posts. If the integration of services becomes more prevalent, which players retain an advantageous position? In particular, what do you think about the Facebook integration with Yelp, Foursquare, Gowalla, et al.
http://twitter.com/denizkural Deniz Kural

Hi Chris, If the “apps” are also federated, why can’t the FB authentication happen over the third party? I.e. some sort of identity amplifier function. One issue is user permitting this as you said.

On the other hand, I don’t see how the approach you’re describing takes care of 3rd party responsibility concerns: In which scenario of using three services in any combination, can the user avoid giving permission to one of them?

Presumably, if the user gave permission to Gowalla, it is for everything it does, which might also be sharing your information further!

What prevents this today?
Darrin Clement

This is a good follow-up to your post earlier this year on the geostack concept. I think one of the fundamentals of this concept is the geo side. Location is a bit hyped now, but probably appropriately so. The location itself, then the polygon it’s in (like neighborhood, or other predefined geofence; not so much the radius-based geofence), and proximity to other venues/pois can all be used to tie together the various APIs or apps. See also http://blog.maponics.com/2010/07/16/the-case-for-predefined-geofences/
Pingback: What it Means to be Social: The Limits of Facebook’s Integration Strategy « NewBizViews
Pingback: Offline/Online Convergence, Mobile Commerce, and Life After Check-ins | TechShadez

Chris Dixon