So the last blog post got me thinking - what if we could access files/resources on web apps as if they were local files?

This would solve one of my major gripes with using web apps. Sure it's great that I can access and work on whatever it is, anywhere in the world, but when it comes to finding things, it is a nightmare. Was that presentation I did a few months ago in Google Apps, Sliderocket, Zoho or somewhere else? How about that photo from that party years ago, right before your memory went blank? Facebook, flickr, myspace? And what if you wanted to use that photo in a presentation done in PowerPoint/Keynote/some other desktop app? You'd have to download it, name it, resize it etc.

There's an initiative going on trying to get web app to web app integration, the data portability initiative, which is also sorely needed as well. The idea is that you would be able to access any relevant resource, from any web app directly - i.e. I would be able to insert any picture from any web app into my Sliderocket presentation, not just from supported services (like flickr) or uploaded ones.

But web app to PC integration is still important. We still interact with web apps via our desktops, media from devices still generally goes through our desktop before going to the web, and even if you firmly believe in the cloud computing idea, desktop apps aren't disappearing any time soon, and our internet connection is nowhere near stable, cost-effective or widely available enough for us to be online 24/7 anywhere anyway.

Besides, there isn't a browser that exists on any OS that integrates well with the file system of the OS. The download/upload concept was good when our connections were 28.8 kbps, when we could only be bothered downloading/uploading a few files. It has not changed a single bit from those days, and I haven't seen anything in the new specs to suggest that it will.

Try downloading say, 15 photos from a search result on a website. It takes way too many clicks, and even after we have all those files, we've lost the organisation metadata that was associated with it on the website, so we now have to waste more time reorganising them. Sure there are hacks that make this easier, e.g. dynamically generated ZIP files on the server, various browser plugins or even desktop apps to overcome the shortcomings. But that's what they are - hacks. There is no uniformity, no compatibility, and with the flakiness of those solutions, some sites don't even bother. Uploading is even worse.

Old technologies

There have been various attempts to solve this in the past, with technologies like FTP and WebDAV, but the problem is that those technologies are not very well promoted, or even implemented, even when they are well integrated with OSs (XP and Vista has support for both, although it is being phased out I believe, as does OSX and Linux).

The overriding problem here I believe, is the mismatch in models. FTP and WebDAV assume you have a standard hierarchical organisational model for the various resources. In reality however, the resources maybe stored on the server in a database, inside other resources, or maybe a non-hierarchical structure is used. How do you represent a tagging system in a hierarchical manner, without confusing the user with the same resource appearing in multiple places? Those technologies are also not very extensible and can't be made to cater for individual site requirements, e.g. requesting associated metadata when you copy a file from elsewhere into it, or even showing existing metadata in the UI.

The new, partial solution

The latest way to tackle this problem is the OpenSearch standard. The goals have been wound back a lot here - this standard only deals with search only. Nevertheless, Windows 7 has already announced in-built OS support for it. All it basically is, is an URL to an XML file that describes how a search can be carried out, and what formats the data can be returned in (generally RSS or Atom). The added bonus with OpenSearch is that this is a simple API that can be used by web apps also, as well as web browsers (Firefox and IE7 already support it).

So with Windows 7, I can do a search for something and get the results back inside Explorer, and copy the results to somewhere local, or use a result in another app. That's great - I now have a central location to find things from, and also a central location where I can use things from various web services seamlessly. The UI even looks quite decent too. (Unfortunately, the implementation in Windows 7 doesn't seem to be smart enough to integrate with the new libraries feature, or even a general search, so unless you know which service to search, it isn't very useful for finding things in unknown places.)

The issue with OpenSearch however, is that it isn't browsable. That means unless all your items are meticulously and consistently organised, or the web app has a ridiculously powerful search algorithm that can read your true intentions (which purportedly Google is close to having), it isn't very useful. And besides, browsing is half the fun when trying to find stuff.

It also doesn't allow any changes to the items returned to be propagated back to the web app, so uploading anything will still be a pain.

A more complete idea

What we need is some kind of web data access standard that allows CRUD (Create, Read, Update, Delete) operations in OSs, and will operate transparently with existing and new apps, web or desktop. This does not necessarily mean that all web apps should write FUSE or Dokan plugins - that would be an enormous effort, and a task that would be out of reach for most web app developers. It also goes against the web ethos of being able to run the same code anywhere. That said, people have written such plugins, e.g. flickrfs for FUSE.

Taking the lead from the OpenSearch standard, web apps would publish a standard description file, describing how to browse, add, edit, delete, download, as well as the type of items it stores. OSs would then be able to download and install them into the shell, integrating them into the UI. 

When a user selects the web app in the file finder (e.g. Explorer in Windows), it would connect to the web app, authenticate if needed, send the request, receive the results (probably in RSS/Atom) and transparently present the results to the user, organised into a hierarchical structure by the web app. The user can then navigate into folders (resulting in further calls to the web app) and view metadata. Via the OpenSearch standard, they can perform searches too. And if the exposed file type happens to be one that the user can manipulate locally or in another web app, the OS can transparently download the file from one web app to a temporary location and pass the path along to the local app, or re-upload it to the other web app, ready for manipulation. When completed, all existing copies can be updated as well.

Maybe with browsing support, it can be integrated into the new Libraries feature in Windows 7 (a way of grouping resources from various sources e.g. different drives, folder etc. into one for the purposes of presentation), making it even easier to browse/search all my content, web or local, from one place.

The difficult part is dealing with the different requirements each web app has when you need to perform changes. Taking adding an item as an example, some web apps may ask for a description and tags, while others may want to know the owner, people within the item etc. Rather than trying to create a standard UI capable of handling all the various characteristics, the OS should prompt the user for these details by integrating a webpage (whose URL is given in the standard description file) into the UI when a file is added to the web app's representation in the OS's UI. The webpage maybe even be different depending on the file type. Once the details are provided, the standard 'in progress' dialog appears (e.g. the one that appears when files are copied locally), and the file(s) are transparently uploaded to the web app via a URL designated by the description file.

A similar situation would occur if a file is to be renamed, or the metadata is to be edited. Same with deletions (if they are allowed).

The advantage of this is no client code is required provided the standard is supported, and this can be reused by web apps for web apps to web apps sharing of resources as well. Because everything is done via standard web protocols, nothing ties the usage to OSs, or even a certain OS - it is inherently cross-platform. Web apps even get the added bonus of a certain amount of branding on a computer's desktop. In fact, for many web apps out there, the APIs to do this would probably already exist - only a small amount of work is needed to tie it all together.

There are, of course, security concerns to be considered, although workable solutions are already out there, the most prominent probably being Facebook and its application platform. Another issue that might arise is that the web app still needs to tie into the standard tree hierarchical structure of a file system in an OS. This may be particularly difficult for those who have strong tagging structures instead, but given the incompatibility between the two models, it would only result in a source of confusion if a different resource navigation model was used for certain things on the OS. Both can be emulated on the other model though.

This kind of integration may not (should not?) supersede manipulation via the web app directly, but it does provide a viable alternative, opening up access to our cloud-based resources. The technical issues here I think have either been solved, or can be without much effort. Whether or not developers will bite is another question, and will likely differ from each one, given it will lower the lock-in effect. The smaller web apps will have much less to lose and a lot to gain, but also unlikely to have a significant effect unless the larger web apps come on board, who may have more to lose than gain, depending on how self-sufficient they are, and how much their strategy depends on the lock-in effect.

UPDATE (23/11/2008): Long's done a flickr search connector for Windows 7 - looks promising, but still stuck with the limitations of OpenSearch.

This has been bugging me for a while, its all over the tech side of the web, and everyone seems to have an opinion about it, so here's mine. It's nerdy stuff, so stay away if you're not that way inclined

For those who don't know the what the whole ODF/OOXML fiasco is, here's a short rundown (google it for more info). The OpenOffice guys decided a while ago to create a set of new XML-based file formats for office documents, collectively called Open Document Format (ODF), and to standardize it so the data in the files is not locked into OpenOffice, i.e. other people could make programs that read those files easily.

Microsoft had a similar idea at a similar time, and started doing the same, collectively called Office Open XML (OOXML), mainly because governments were demanding it. While Microsoft were busy trying to get Microsoft Office 2007 out the door, the more nimble OpenOffice guys got ODF out the door and passed the major standard bodies first. So when Microsoft Office 2007 was released, they faced a challenge when trying to standardize it because there was already a standard.

Microsoft of course did not back down (and still hasn't even after they lost the initial ISO vote), continued forward, and insisted that the two standardized document formats can co-exist. The ODF guys got really pissed off, punches were thrown, the FUD machines on both sides went into overdrive, bribes were apparently made, voting was apparently tampered with, and personal death threats were even issued.

The ugly side of the technology world came to light, and personally I think its quite disgusting. It's one thing to spread FUD and offer bribes (and that's bad enough), but death threats are a completely different level altogether, one that should never have been reached. It's amazing how riled up people can get over file formats - I sometimes wonder what they people are like in real life; I'd like to think they're all creeps and look like a murderer's mugshot so at least its somewhat understandable, but I would not be surprised if they were your typical family guy, with a wife and 2 kids. Anyway, I digress.

Here's what I think. Microsoft cannot adopt ODF as a first-class citizen in Office even if they wanted to because ODF doesn't cater for everything Microsoft Office can do. And the ODF guys will never accept all the submissions to make it cater for all of Office's features (and backwards compatibility), because that completely undermines the community direction behind ODF, and because some Office features won't fit in to the rationale of ODF, instead bloating it. That's all completely reasonable, after all the reasoning behind ODF wasn't to standardize Office's file format, but to create a new, practical and usable file format for documents.

Even if Microsoft could adopt ODF, they wouldn't because it makes little commercial sense - they'd have to reveal any new features that require changes to the file format before their product release, meaning their competitors could copy it (at least the idea, if not the implementation). Remember, their entire product relies on this file format, unlike the something like an image file format which only form part of the product does. The exception of course is if they time the file format spec release to coincide with the product release, but I doubt that's reasonable for a standard.

So standardizing OOXML is really just paying lip-service to the idea of a standard document format. What should've happened is this:

• Microsoft should've released the spec to OOXML using a licence that permits reimplementation for all publically. Don't bother standardizing it because that's pointless and everyone can see that.
• They should then implement support for ODF as much as possible, allowing users to use it as the primary file format in Office if they wish (make an 'ODF mode'). For features that can't be implemented in ODF right now, disable them in the UI - after all, similar functionality exists when you default to an older Office file format anyway. Submit new features to ODF, and implement them when ODF supports them.

This way those who require a standardized file format (ODF) can use it, and those who don't can use OOXML and get all the features. Other programs can implement OOXML support using the Microsoft-provided spec if they want to (as many already do). Most people couldn't care less if their document file format was standardized anyway, so they can stick with OOXML and use all of Office 2007's features. OOXML may even become the de facto standard.

Yes I know this somewhat undermines the idea of having a standard file format for documents, but it is a compromise that will work to benefit of both parties. Having a standard file format for documents and competing office suites using the same format is an idealistic idea that is unlikely to work in reality. Although it is very beneficial to the consumer, it in many ways removes the ability for a commercial company to be rewarded for its innovation, especially in this case given a document file format dictates essentially what an office suite app can do; the only difference will be how each competitor does it. Therefore the industry stagnates, profits fall, companies leave, and consumers ultimately lose out.