Drive:Activated : techCollective, windows

Web app to PC resource integration, anyone?

Sam — Sun, 16 Nov 2008 07:16:00 GMT

So the last blog post got me thinking - what if we could access files/resources on web apps as if they were local files?

This would solve one of my major gripes with using web apps. Sure it's great that I can access and work on whatever it is, anywhere in the world, but when it comes to finding things, it is a nightmare. Was that presentation I did a few months ago in Google Apps, Sliderocket, Zoho or somewhere else? How about that photo from that party years ago, right before your memory went blank? Facebook, flickr, myspace? And what if you wanted to use that photo in a presentation done in PowerPoint/Keynote/some other desktop app? You'd have to download it, name it, resize it etc.

There's an initiative going on trying to get web app to web app integration, the data portability initiative, which is also sorely needed as well. The idea is that you would be able to access any relevant resource, from any web app directly - i.e. I would be able to insert any picture from any web app into my Sliderocket presentation, not just from supported services (like flickr) or uploaded ones.

But web app to PC integration is still important. We still interact with web apps via our desktops, media from devices still generally goes through our desktop before going to the web, and even if you firmly believe in the cloud computing idea, desktop apps aren't disappearing any time soon, and our internet connection is nowhere near stable, cost-effective or widely available enough for us to be online 24/7 anywhere anyway.

Besides, there isn't a browser that exists on any OS that integrates well with the file system of the OS. The download/upload concept was good when our connections were 28.8 kbps, when we could only be bothered downloading/uploading a few files. It has not changed a single bit from those days, and I haven't seen anything in the new specs to suggest that it will.

Try downloading say, 15 photos from a search result on a website. It takes way too many clicks, and even after we have all those files, we've lost the organisation metadata that was associated with it on the website, so we now have to waste more time reorganising them. Sure there are hacks that make this easier, e.g. dynamically generated ZIP files on the server, various browser plugins or even desktop apps to overcome the shortcomings. But that's what they are - hacks. There is no uniformity, no compatibility, and with the flakiness of those solutions, some sites don't even bother. Uploading is even worse.

Old technologies

There have been various attempts to solve this in the past, with technologies like FTP and WebDAV, but the problem is that those technologies are not very well promoted, or even implemented, even when they are well integrated with OSs (XP and Vista has support for both, although it is being phased out I believe, as does OSX and Linux).

The overriding problem here I believe, is the mismatch in models. FTP and WebDAV assume you have a standard hierarchical organisational model for the various resources. In reality however, the resources maybe stored on the server in a database, inside other resources, or maybe a non-hierarchical structure is used. How do you represent a tagging system in a hierarchical manner, without confusing the user with the same resource appearing in multiple places? Those technologies are also not very extensible and can't be made to cater for individual site requirements, e.g. requesting associated metadata when you copy a file from elsewhere into it, or even showing existing metadata in the UI.

The new, partial solution

The latest way to tackle this problem is the OpenSearch standard. The goals have been wound back a lot here - this standard only deals with search only. Nevertheless, Windows 7 has already announced in-built OS support for it. All it basically is, is an URL to an XML file that describes how a search can be carried out, and what formats the data can be returned in (generally RSS or Atom). The added bonus with OpenSearch is that this is a simple API that can be used by web apps also, as well as web browsers (Firefox and IE7 already support it).

So with Windows 7, I can do a search for something and get the results back inside Explorer, and copy the results to somewhere local, or use a result in another app. That's great - I now have a central location to find things from, and also a central location where I can use things from various web services seamlessly. The UI even looks quite decent too. (Unfortunately, the implementation in Windows 7 doesn't seem to be smart enough to integrate with the new libraries feature, or even a general search, so unless you know which service to search, it isn't very useful for finding things in unknown places.)

The issue with OpenSearch however, is that it isn't browsable. That means unless all your items are meticulously and consistently organised, or the web app has a ridiculously powerful search algorithm that can read your true intentions (which purportedly Google is close to having), it isn't very useful. And besides, browsing is half the fun when trying to find stuff.

It also doesn't allow any changes to the items returned to be propagated back to the web app, so uploading anything will still be a pain.

A more complete idea

What we need is some kind of web data access standard that allows CRUD (Create, Read, Update, Delete) operations in OSs, and will operate transparently with existing and new apps, web or desktop. This does not necessarily mean that all web apps should write FUSE or Dokan plugins - that would be an enormous effort, and a task that would be out of reach for most web app developers. It also goes against the web ethos of being able to run the same code anywhere. That said, people have written such plugins, e.g. flickrfs for FUSE.

Taking the lead from the OpenSearch standard, web apps would publish a standard description file, describing how to browse, add, edit, delete, download, as well as the type of items it stores. OSs would then be able to download and install them into the shell, integrating them into the UI.

When a user selects the web app in the file finder (e.g. Explorer in Windows), it would connect to the web app, authenticate if needed, send the request, receive the results (probably in RSS/Atom) and transparently present the results to the user, organised into a hierarchical structure by the web app. The user can then navigate into folders (resulting in further calls to the web app) and view metadata. Via the OpenSearch standard, they can perform searches too. And if the exposed file type happens to be one that the user can manipulate locally or in another web app, the OS can transparently download the file from one web app to a temporary location and pass the path along to the local app, or re-upload it to the other web app, ready for manipulation. When completed, all existing copies can be updated as well.

Maybe with browsing support, it can be integrated into the new Libraries feature in Windows 7 (a way of grouping resources from various sources e.g. different drives, folder etc. into one for the purposes of presentation), making it even easier to browse/search all my content, web or local, from one place.

The difficult part is dealing with the different requirements each web app has when you need to perform changes. Taking adding an item as an example, some web apps may ask for a description and tags, while others may want to know the owner, people within the item etc. Rather than trying to create a standard UI capable of handling all the various characteristics, the OS should prompt the user for these details by integrating a webpage (whose URL is given in the standard description file) into the UI when a file is added to the web app's representation in the OS's UI. The webpage maybe even be different depending on the file type. Once the details are provided, the standard 'in progress' dialog appears (e.g. the one that appears when files are copied locally), and the file(s) are transparently uploaded to the web app via a URL designated by the description file.

A similar situation would occur if a file is to be renamed, or the metadata is to be edited. Same with deletions (if they are allowed).

The advantage of this is no client code is required provided the standard is supported, and this can be reused by web apps for web apps to web apps sharing of resources as well. Because everything is done via standard web protocols, nothing ties the usage to OSs, or even a certain OS - it is inherently cross-platform. Web apps even get the added bonus of a certain amount of branding on a computer's desktop. In fact, for many web apps out there, the APIs to do this would probably already exist - only a small amount of work is needed to tie it all together.

There are, of course, security concerns to be considered, although workable solutions are already out there, the most prominent probably being Facebook and its application platform. Another issue that might arise is that the web app still needs to tie into the standard tree hierarchical structure of a file system in an OS. This may be particularly difficult for those who have strong tagging structures instead, but given the incompatibility between the two models, it would only result in a source of confusion if a different resource navigation model was used for certain things on the OS. Both can be emulated on the other model though.

This kind of integration may not (should not?) supersede manipulation via the web app directly, but it does provide a viable alternative, opening up access to our cloud-based resources. The technical issues here I think have either been solved, or can be without much effort. Whether or not developers will bite is another question, and will likely differ from each one, given it will lower the lock-in effect. The smaller web apps will have much less to lose and a lot to gain, but also unlikely to have a significant effect unless the larger web apps come on board, who may have more to lose than gain, depending on how self-sufficient they are, and how much their strategy depends on the lock-in effect.

UPDATE (23/11/2008): Long's done a flickr search connector for Windows 7 - looks promising, but still stuck with the limitations of OpenSearch.

Why can't hardware makers do software?

Sam — Mon, 25 Feb 2008 12:35:00 GMT

I just had to fix a problem on a computer with Windows Live Messenger unexpectedly closing due to DEP violation, and guess what the problem was - Acer's eDataSecurity Management tool was doing something screwy with it. I gotta give them credit for providing a fix to the problem (problem and solution here), but why do problems like these happen time and time again? And even when the software works, it nearly always seems to be some clunky prototype someone was paid peanuts to do.

It annoys any IT guy to no end the amount of crap desktop/laptop manufacturers preload on their systems. The first thing I do for the machines we do at work is unload all that crap, or do a complete OS restore without those bits. I'd be interested to see how many people actually use the crap that's generally on them. Do they really think they're adding value? Who the hell buys desktops/laptops based on the additional crap it has on it?

That crap is generally unsupported too, written by the new guy in the office with the shit tasks list, and as long as it worked the day he wrote it, it's assumed to be rock-solid and will work anywhere for eternity. Great.

If they want to add value by preloading software, do it well, do it unobtrusively, give us the option if whether or not to use it, and respect the fact that I don't want it scattered everywhere on my system (and also good Windows programming guidelines, i.e. don't write stuff to the Program Files folder!).

Motherboard manufacturers are repeat offenders too. Here's a screen dump of my MSI mobo's temperature monitoring app:

Why do they always look so ridiculous? This isn't the first motherboard I've seen with a monitoring app like this - Gigabyte, Asus, Soltek all have apps that look like this or worse. Is there some kind of competition between mobo manufacturers to try and make the ridiculous-looking monitoring app ever?

Looks aside, I thank my lucky stars every time this thing loads up and doesn't make my computer spontaneously restart after a few minutes. You have no idea how nervous I get when I have to use their utility to do a BIOS update...

Many peripheral makers aren't much better either. Here's one from my Dvico FusionHDTV card.

Don't get me started on how unintuitive it is (reservation? automatic date maker? where's the bloody close button?) - I'm just thankful it works. It took them years to get the software to this working state - the software I got with it originally was unworkable to say the least. In the mean time, I got all sorts of weird 'UI enhancements' from the following versions, but the software still did all sorts of weird things until they picked up their game a bit recently. Yet I still have no confidence in pressing the 'record TV' button, and dread the day I have to reinstall it on another computer.

A friend's Linksys CIT200 Skype phone is the same - when that app is loading, everything in XP freezes until it's done.

The pain isn't on and off though. It's full on. There seems to be some implicit rule that they all have to have as much crap as they can get away with running on your computer on startup. That MSI program above automatically loads on startup after install. Others like Logitech, add innocent things like an update checker (I've lost count of the number of update checkers I've had to disable - I wish Windows had the Linux package system sometimes, even with its annoying quirks) to the startup list. Others add more annoying things like services that are useless unless the desktop app is running, while the more evil ones add apps that sit in the task bar quietly chewing up CPU and RAM until your computer slows to a crawl.

Toshiba's Bluetooth stack is a serious offender here - unlike the Microsoft stack, it requires an app for each BT protocol to be loaded on startup, each of them taking up valuable resources that could be shared, even if my BT is switched off. In fact, while I haven't looked on other laptops lately, Toshiba is a serial offender to startup apps - my laptop had over 10 apps attributable to Toshiba loading on startup when I first got it, some useful, others not so. Regardless - there must be a better way to manage all these functions without all this useless overload.

Why is the software component nearly always an afterthought when it comes to the overall product? Is it because its not physical therefore it doesn't matter? Or are the hardware guys somewhat superior, and the software guys are just minimum-paid dropkicks hired to build software because the company had to? In most cases, the software is as important as the hardware is, so it doesn't matter if you have awesome hardware but your software sucks - bad user experience overall (and unfortunately Windows cops most of the blame due to the nature of the bugs, and hence the hardware manufacturers have no incentive to fix it).

Whatever it is, if I'm ever in the hardware device industry, I pledge to give software the attention it deserves