Semantics and Computing

>>users can almost immediately acclimate themselves in a GUI environment and be productive

absolutely not. this is not a given, and I think it's relevant to explicitly state that here. I can't tell you how many _terrible_ guis I've seen that are just as bad as reading some of the more cryptic man pages.

re your "Move Beyond File Managers" - Agreed, but this must also be used carefully. for example, "find" on the command line is pretty simple, but if you build a complex enough network of pipes etc you can get _exactly_ what you're looking for.

I suppose someone coule build some sort of syntax-network-builder gui, but it seems overkill as compared to writing some better man pages with _examples_ :)

--

how is the emblems idea "new" ? I do that stuff all the time on winblows and mac OS 9. contextual menus, I believe. actions one can perform "in context"

most of my problems with HCI are not related to intelligent technologies, or anything fancy. I lack the following things:

-complete, properly thought out separation of: -apps -configuration -files

i.e. every time I build a new box, I have to spend a week reconfiguring everything, and yes, I can move some .confs over, but there are no good tools for managing all of that data as "valuable user input"

files, for the most part, can be separated properly... but I'm sure we can all think of times when we've lost things because they were for some necessary reason stored outside /home :)

-system management I would prefer not to have to babysit. Logs, etc.

-knowledge management this requires a bit of explaining: I don't need metadata attached to all my files, I know what my files are :) I _do_ need a way to quickly learn about something I have not encountered before. I am of the opinion that documentation is not the only solution. For example, man pages are the bain of my existence: some are clearly written, easy to understand... others are a tangled nightmare, many lack examples, etc. Like the annotated manuals at php.net, I would like to see a central way to build knowledge about these systems which is then incorporated into the system. Sort of a guide. Hell, that could just be a website :)

----

In short, realistically I see the next step in HCI as one of communication: we need to find ways of sharing this copious amount of knowledge. I don't have the time to help with a documentation project, but if I could leave notes for others, I would.

Past that, I expect that we'll just ditch the mouse and all that other hoo and just talk to these machines, and have them talk right back. But that's another discussion, and we should probably give a call to mr. kurzweil :)

_alex

I'm going to respond to the first section of the article and leave the part about meta-data and such for others to discuss. I think that melding the concepts of a literate UI (ala the command line) and an intuitive UI lies in the ability to send discrete messages to your applications. For example, I'm talking about being able to send your MP3 player a "pause" message, or maybe a "load and play" message with a filename argument. Currently most GUI applications are completely unscriptable (although that's changing), which means that everything must be done via the default non-literate interfaces that they offer. Those that are scriptable offer their own special scripting interface that is incompatible with other programs. This would be like if every command-line tool required a different scripting language.. it would make a CLI useless. If instead, all the applications that you used for your everday tasks were scriptable, then you could hook up a nice command-line *style* interpreter to them so that you could have all the literate interface advantages with intuitive graphical applications.

One way to accomplish this sort of extreme scriptability is with reflection, so that any given component's "methods" are made available to anything (say, your particular user interface) that queries it. But that's another article altogether. :)

[T]uring, I think you may be missing Mr. Muldoon's point.

I agree that GUIs are not always the easiest thing to use, but in general, GUIs make (somewhat) clear what options are available whereas a command line gives the user a flashing cursor. Sure we know that ``pine'' means check mail, but that isn't as clear to the uninitiated user as is ``click on the mail icon.'' Either way, I don't think that was the point of the article.

What I do think was the point is, instead of focusing on HCI, focusing on what the computer does for the user. We can improve HCI all we want... talk to the computer, ESP, whatever, but if the computer still just sit's and only does the user's explicit bidding, then there is only so far things can go.

I agree with you that ``next step in HCI as one of communication.'' I would just argue (with thae original poster) that the computer should know what your files are just like you do. How are you going to find out what example to look at for your particular problem if the computer cannot know what your problem is and what sorts of information are what? The idea with a metadata-ritch environment is that it allows you to be more expressive to the computer. Metatada is what would allow you to search for documents that are ``letters to so and so'' or to find pictures of birds or many other things.

If the computer knows what you want in human terms, it can better serve you.

--Ben

turing: While maybe not all guis are intuitive, they are by nature much more likely to be than a command line (as BenFrantzDale points out). Also, I didn't mean to suggest that Nautilus introduced the notion of emblems.....I meant that they introduced them to the UNIX desktop. However, what is slightly more worrisome is that I think you may have misunderstood my point with metadata. It is not so much to help users (although, combined with a better filesystem, it would), but help computers. Computers don't really have much information to go on right now. If they had more knowledge of our files, they could do more complex tasks for us with less effort. That means we can leave more work to the computer. And even if you know what your files are, what happens if you want to open all files that have some weird thing in common (like all your mellow mp3s) that you don't organize your data by? It is a long, annoying process of going through every single one to get what you want. With more metadata, you just tell your mp3 player to play mellow music, and it generates the playlist for you. That seems like a useful thing.

You might like to look at the W3C's recently announced Semantic Web Activity. This plans to use RDF as a knowledge representation.

It' too early to say whether it will get anywhere, although quite a bit of research was done ebfore the announcement, I think..

The primary basis for a system of metadata needs to be at the filesystem level. Hans Reiser has proposed a "Future Vision" on www.reiserfs.org that gets to the heart of a framework for a solution.

From there, metadata can be used for all sorts of purposes. Applications can automatically gain some level of intelligence - they can "know" what files you typically use together, and can make them easily available to you.

Rather than having normal directories, you have virtual folders based on search criteria.

Have a look at DomainOS (from Apollo Inc.) before you say that filesystems have a "new" or "futuristic" idea. Those guys (bought out by HP) had some of the most innovative filesystem design ideas ever.

This article poses a good point about expression when dealing with shells and environments. From what I gathered, the article was trying to establish a much broader perspective as to how shells provide the much needed facility for improving semantics in the GUI world.

Take for example the command line model where a user types in a series of characters in which the shell parses and executes them based on the grammar action rules. That would pretty much work for users who know the grammar, but wouldn't work so well for those who don't. The beauty of this model is very much similar to spoken language. Communication becomes smooth once the grammar and vocabulary is learned. The hard part of it is the steep learning curve which most new users avoid.

In the GUI model, there are also grammar rules except that actions are not really based on text processing, instead, they're based on event handling. In the GUI world, commands are expressed diferently because the processing of a command is triggered by event handlers, not by lexers and parsers. Gestures like mouse, keyboard, I/O and object movements are treated as commands while the object themselves are the parameters. And metadata are then used to refine the gesture, completing the function by getting other values from the metadata. However, not all commands can be expressed using gestures alone which brings up the idea that communication by gestures is not that really much effective either.

What I consider to be a good model is the marriage of them, leading to the ability to command by means of written, gesticulated and probably articulated methods.

Ankh: I hadn't heard of the Semantic Web stuff - thanks for the pointer! I read through a few of the documents on the W3C's site, and they seem to be pushing for the same things that I want. So that is good. RDF seems like a reasonable place to start for metadata issues. I was also intrigued by the notion of a "reasoning engine" that could validate proofs for the correctness of query results. I guess it was implicit in my thoughts, but I hadn't directly seen the need for it. The trouble still remains, however, how does this metadata get created? Where is it stored? What is the means for universally accessing it? If the files that the RDF describes move, what happens? Also, how are metadata types defined so that they can be universally recognized? The things that I read seem to suggest that there shouldn't be a set of standard types, and that there would just be equivalence tables stored in various central locations. This seems like a problem. I see why they make the suggestion, but that seems like they are opening themselves up to incompatible data types, and easily-definable points of vulnerability to the system. It would be interesting if there were some gnutella/napster-like RDF exchange system, but I don't know how practical that would be. It does seem, however, like a more limited scope (ie - defined vocabulary) might meet with more (initial) success. It is also easier to transition from strict rules to more lenient ones than the other way around, as is evidenced with the evolution of HTML. I'm definitely going to have to read more about some of what the W3C is working on though.

file management always strikes me as a non-sequitur. I can arbitrarily and unjustifiably identify 2 broad groups of people who want to use computers: those who are actually interested in knowing how it works so they can "do whatever they want" with it, and those who have a particular task in mind (word process, email, paint, surf) who want to know nothing about how it works, and just want it to get on with doing their task.

imho the former group is more than happy to use commands because commands are faster and more powerful. humans have way more linguistic capacity than we have "pointing and grunting" capacity. if anything we need more complex linguistic interaction. more things like es's higher order shell functions, and zsh's programmable completions.

the latter group, otoh, doesn't care a whit about files. files represent "previous states the computer was in". they store all their files in a single root directory, and never delete, move or rename anything. the fact that files exist, or indeed that operating systems or programs exist, is an entirely boring technical detail they'd rather not be concerned with, much like telling someone all about how their VCR head works when they want to watch a movie.

I do like the idea of metadata associated with files at the filesystem level, and I do like the idea of a revisioning filesystem - I use CVS on a lot of my personal stuff for precisely that purpose.

On revisioning filesystems: Plan9 has a system whereby every day a diff of the filesystem is taken at block level and stored on a WORM drive. The system has a mounted psuedo-filesystem which contains dated subdirectories which contain the whole filesystem as of that date. This seems to me like a fantastic idea, although it relies on an almost infinite amount of archival storage - not for the faint hearted!

On metadata storage: Does the Mac not do something like this? Perhaps someone who knows a bit about Mac's or remembers the original source could help me out here? My recollection of it is something along the lines of it being difficult to provide full Mac filesystem support in Linux because the Mac has `resources' attached to each file. Linux, of course, has no way to represent this data so it is lost if you manipulate files on a Mac partition from within Linux. These `resources' contained things like the icon for the file.

My thoughts: How good any method of attaching metadata to files will be is ultimately going to rely on how much people are willing to extend standards. I'm sure that new filesystems for this thing could be introduced into the OSS kernels, but could POSIX be extended to have a set of functions like fmdopen() (to open the metadata of a file) as well as fopen()? Can we extend FTP and HTTP to transport the metadata as well? And so on...

dirtyrat:MacOS, and to a greater extent, BeOS have notions of built-in metadata support. The hold a series of predefined things, like icon, preferred editor, preferred viewer, MIME type, etc. This is a good beginning, but I think it should be extended to file-to-file associations, and even just arbitrary classification (some things make sense for one MIME type and not another).

The really annoying part is that the best, cleanest solution is where metadata is stored in the file itself, in a well-defined footer to every file format. But that can't really happen, for obvious reasons. So we're stuck with doing it at the filesystem level. But the problem is that if we want to do it at the filesystem level, we also need to do it at the protocol level, as you suggest. Otherwise, metadata can't really be transferred along with the files. These are pretty massive infrastructural changes. But, the advantages seem huge. Searching of pretty much every kind would be vastly better. As a semi-tangential example, imagine using some napster-like system of the future, and searching for songs "LIKE 'Radiohead, black star.'" You would easily be able to expand musical horizons, and something like Napster would more legitimately able to claim that it helps smaller acts. But that is just one small example. But with things like the Semantic Web, and increased metadata, a lot of this becomes a possibility with much less effort. We just need to give computers more knowledge of what our files are, and what we use them for.

Just do not forget about Language and charset of file name and inside data :)

Also you could extend HTTP with special headers, and just put metadata in "file.ext.mtd" file.
It then could be supported by other FSs (if you do it with care ;)

Admit it or not, most UNIX lovers are expert users, part of a rather small elite compared to the vast majority of GUI users out there.

I doubt it somewhat that we open source folks will ever put something together that "normal" users will love. This is probably a task that commercial companies like Apple Inc. do better.

Be honest - would you add a feature, or make it easier to use - to achieve coolness?

Because I have to work on the GUI part of a new commercial product lately, I'am sinking myself deeper into stuff not programmers find attractive, but what user interface experts have to say. Books by graphics designers like Edward Tufte or the book from Mac Pioneer Jef Raskin ("The Humane Interface") are worth reading to look outside, into ordinary reality.

I think that I'm going to disagree with the claim that the ideas are too technical, or only suited for advanced UNIX users. Much of the proposal was reasonably technical, but the idea is that it will hide more of the complicated stuff from the user. No artificial structure needs to be imposed, as it is now. If the user chooses to ignore the features, they can. Also, I would argue that today's (and tomorrow's) computer users are far more sophisticated than they used to be. People are now growing up with computers. They are becoming largely comfortable with them. We may as well take advantage of this. I also strongly disagree with the claim that the Free Software community is unable to improve interfaces. Granted, it is a little harder than just programming, but there is nothing magical about Apple that can't be reproduced in the Free Software world. Of course, we haven't done as good a job as we could so far, but that doesn't mean that we're stuck this way. It would be a bad attitude to just give up.

Straight filesystem support of metadata is too restrictive. One user may categorize a particular file one way, while another user may completely disagree. Sure, metadata is fine for a non-contextual system, but if we truly want to innovate, we need something a little more intelligent.

We need to take the "magic number" Unix philosophy one step further. We need a filter system in place that can analyze a file, determine its type (a la its magic number), then apply a series of filters-- for instance, the "mellow mp3" idea would be a filter that would analyze the mp3s, figure out which ones were mellow, and add them to the play list.

Yes, a method of storing the resulting analysis would be good-- in this respect, storage of meta-data would be a Good Thing. However, meta-data management shouldn't fall on the user. It should be an operating system function. And I'm not sure that meta-data should necessarily follow a file when transferred between systems. Perhaps a core set, such as "Author," or "Artist," and "Title." But that kind of information probably belongs in the file format, anyway.

Peer-to-peer systems like Gnutella should then be able to query other peers based on desired metadata information, as suggested before. But a file transferred via a non-MD compliant app (say, FTP) would still get an intelligent set of meta-data.

Once all that was in place, we'd need a way to "train" the filters to our particular taste, so my mp3 filter would know what *I* meant by "Mellow."

Yeah, I know this all harks back to the dreams of "user agents." But we have the capability to build them now, and the ability to share information between systems. I think a good framework for intelligent filters is feasible. There are programs designed to analyze images to decide if they are pR0n or not, so why can't we design filters to categorize our music for us?

If we're gonna dream, we might as well dream big.

It occurred to me that the meta-data manager should be completely orthogonal to the filesystem. It should *not* be the filesystem's job to manage metadata. The meta-data manager (mdm) should have an exported interface (via both libraries and RPC-- perhaps CORBA), and those interfaces should be standard across MDMs, but the filesystem itself should not implement the MDM. (Not to say that an MDM couldn't be designed for a particular filesystem.) A filesystem may be MDM aware, and cooperate with the MDM (notifying it of file location updates due to rm or mv, for instance), but the filesystem should not dictate policy.

Then, users could customize their MDM based on preference, and not based upon which filesystem they choose.

The first step of any MDM initiative should be the development of these interface standards; any subsequent MDM project could then simply implement the interfaces.

As always, this is all just my opinion.

I agree, just having metadata built into a filesystem does little good. However, I don't think that I really said that ;-) What I am trying to figure out are what the foundational elements need to be, and how they can be implemented realistically. The filesystem, I agree, should not be what manages the metadata. It should be aware of the metadata, and keep track of a file's metadata. (I talked about this in my latest diary entry.....you may be interested in reading it). I also think that "information agents" are a great thing, and I want to see them developed. But before we can have them, we need a way to make them "smart" - and that requires metadata. As I suggest in my diary, a standardized set of fields, broken down by MIME type, could really help organizing information.

As for your contention that metadata should not follow the files, that seems like a waste of resources. I think that there *is* a distinction between metadata that is per-user (which should not travel with the file), and metadata that is general. One example of per-user metadata is file-file association: I listen to a group of mp3s together regularly, but if you get one of them, you're probably not going to listen to the same songs together. However, I would contend that most people can agree what "mellow" means.

As for the extension of the file magic number stuff, that seems like an enormous waste of resources. Not only would your computer have to be able to process every MIME type in existence, but it would have to be able to quantify which categories each file best falls under. Combine that with your idea that we shouldn't pass metadata on with files, CPUs will be at 100% utilization fairly constantly. I don't want to make the user determine the metadata for every file at all. What I do suggest is that the user should be able to add information to the metadata as he/she wishes. I think it should be up to OS designers to come up with clever algorithms to fill in the blanks in a file's metadata based on what other files it is being used with. If some of those methods include your file magic number extension, so be it. But it seems like a hefty requirement...especially if we want this to be a part of all computing - which includes portable devices, toasters, and whatever else gets an internet connection in the coming years. I'm trying to find a minimal set of requirements that everyone can implement, and have room to innovate on top.

I agree, just having metadata built into a filesystem does little good. However, I don't think that I really said that ;-) What I am trying to figure out are what the foundational elements need to be, and how they can be implemented realistically. The filesystem, I agree, should not be what manages the metadata. It should be aware of the metadata, and keep track of a file's metadata. (I talked about this in my latest diary entry.....you may be interested in reading it). I also think that "information agents" are a great thing, and I want to see them developed. But before we can have them, we need a way to make them "smart" - and that requires metadata. As I suggest in my diary, a standardized set of fields, broken down by MIME type, could really help organizing information.

As for your contention that metadata should not follow the files, that seems like a waste of resources. I think that there *is* a distinction between metadata that is per-user (which should not travel with the file), and metadata that is general. One example of per-user metadata is file-file association: I listen to a group of mp3s together regularly, but if you get one of them, you're probably not going to listen to the same songs together. However, I would contend that most people can agree what "mellow" means.

As for the extension of the file magic number stuff, that seems like an enormous waste of resources. Not only would your computer have to be able to process every MIME type in existence, but it would have to be able to quantify which categories each file best falls under. Combine that with your idea that we shouldn't pass metadata on with files, CPUs will be at 100% utilization fairly constantly. I don't want to make the user determine the metadata for every file at all. What I do suggest is that the user should be able to add information to the metadata as he/she wishes. I think it should be up to OS designers to come up with clever algorithms to fill in the blanks in a file's metadata based on what other files it is being used with. If some of those methods include your file magic number extension, so be it. But it seems like a hefty requirement...especially if we want this to be a part of all computing - which includes portable devices, toasters, and whatever else gets an internet connection in the coming years. I'm trying to find a minimal set of requirements that everyone can implement, and have room to innovate on top.

I agree, just having metadata built into a filesystem does little good. However, I don't think that I really said that ;-) What I am trying to figure out are what the foundational elements need to be, and how they can be implemented realistically. The filesystem, I agree, should not be what manages the metadata. It should be aware of the metadata, and keep track of a file's metadata. (I talked about this in my latest diary entry.....you may be interested in reading it). I also think that "information agents" are a great thing, and I want to see them developed. But before we can have them, we need a way to make them "smart" - and that requires metadata. As I suggest in my diary, a standardized set of fields, broken down by MIME type, could really help organizing information.

As for your contention that metadata should not follow the files, that seems like a waste of resources. I think that there *is* a distinction between metadata that is per-user (which should not travel with the file), and metadata that is general. One example of per-user metadata is file-file association: I listen to a group of mp3s together regularly, but if you get one of them, you're probably not going to listen to the same songs together. However, I would contend that most people can agree what "mellow" means.

As for the extension of the file magic number stuff, that seems like an enormous waste of resources. Not only would your computer have to be able to process every MIME type in existence, but it would have to be able to quantify which categories each file best falls under. Combine that with your idea that we shouldn't pass metadata on with files, CPUs will be at 100% utilization fairly constantly. I don't want to make the user determine the metadata for every file at all. What I do suggest is that the user should be able to add information to the metadata as he/she wishes. I think it should be up to OS designers to come up with clever algorithms to fill in the blanks in a file's metadata based on what other files it is being used with. If some of those methods include your file magic number extension, so be it. But it seems like a hefty requirement...especially if we want this to be a part of all computing - which includes portable devices, toasters, and whatever else gets an internet connection in the coming years. I'm trying to find a minimal set of requirements that everyone can implement, and have room to innovate on top.

Ryan,

Although you didn't say the metadata should be tied to the filesystem, other replies here have. I was only questioning the necessity, or even desireability, of that.

I think the MIME system (as generally implemented) is fundamentally broken. Most applications' MIME recognition rely on filename extensions, which is terribly inadequate, and just downright stupid. Filename extensions are useful for humans, but the computer should not need them. The magic number system is not that onerous; do a "file" on any random file in your system, and time the results. /usr/share/misc/magic is updated on a regular basis, so initial detection of filetype is quick, and accurate. Once determined, the metadata manager (MDM) would store this data for that particular file, including other useful MD, such as the file's hash (or MD5 signature), the creators username (which might be different from the owner), or what-have-you. If the filesystem kept the MDM apprised of filesystem activity, the overhead could be kept to a fair minimum.

And I think I can compromise with you on the transference of metadata-- there should be an MD policy manager, as well, to determine what kinds of metadata the user would like to share, and to accept. Because I guarantee you, my definition of mellow music is completely different from yours-- we might agree on Barry Mannilow, but probably not on Gus, or Radiohead, both of which I consider mellow. (Their first albums, too.)

So, to flesh out *my* views on metadata (yeah, I know you're tired of my rambling by now):

There should be a metadata manager separate from, but available to, the filesystem. Filetypes can be determined by any means necessary-- MIME type, or magic number, or... it's up the the particular MDM implementation, as long as the MDM can provide true RFC-2046 MIME media type headers when queried.

The MDM should be able to provide metadata for any given file based upon the file owner's policy. This can also be implementation-specific, I suppose, so some MDM might not provide any policy, while others might have policy overkill. Also, it might only import the metadata as described by the policy manager.

Sorry about my rambling. This project has piqued my interest, so I'm making you all sit through my stream-of-consciousness. I don't mean to sound brusque, though I suppose I do.

I have some available web space-- should I set up some discussion board, or a mailing list, to discuss some of these ideas? Maybe even work up some interface standards and protocols? Or is someone else already setting up something? Email me with your thoughts and opinions.

First, my apologies for triple-posting my last comment....mozilla kept timing out when I clicked "post" and I didn't realize tha it was actually posted. oops.

While I agree with tony that the metadata should be largely independent of the filesystem, the filesystem needs to be aware of the metadata. Otherwise, it is virtually impossible to keep track of the files that you have metadata about. This is the major stumbling block for any project that wants to do something with metadata. To do it cleanly, you absolutely need filesystem support. Practically, this means adding a couple of calls to the API to talk to the metadata manager as file operations take place.

I think that you're giving MIME types a hard time. I do completely agree with you that computers should NOT use the extension of the file to determine what it is, except in cases where mime magic numbers can't do anything to help (for example, a .c file and a .java file, or xml files that use different DTDs). Magic numbers should be used first, and then extensions should be used if magic number isn't giving enough information.

The other important thing that I don't know if you agree with me on, or just don't think is important, is the need for MIME type based metadata. I think we should have some standard fields (like the Dublin Core metadata fields) for */*, and then additional fields where applicable for other supertypes (like audio/* should have "mood" that can have an enumerated field value, such as mellow). I also think that it is important that some fields that can only have a small range of values (like mood) should have enumerated values, so there is no need to "translate" between "mellow" and "relaxing", for instance. Otherwise metadata gets a lot less useful.

I like your thoughts on the ability to have metadata managers that have varying policies and such......I think what we need to work out is what a reasonable base spec is, and then what is mandatory, and what is up to implementors. Once that is determined, we can worry about how we get protocol and filesystem support.

Could you mail tony to be add to mail list?
(I have had only e-mail on you web site).

Pipes and filters, building big things from little things. Encapsulate THAT in a GUI and you're home free. It's a dataflow control language, so build graphical dataflow diagrams... visual pipelines. Take pipes and scripts and encapsulate them into processors you can drop files into, and include a bunch of canned processes and a nice field-oriented report layout.