RATINGS: proposal

Eric: I was about to send this to the cpunx list, but got your message
first. I'll send this to you first, and maybe we can hash out
something better before 'going public' with it... here it is. I had
not thought about the possibility of rating multiple messages in one
"rating message." My scheme doesn't address this, although simple
changes could accomplish it. I'm not sure that I understand why a mail
message could be rated multiple times by the same rator, unless you
mean that one might define "axes of rating", like "content",
"spelling", "novelty", etc.  I think that such a scheme is good, but
is starting to place more load on the rator.  I had hoped that I could
use a slider widget, and have the user generate somewhat reasonable
ratings just by setting the slider to a value between 0 and 100 and
hitting a "rate" button. This would automagically put in motion the
scheme outlined below.

As an MUA implementor, here's my first cut of a proposal for a rating
system that would hopefully meet the goals that eric outlined and be
quickly implementable.

1) Mail to cypherpunks-ratings will be gatewayed back to all members
of the list if it has the following lines in its body [Headers are
ignored...] Lines in brackets are optional:

[whitespace]
Target-Message-Id: <messageId>
Rating: <integer>
[Comment: <newline terminated string>]
[Subtopic: <one word string>]
[Rating-originator: <newline terminated identity string: preferably
PGP public key ident]

[PGP signature]

Body lines which don't have the syntax mentioned above are ignored.

The comment field is intended to allow the person doing the rating to
give info about his rating or the rated message to the rated message
reader, when the message is read. I.e. when I read a message from the
cypherpunk list from joe blow which has been rated by
iansmith@cc.gatech.edu with a comment of "this message is fantastic"
I can get something like:

From: joe blow
Subject: foobar
Rating-Comment: (iansmith@cc.gatech.edu) this message is fantastic! 
Rating: 100

Similarly the subtopic field is to provide an automatic version of
what is now being done by hand with ADMIN: and RATING: in subject
fields or the like.  This would allow a mail user agent to provide
some automatic content help, provided that a displayed mail message
meets enough of its criteria for this (have enough ratings been
received? do 75% of the ratings agree on the subtopic for the mail
message? etc).

The PGP information is intended to facilitate "rating reputations" so
that MUAs could be configured to "trust" ratings from people with good
reputations for rating in ways that meet the user's idea of "goodness."

Now, this strategy doesn't say anything about what the numeric rating
scale means. I favor a 0-100 scale, with 0 being a "don't read this,
its crap" and 100 being "must read." Although ideally this should be a
curve with 50 being very common and 100's and 0's being very uncommon,
it seems unlikely to me that such a system would occur in practice, as
the common case (50) is the MOST unlikely to motivate someone to issue
a rating message.  I'm not sure what to do about this problem.

When the mail messages is echoed out to the ratings mailing list, it
would bear the additional header line: 
X-Mail-Rating: cypherpunks

This would allow the MUA of the recipient to easily recognize the
"ratings mail" messages and not display them to the user.  The MUA
should be building a database or otherwise storing the ratings of mail
messages, so that when it displays a message it can bring up the
appropriate ratings information.  This allows clever user interfaces
to be constructed based on ratings information, especially
reputation-based arragements.  Consider our message from Joe Blow
above and the message display below:

From: joe blow
Subject: foobar

PGP-Signed-Message-Ratings: 7
PGP-Signed-Rating-Average: 67.5
PGP-Signed-Common-Subtopic: DIGICASH
Highest-Rator-Comment: Good discussion of Chaum's methods
Lowest-Rator-Comment:  Stupid rehash of Chaum's approach

Now back the MUA implementation: This database (or other storage)
which I have been discussing is somewhat irritating in its complexity
and storage requirements, but mostly unavoidable because of the
ordering properties of the problem. It is quite likely that one will
receive ratings *before* the mail message itself arrives (in fact,
this may be exactly what you want!) so the MUA will be forced to "sit
on" ratings for messages it hasn't seen yet. 

To make the database/storage problem a bit more managable, MUA's can
delete ratings information when they delete a mail message.  However,
it seems like a good idea to keep this information lying around for
potential searches of an archive of a mailing list. I.e. "Give me
messages which have the rating subtopic of DIGICASH and average
ratings over 50"

Major problems with this scheme:

1) Heavy dependance on Message-Id: field of messages and not all
messages bear one of these. I don't see that is avoidable, since we
must uniquely identify mail messages and the Message-Id is designed to
do this.

2) This scheme rewards people who wait on the mail message ratings to
come in then read the mailing list.  This could be problematic if we
get into a situation where people who should be doing the rating
aren't reading messages (and rating them) because they are waiting on
others to do so.  I'm not really sure how to address this problem;
timeliness cannot be used a factor when considering ratings, because
of the speed differences in different peoples mail transport system
(its unfair to penalize those that have long mail delays or are
vacation).  I think that perhaps some sort of "carrot" should be used
to encourage to rate messages, but I'm not sure what this carrot would
be.


what do ya think?
ian

Forgive my ignorance, but isn't this a lot of overkill?  I mean, one 
could simply set up a filter for subjects/people you don't want to see or 
press the 'D' key.

Or is there a larger picture that I'm still failing to grasp (very probable.)

____        Robert A. Hayden          <=> hayden@krypton.mankato.msus.edu
\  /__          -=-=-=-=-             <=>          -=-=-=-=-
 \/  /   Finger for Geek Code Info    <=> In the United States, they
   \/  Finger for PGP 2.3a Public Key <=> first came for us in Colorado...
-=-=-=-=-=-=-=-
(GEEK CODE 1.0.1)  GAT d- -p+(---) c++(++++) l++ u++ e+/* m++(*)@ s-/++
		       n-(---) h+(*) f+ g+ w++ t++ r++ y+(*)

>I'm not sure that I understand why a mail
>message could be rated multiple times by the same rator, unless you
>mean that one might define "axes of rating", like "content",
>"spelling", "novelty", etc.  

This is exactly the reason.

We've already discussed saliency.  There are a few more criteria I can
think of immediately (including the one we know):

-- salience.  What is the article about?

-- clarity.  In the age of information overload, clarity and brevity
are the soul of politeness.  Consider this.  When you post to
cypherpunks, several hundred people may read your message.  If you can
spend one minute making your words clear, you will save hours in
aggregate for all involved.  But in fact, if it's not clear enough, I
don't want to read it at all, saving even more time in aggregate.

Example of a characteristically low clarity rating: L. Detweiler

-- novelty.  Repeated arguments have as their primary quality that
they are ... repeated.  Do I want the same rehash over and over?  How
many times do I want to hear about hidden trapdoors in DES?  Zero.

Example of a characteristically low novelty rating: Sternlight

These two examples are not hypothetical.

-- fact/query/opinion.  What is the balance between verifiable claims
of fact, question or request for help or information, and mere
assertion?  People who wish to help newbies should be able to do so,
and those who wish to ignore them should be able to do that.

-- readware.  A fellow at Bell Labs is working with 'readware', which
is a computer analog of the smudged edges of a reference book in the
place where it's opened to most.  A simple readware scheme could
deliver the number of lines that were read before the article was
deleted.  This information is pretty easy to collect, and requires
almost no user intervention.

>I think that such a scheme is good, but
>is starting to place more load on the rator.

Each rater need not be required to publish a full rating, nor even
rate each article.  No one is supposed to rate an article, and anybody
should be capable of it.

>[proposes an email-header based syntax for a rating]

>The PGP information is intended to facilitate "rating reputations" so
>that MUAs could be configured to "trust" ratings from people with good
>reputations for rating in ways that meet the user's idea of "goodness."

Certainly the ratings format should allow for digital signatures.  The
identity of the rater is certainly relevant to a decision process.

One of the immediate reasons for this is that one might easily want
one's ratings to be private, and yet participate publically.  Here is
a use for pseudonyms that an ordinary person can understand.  If you
don't want someone to know that you think badly of them, don't tell
them.  But you can tell the world under a pseudonym.  It's like an
anonymous referee.

>[on 0-100 scale]
>the common case (50) is the MOST unlikely to motivate someone to issue
>a rating message.  I'm not sure what to do about this problem.

The Central Limit Theorem comes to the rescue.  It says that if you
add together enough instances of random variables with the same
distribution, you always get a Gaussian distribution (a bell curve).

[ An aside.  This is the secret reason that statistical mechanics
works.  Add up enough atoms, and you _can_ assume a Gaussian.  My
physics professor did not tell me this.  Grr. ]

Get enough raters, and the ratings can be first-approximated to good
accuracy by the mean and variance.  High variance means it's
controversial, sometimes a positive characteristic in its own right.
And if you get a bimodal distribution, so much the more.

>X-Mail-Rating: cypherpunks

Certainly a list identifier for mail handling would be useful, but
that's not part of a rating syntax.

>1) Heavy dependance on Message-Id: field of messages and not all
>messages bear one of these.

You check.  Every single one from toad.com does.  Message-Id is a
required field.  If mail doesn't have it, the mailer is misconfigured.
What most mailers do is that if they don't see a Message-Id, they add
their own; this is what toad.com does.

>2) This scheme rewards people who wait on the mail message ratings to
>come in then read the mailing list.  

That is the idea.  Some people want to read everything, some don't.
Those who read early will tend to get their own words read more often,
and this may be reason enough to rate.  A good reputation for rating
may also translate into a good reputation for writing.

>(its unfair to penalize those that have long mail delays or are
>vacation).  

It's also completely unavoidable.  Live with it.

Eric