Usenet Propagation Sucks

Mike Duvos <enoch@zipcon.net> wrote Fri 29 Aug 1997:
>Theoretically, I was under the impression that Usenet consisted of a large
>number of machines, which compared their news spools continuously, with
>each giving the other all articles that were not on both machines. 
> 
>Practically, this seems to work a lot less well than the designers
>envisioned.

In practice, it's really tough to keep a good feed going. We happen
to keep the alt.anon* stuff for a longer period of time. We've started
keeping individual groups longer as requested by readers. This makes
them happy, for the most part. It needs so much disk! We have two incoming
server feeds and three outgoing and people still complain that their
groups don't have enough messages for their liking. Plus the damn thing's
history database has to be rebuilt twice a month, and at night lest the
customers come for blood when they don't get their daily usenet, and it
never seems to go right without someone (me namely) sleeping in the office
keeping it company during the process.

A server would need terabyte upon terabyte to store a good archive
of Usenet for, say, the past year.

I wonder if the designers could forsee Usenet's explosive popularity,
or the taxing load the spammers place.

Mark Hedges

Mark Hedges <hedges@sirius.infonex.com> wrote:

> In practice, it's really tough to keep a good feed going. We happen
> to keep the alt.anon* stuff for a longer period of time.

As do I.  The use of alt.anonymous.messages as an anonymous message pool
has become quite popular lately, to the point where the traffic often
reaches several MB per day.


> We've started keeping individual groups longer as requested by readers.
> This makes them happy, for the most part. It needs so much disk! We have
> two incoming server feeds and three outgoing and people still complain
> that their groups don't have enough messages for their liking.
...
> A server would need terabyte upon terabyte to store a good archive
> of Usenet for, say, the past year.

Actually, I am suprised at how small a segment of the internet populace
actually posts.  Can you imagine what would happen if a hundred million
people posted every day?  You'd need a terabyte for just one day!

One nice thing is that it is highly redundant, most usenet traffic can
be compressed by a 4:1 ratio.  You could probably fit your one-year
archive into a few hundred GB.

Long-term, it would probably be more reasonable to have people just store
and forward the groups that they are interested in, more like mailing
lists.  But I must admit that I read this list via nntp, because it's
just so much easier to have everything in one place, so until things
seriously break, I suspect most people will want to have big nntp servers
with everything on them.