Re: Spiderspace

> From: tcmay@got.net (Timothy C. May)

> Sure enough, a search of "Cyberia-l" in Alta Vista showed all sorts of
> hits, including what appeared to be several _private archives_ of parts of
> the traffic. (By "private" I mean in the sense that they were someone's
> personal archives, and not necessarily complete or even semi-officially
> sanctioned.)

There are any number of reasons this migh have shown up - if the private
archives are accessible to the public, for example ... but ...

> Declan McCullagh, on the Cyberia-l list, followed up to my post on this
> topic by noting that things will really get interesting when the internal
> file systems of many sites are made searchable, such as with the Andrew
> File System (AFS) at CMU and elsewhere. Apparently most users make their
> directories accessible to others.

... I was under the impression that the only documents that most web crawlers
will search are documents that are link-accessible.  Are you saying that this
isn't true?  Are you saying that Alta-Vista will search EVERYTHING that's
publicly accessible, whether by anonymous FTP or web?

Ed Carp writes:
 > ... I was under the impression that the only documents that most web crawlers
 > will search are documents that are link-accessible.  Are you saying that this
 > isn't true?  Are you saying that Alta-Vista will search EVERYTHING that's
 > publicly accessible, whether by anonymous FTP or web?

Ah, but if it hits a site that's set up with a top-level directory
which *does* contain an "index" page but whose server *doesn't*
recognize the index page name, then when you hit the site you
(probably) get one of those server-generated indices.  Those things
generally have *everything* in the directory visible (except those
files blocked by the server configuration, usually stuff like emacs
temp files), and so there you go...

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Nobody's going to listen to you if you just | Mike McNally (m5@tivoli.com) |
| stand there and flap your arms like a fish. | Tivoli Systems, Austin TX    |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~