Search engines and https

While trying to submit the cypherpunks.to website to a few search engines,
I noticed that none of them seems to support indexing of https URL's. Is
anybody here aware of a search engine that indexes secure web pages? And
if there is no such search engine, what are the thoughts on using https to
deliberately keep pages out of indices?

Thanks,
-- Lucky Green <shamrock@cypherpunks.to> PGP v5 encrypted email preferred.
   "Tonga? Where the hell is Tonga? They have Cypherpunks there?"

Lucky Green <shamrock@cypherpunks.to> writes:
> While trying to submit the cypherpunks.to website to a few search engines,
> I noticed that none of them seems to support indexing of https URL's. Is
> anybody here aware of a search engine that indexes secure web pages? And
> if there is no such search engine, what are the thoughts on using https to
> deliberately keep pages out of indices?

One thing you could do is to have your server use http (no s) iff the
connection comes from a known search engine.  Reasonably easy to do --
set up an http server, and block all sites, and put in allow
directives for the search engines.

It's not as if you're insisting on client certs for the HTTPS server.

Adam

On Wed, 19 Nov 1997, Adam Back wrote:
> 
> One thing you could do is to have your server use http (no s) iff the
> connection comes from a known search engine.  Reasonably easy to do --
> set up an http server, and block all sites, and put in allow
> directives for the search engines.

Then the search engine would list the wrong URL. The idea is to get
people to use crypto. In  fact, as soon as I find the time, cypherpunks.to
will reject weak crypto browsers and provide those unfortunate enough to
use such a browser with pointers to upgrade options.

 > It's not as if you're insisting on client certs for the HTTPS
server.

Actually, there are pages on the server that do require client certs.
Of course these pages need not be listed  in any search engine. But now
that you mention it, it might be fun to set up an automated enrollment
page and require client certs for everything.

-- Lucky Green <shamrock@cypherpunks.to> PGP v5 encrypted email preferred.
   "Tonga? Where the hell is Tonga? They have Cypherpunks there?"

Lucky Green <shamrock@cypherpunks.to> writes:
> On Wed, 19 Nov 1997, Adam Back wrote:
> > 
> > One thing you could do is to have your server use http (no s) iff the
> > connection comes from a known search engine.  Reasonably easy to do --
> > set up an http server, and block all sites, and put in allow
> > directives for the search engines.
> 
> Then the search engine would list the wrong URL. 

Yes; so make the failure page from your http server say: connect to
https://www.cypherpunks.to.

More elegant, but a bit more difficult, would be to do the right
thing.

That is: 

1. connection on https port & coming from *.altavista.com allow.
2. connection on https port & coming from somewhere else insist on SSL

(Would this work?  Wander if their spider will talk to https port with
http?)

> The idea is to get people to use crypto. In fact, as soon as I find
> the time, cypherpunks.to will reject weak crypto browsers and
> provide those unfortunate enough to use such a browser with pointers
> to upgrade options.

Kewl.  Anyone know where those patches to convert for netscape 40 bit
crippleware to 128 bit crypto are?

Save download time -- a version 4 browser is quite large to download.

Adam

-----BEGIN PGP SIGNED MESSAGE-----

In <199711191459.OAA02975@server.test.net>, on 11/19/97 
   at 02:59 PM, Adam Back <aba@dcs.ex.ac.uk> said:

>Kewl.  Anyone know where those patches to convert for netscape 40 bit
>crippleware to 128 bit crypto are?

>Save download time -- a version 4 browser is quite large to download.

ftp://ftp.replay.com has the patches.

- -- 
- ---------------------------------------------------------------
William H. Geiger III  http://users.invweb.net/~whgiii
Geiger Consulting    Cooking With Warp 4.0

Author of E-Secure - PGP Front End for MR/2 Ice
PGP & MR/2 the only way for secure e-mail.
OS/2 PGP 2.6.3a at: http://users.invweb.net/~whgiii/pgpmr2.html                        
- ---------------------------------------------------------------

-----BEGIN PGP SIGNATURE-----
Version: 2.6.3a
Charset: cp850
Comment: Registered_User_E-Secure_v1.1b1_ES000000

iQCVAwUBNHMeVo9Co1n+aLhhAQH4qQP/ZNTBU6l4MGDblZ19/AN2lqv7MaKcoTKj
GJzhLkGoDBve49sZ0JSIKfMA2q+UL9fIgTjBYdewhvPNDrb1stmUCWHGybfg57XR
u2vGg4AfykWCmYWDAjTACMtJr4jZ8K5hRYbe8Q7/BaimU/zMcO+SMFX7fIGYkavG
eiHKdE0Ebig=
=rvXb
-----END PGP SIGNATURE-----

> One thing you could do is to have your server use http (no s) iff the
> connection comes from a known search engine.

Why not set up http pages with just "description" and "keywords" headers
for the search engines, and a link to the https page for users that
come from the search engine?

Lucky, Infoseek will actually store id/passwords to get into
sites that it really wants to index. I've forwarded your question
to the infoseek list. I'll let you know what I find out about https.

 - Joi


>  > It's not as if you're insisting on client certs for the HTTPS
> server.
> 
> Actually, there are pages on the server that do require client certs.
> Of course these pages need not be listed  in any search engine. But now
> that you mention it, it might be fun to set up an automated enrollment
> page and require client certs for everything.
> 
> -- Lucky Green <shamrock@cypherpunks.to> PGP v5 encrypted email preferred.
>    "Tonga? Where the hell is Tonga? They have Cypherpunks there?"
> 
> 
> 
--
PGP Key: http://pgp.ai.mit.edu:11371/pks/lookup?op=get&search=0x2D9461F1
PGP Fingerprint: 58F3 CA9A EFB8 EB9D DF18  6B16 E48D AF2A 2D94 61F1
Home Page: http://domino.garage.co.jp/jito/joihome.nsf
To subscribe to my personal mailing list send mailto:friends-subscribe@ji.to