Today's Dumb Question?

I've got what may turn out to be Today's Dumb Question....

What Happens If, instead of using prime numbers or logarithms for the
basis for a public-key crypto system, we instead generated out public key
thus:

1> pick an arbitrary bit stream (large [pseudo?]random number, binary
representation of selected chunk of text or data file, etc).  1024 bits or
more (in 256 bit chunks?)
2> enter a passphrase
3> XOR the bit stream with the binary representation of the passphrase,
cycling the passphrase as necessary.  This makes the 'large' component of
our public key.
4> hash the passphrase to 128 or more (in blocks of 64?) bits.  This makes
the 'small' component of the public key.

5> We then use these components as in normal public-key algorithms.

Conceptually (to me), this would seem to work, and have the advantage of
not being dependent on the factorability of any number; that is, the
numbers could be extended as necessary fairly simply.  It would also seem
to depend on the entry of a passphrase that would be securely 'locked'
inside someone's mind :-)  
Too, it wouldn't seem to be subject to any kind of patents.  Finally, if
the arbitrary bit stream were taken from something like a section of text
in a file, a sequence of bytes in a data file, or even absolute
track/sector reads from a floppy/hard disk, the entire thing could be
rendered useless by the user by simply erasing/wiping a single file or
track/sector.

Would something like this work, or am I missing one of the trees because
of the forest?

Dave Merriman
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
'That's odd.... the computer model didn't do that....'