NSA spy machine

Tim posted a few days ago:

I was intrigued by this disappearance, so I sent an e-mail 
message to  Gunter Ahrendt. Through the joys of time zones, my 
message this evening was  answered within minutes, from 
Australia.

He told me the NSA machine remains, though it has been renamed, 
has been  put under another site, and its performance rating 
has been recalculated  based on a new metric. Gunter's latest 
report (in comp.sys.super) explains  the new metric.

Grepping for the name "SMPP," here's where I found it:

58) 16.46 - (APR-1994) [SRC]
        Supercomputing Research 
Center,Bowie,Maryland,US,root@super.org
        1) Cray 3/4-128 [-4Q96]           11.46?
        2) SRC Terasys                  ~  5
        3) SRC SMPP-4/2M [+4Q96]         503.33?

This is also very intriguing. The machine formerly called the 
"NSA  SMPP-2/2M" and expected to be located at NSA Central 
Security Service, is  now to be located in nearby Bowie at the 
Supercomputing Research Center.

End Tim ----------------------------


The NY Times says (in a business report):

"The new Cray computer will be a hybrid design called the Cray 
3/Super Scalable System.  It will link two supercomputer 
processors with an array of chips containing half a million 
inexpensive processors that were designed by a Government 
laboratory connected with the NSA.

* * *

The Cray 3 supercomputer, two years late to market when it 
appeared last year, has not yet found a customer, and Cray 
executives said they were pinning their hopes for survival on 
the Cray 4, due to be completed in the first quarter of next 
year."

End Times ----------------------------


Does Gunter's "Cray 3/4" = hybrid design as Tim suggests today?

How do the numbers compare to Peter's?

On another point, then, does today's contract report merely 
tell an out-of-date story, and if so, why?

A way to keep Cray afloat?  If so, why not Thinking Machines?

Mr. Cray has been a loyal NSA supplier for many years, perhaps 
this is for his well-earned retirement.  Okay by me.  Maybe 
then he can afford to share all his supercomp secrets.


John

On Aug 18,  7:41pm, John Young wrote:
> The Cray 3 supercomputer, two years late to market when it
> appeared last year, has not yet found a customer, and Cray
> executives said they were pinning their hopes for survival on
> the Cray 4, due to be completed in the first quarter of next
> year."

Don't confuse Cray Computer Corporation (CCC) with Cray Research
Incorporated (CRI).  The former was formed from the later, with a cash grant of
several hundred million dollars and Seymour Cray as head designer.  Prior
to this, the Cray vector range had split into two different streams.
One was the series which went from the Cray I through the X-MP into the
Y-MP series.  They were essentially variations on the same architecture, and
stressed compatibility with previous models.  This range is still aggressively
supported by CRI, which is doing quite well for an exclusively supercomputing
vendor.  They're even learning that the entire world doesn't have multibillion
dollar budgets (hence the EL, EL92 and Jedi models).

The second range began at the Cray I as well, and then went to the Cray II
(designed by SC, still part of CRI at the time).  Then came the split,
and Seymour headed off into CCC, taking his GaAs Cray III project with him,
and CRI stayed with the highly successful [XY]-MP line.

>From what I know of the Cray III, it is a flourinert cooled system about the
size of a small filing cabinet.  It's CPU is manufactured from GaAs, although
the main memory is still silicon.  Because of the signal propogation timings
involved in running with a 2nS clock, they've ground the wafers down to
0.125 mm thick to pack more of them into the same space.  There is a very
interesting ACM talk given by Seymour Cray which details all of this, and it
is widely available on video.  This is not meant with any disrespect to him,
but I was surprised to find that he is a very entertaining speaker.

> A way to keep Cray afloat?  If so, why not Thinking Machines?

Why Thinking Machines over CCC, or even CRI for that matter?  After all
CRI have the rather interesting T3D system.

> Okay by me.  Maybe
> then he can afford to share all his supercomp secrets.

Secret: take lots and lots and lots of money, use the most exotic packaging
technologies you can find, pay lots and lots of attention to your memory
system and cache, don't forget the importance of a nicely balanced
architecture (meaning that I/O does matter), don't forget the importance of
good compilers, and implement bit counting instructions just like the NSA
tells you to.

Hardly a secret, don't you think?

							Ian.

#include <std.disclaimer>  I am not speaking for SGI, folks.

Ian F. writes:

> Secret: take lots and lots and lots of money, use the most exotic packaging
> technologies you can find, pay lots and lots of attention to your memory
> system and cache, don't forget the importance of a nicely balanced
> architecture (meaning that I/O does matter), don't forget the importance of
> good compilers, and implement bit counting instructions just like the NSA
> tells you to.
> 
> Hardly a secret, don't you think?

With no disrepect meant to Ian (indeed, this is my second reply of the
day to him), I think his point is dead wrong.

The "secret" to general success in this market is not "lots and lots
of money," at least not when "lots and lots" is the tens of millions
of dollars that Cray Computer will apparently being getting from NSA
and the Supercomputer folks in Bowie for the delivery of an ostensible
Cray 4 or whatever it ends up being called (SMPP, etc.).

$10 million is pocket change. Anyone building a company on that chump
change is already preparing Chapter 11 papers.

Here's what "lots and lots of money" *really* is:

- $1 billion to complete a wafer fab in Ireland, finished last February

- $1.3 billion to build a wafer fab in Albuquerque, to be finished
later this year

(said to be the most expensive privately funded building in the world)

- $1.3 billion to build essentially a duplicate of the above
facilities, in Chandler, Arizona...construction to start this year

- $2 billion to build yet another wafer fab, in Hillsboro,
Oregon..construction to start in 1995

Intel is already the world's largest chip comany (in _all_ chips, not
just one particular type). If this series of expansions works out (and
the Ireland plant is churning out Pentiums on 200 mm wafers with very
high yields), then Intel will be nearly twice the size of its nearest
competitor. 

Intel Corporation, my employer from 1974 to 1986, may not have the
most elegant architecture in the world, but its microprocessor
fabrication facilities are clearly the best in the world. The
economies of scale are amazing to comtemplate. (And I was near the
group in Oregon that tried "elegance"...the iAPX 432 object-oriented
processor. I only hope the new Intel-H-P alliance on VLIW is not
similarly stillborn.)

(And a new generation of hackers are using Linux on cheap Pentium
boxes to easily outrun Suns.)

Is a massively parallel system of Pentiums or 200 SPECInt P6s or 400
SPECInt P7s the "best" way to go? Given the economies of scale, the
familiarity many people just like you will have with the Pentium, it
probably is.

I'm a fan of the Mac, and may soon be buying a PowerMac, but the
PowerPC does not seem to have the same economies of scale. At least,
Motorola is not expanding rapidly enough to keep up. 

(A hot rumor, to take with some skepticism: a friend of mine told me
tonight that the rumor going around MIPS is that Motorola plans
nothing beyond the 603, that they are fed up with the politics of the
Somerset group (IBM, Motorola, Apple), and that they just don't have
$5 billion laying around to remain competitive with Intel. The rumor
is that they plan to concentrate on telecom, cellular, Iridium, etc.,
and not fight Intel head-on with a come-from-behind architecture.)


So you see why I consider the "lots and lots of money" flowing into
Cray Computer to be spitting into the ocean. I'm not worried.


--Tim May


-- 
..........................................................................
Timothy C. May         | Crypto Anarchy: encryption, digital money,  
tcmay@netcom.com       | anonymous networks, digital pseudonyms, zero
408-688-5409           | knowledge, reputations, information markets, 
W.A.S.T.E.: Aptos, CA  | black markets, collapse of governments.
Higher Power: 2^859433 | Public Key: PGP and MailSafe available.
"National borders are just speed bumps on the information superhighway."

> (And a new generation of hackers are using Linux on cheap Pentium
> boxes to easily outrun Suns.)

You think a Pentium/60 is fast, try running Linux on a DX4/100 - now
*that's* FAST!  Beats the hell out of a Sparc 2... :)

On Aug 18, 11:37pm, Timothy C. May wrote:
> > Secret: take lots and lots and lots of money, use the most exotic packaging
> > technologies you can find, pay lots and lots of attention to your memory
> > system and cache, don't forget the importance of a nicely balanced
> > architecture (meaning that I/O does matter), don't forget the importance of
> > good compilers, and implement bit counting instructions just like the NSA
> > tells you to.

> > Hardly a secret, don't you think?

> With no disrepect meant to Ian (indeed, this is my second reply of the
> day to him), I think his point is dead wrong.

No, it's not.  See below.

> The "secret" to general success in this market is not "lots and lots
> of money," at least not when "lots and lots" is the tens of millions
> of dollars that Cray Computer will apparently being getting from NSA
> and the Supercomputer folks in Bowie for the delivery of an ostensible
> Cray 4 or whatever it ends up being called (SMPP, etc.).
>
> $10 million is pocket change. Anyone building a company on that chump
> change is already preparing Chapter 11 papers.

I agree, but I was not talking about that amount of money.  My point possibly
would have been better stated as being "don't worry about the price".

> Here's what "lots and lots of money" *really* is:
> - $1 billion to complete a wafer fab in Ireland, finished last February
> - $1.3 billion to build a wafer fab in Albuquerque, to be finished
> later this year

[...]

You're talking about silicon fab lines here, Tim.  As far as I am aware, Cray
has never fab it's own chips.  Indeed, most of their boards which I have
seen (I, II, Y-MP/8 and Y-MP/EL) have used chips sourced from fairly well-
known vendors, such as VLSI Technologies.

The original series of systems (I, II and X-MP) actually used huge numbers
of three types of chips.  From memory, one was a couple of NAND gates, one
was a register chip, and the third was a couple of K of SRAM.  More recently,
the full Y-MP's have been implemented in commercial ECL gate arrays (6500
gates per chip for the full Y-MP's), and the original EL used CMOS 100K
arrays.  I have been trying to get one of the computing industries choicest
pieces of marketting junk: the Cray Y-MP Gate Array paperweight. :)  Of
course, given my recent career change, I don't think one will be coming
my way anytime soon for some strange reason....

Comparing Cray with Intel is rather specious, because the companies are
entirely different beasts.  Intel's supercomputing division is a tack-on
to it's high-end chip line.  Cray never has had a division even vaguely
like the Intel CPU divisions.

> (And a new generation of hackers are using Linux on cheap Pentium
> boxes to easily outrun Suns.)

Not a fair comparison, really.  Sun is the bottom end of the RISC system
market, and is being continually trounced by almost everyone else.  Comparing
the Pentium to our R4400 chips, or HP's PA, or DEC's Alpha would be much more
instructive, and not nearly as favorable to Pentium.

No, I am not knocking Pentium.  Within it's design limitations it's an
interesting accomplishment.  But those design limitations are crippling.

> So you see why I consider the "lots and lots of money" flowing into
> Cray Computer to be spitting into the ocean. I'm not worried.

Agreed.  But the points I was making concerned a comment from the previous
posters about Seymour Cray's design "secrets", NOT the current grant of money
from our good friends at the Puzzle Palace.  Ok, let me explain what I wrote:

> > Secret: take lots and lots and lots of money,

Cray's traditional client-base is money rich, and possess problem sets which
are not practical on conventional architectures.  Those conventional
architectures exist within a cost/afforability framework which limits the
technologies that they can use.  Cray is not unlimited, but it is not nearly
as limited.  They've also got a hidden advantage in that if they do make
a _really_ bad business decision (and I'd say personally that the Cray-III had
been one such), then their customers will probably support them just to
maintain
their current systems.  It's a nice position to be in.

So it's not that Cray has lots and lots of money, but that you can assume that
your customer base will have.

> > use the most exotic packaging
> > technologies you can find,

Lots of people disregard the implications of putting quarter of a million ECL
chips into a column a metre round and a metre and a half high (ie. the Cray
II).
You have BIG heat problems, and in some configurations even flourinert
immersion isn't going to work.

A lot of the cost of these systems is packaging, and Cray really pushes
the state of the art here.  In Seymour Cray's speech to the ACM, he mentions
that fluid immersion of PCB's had never been tried before, as everyone thought
that the boards would bloat.  But they tried it, and the boards were fine,
with the added bonus that because the heat was much more evenly distributed,
the machines were considerably more reliable than expected.

> > pay lots and lots of attention to your memory
> > system and cache,

(BTW, for those people who want to lecture me that Cray's don't have caches,
just consider the different terminology.  Vector registers are nothing more
than user directed data caches, and Crays call their instruction cache
an "instruction buffer".)

As many RISC system manufactures have found, you can put the fastest CPU's
into machines, but without a damn good memory design they spent most of their
time waiting.

Cray's CPU's are not particularly complex.  Cray estimates 1.5 million
transistors to implement a Y-MP CPU.  Most modern RISC CPU's are considerably
more complex than this.

It's the Cray memory system, which on most of the traditional vector machines
is implemented in 10-15nS SRAM with four ports to memory from EACH CPU, that
is the spectacular part of the design.  What Cray uses for main memory (M90
and EL series excepted), most other vendors use for cache.

(Of course, I have to say that SGI's Power Challenge memory systems are getting
pretty impressive too, now.  You can't avoid it if you're supporting the
sorts of performance our newer supercomputer-class systems provide.)

> > don't forget the importance of a nicely balanced
> > architecture (meaning that I/O does matter),

Lots of the Japanese supercomputing vendors forgot this.  Their peak MFLOP
performance was really spectacular, but with real applications they looked
a lot less spectacular.  The data set size of most applications which are
worth running on supercomputers is HUGE, but it's useless if you cannot get
the data too and from disk in less time than it takes to process it.

> > don't forget the importance of
> > good compilers,

Cray has been known to ship systems without even an operating system, it's
true, but only very early on.  Unless you ship _good_ compilers, most of the
applications for the machine won't get written.  Sure, you can program in
CAL, but most people won't.

> > and implement bit counting instructions just like the NSA
> > tells you to.

Ok, I was being facetious here.  But it masks a good point: customer service
is important, and companies with tiny installed bases (eg. the Cray II sold
a total of 31 systems), need to look after their customers.  Lots of other
supercomputer vendors who rolled nice boxes out of the door and then just
went into a backroom to design the next without any customer involvement
don't exist anymore.

							Ian.

Disclaimer: I am NOT speaking for SGI.

Ian F. writes:

> You're talking about silicon fab lines here, Tim.  As far as I am aware, Cray
> has never fab it's own chips.  Indeed, most of their boards which I have
> seen (I, II, Y-MP/8 and Y-MP/EL) have used chips sourced from fairly well-
> known vendors, such as VLSI Technologies.

Yes, of course I was. My point was that the $5-10 M that NSA will put
into to Crayco to keep it on life support for another couple of years
is chump change compared to the investments being made which actually
_will_ alter the economics of things. (And the Pentium is neither here
nor there in this point.)

> > (And a new generation of hackers are using Linux on cheap Pentium
> > boxes to easily outrun Suns.)
> 
> Not a fair comparison, really.  Sun is the bottom end of the RISC system
> market, and is being continually trounced by almost everyone else.  Comparing
> the Pentium to our R4400 chips, or HP's PA, or DEC's Alpha would be much more
> instructive, and not nearly as favorable to Pentium.

My point was that the world is being changed by cheap processors. This
is what will allow VoicePGP to be spread widely, not the fairly slight
performance advantages of R4400s or Alphas.

(There's an interesting thread in the PowerPC and Intel newsgroups
about the performance of a dozen or so machines in running actual
Mathematica code. I'm not trying to start a benchmark debate
here...the point is that PowerMac 8100s were right up near the top, as
were Pentium P90s. The H-P PA machines were the only machines
consistently faster. Alphas often lagged, for various reasons.
Indigos I don't recall the ratings of. The stunner is that machines
people are buying for _home use_ are essentially as fast as the
fastest workstations.)

> Cray's traditional client-base is money rich, and possess problem sets which
> are not practical on conventional architectures.  Those conventional

Crayco has not a sold a single Cray III, which means of course they've
never sold a single machine. Not a single one. Hence the latest
infusion of life support from NSA. (Ian and others of course know
this, but for anyone who is confused: Cray Research and Cray Computer
are two entirely separate companies. Different locales, different
staff. Crayco is developing the Cray III and Cray IV, as we've seen
here. No sales for the Cray III spells dire problems for them.)

> Lots of people disregard the implications of putting quarter of a million ECL
> chips into a column a metre round and a metre and a half high (ie. the Cray
> II).
> You have BIG heat problems, and in some configurations even flourinert
> immersion isn't going to work.
> 
> A lot of the cost of these systems is packaging, and Cray really pushes
> the state of the art here.  In Seymour Cray's speech to the ACM, he mentions
> that fluid immersion of PCB's had never been tried before, as everyone thought

I saw the first Cray 2 running during its shake-down cruise at LLL, in
January 1984. 

As to running boards in Fluorinert, we'd been doing it at Intel since
the late 1970s. We did liquid burn-in of hundreds of chip-filled
boards, at just below the boiling point of the liquid (I think it was
FC-76, but it could've been one of other variants). A lot of people
knew about this, and there was a lot of discussion that I can recall
personally about cooling computers with direct flow Fluorinert. 

(The guy who showed my the Cray 2, Howard Davidson, was already
working on a system involving water at high speeds coursing through
silicon microchannels. Flourinert was rejected as not having enough
kW/cm^2 heat transfer properties.)

I'm not knocking Cray's designs, nor his packaging. Just clarifying
things as I understand them.

I expect to see both Crays eventually go the way of Thinking Machines
and other largely-captive suppliers to the national security
apparatus.


--Tim May


-- 
..........................................................................
Timothy C. May         | Crypto Anarchy: encryption, digital money,  
tcmay@netcom.com       | anonymous networks, digital pseudonyms, zero
408-688-5409           | knowledge, reputations, information markets, 
W.A.S.T.E.: Aptos, CA  | black markets, collapse of governments.
Higher Power: 2^859433 | Public Key: PGP and MailSafe available.
"National borders are just speed bumps on the information superhighway."

Ian F. writes:

> You're talking about silicon fab lines here, Tim.  As far as I am aware, Cray
> has never fab it's own chips.  Indeed, most of their boards which I have
> seen (I, II, Y-MP/8 and Y-MP/EL) have used chips sourced from fairly well-
> known vendors, such as VLSI Technologies.

In my last post, I forgot to mention that Cray Computer Company
actually *did* and *does* fabricate its own chips! They committed to
GaAs from a supplier and then bought the supplier when it faltered.

My recollection is that it was Gigabit Logic, but it _might_ have been
the "other" GaAs supplier, whose name escapes me this minute.

I'm not claiming this as the proximate cause of Crayco's current
problems. But I do think committing to GaAs *was* a factor, and this
relates to the levels of integration in CMOS and BiCMOS versus the
levels currently obtainable in GaAs.

A huge fraction of Crayco's spending went into the advanced robotic
wirebonding and packaging of thousands of GaAs chips. IBM spent vast
fortunes on its advanced packaging/cooling systems, while Intel and
other chip companies concentrated on CMOS VLSI, with much lower
overall packaging and cooling costs for the same performance.

--Tim May



-- 
..........................................................................
Timothy C. May         | Crypto Anarchy: encryption, digital money,  
tcmay@netcom.com       | anonymous networks, digital pseudonyms, zero
408-688-5409           | knowledge, reputations, information markets, 
W.A.S.T.E.: Aptos, CA  | black markets, collapse of governments.
Higher Power: 2^859433 | Public Key: PGP and MailSafe available.
"National borders are just speed bumps on the information superhighway."