Brown University Installs First of Two Cray Research Supercomputers

This is a press release from Oct. 26, 1995 announcing that the Brown University Department of Physics was installing a Cray EL98 and J916.

---

Cray:           Mardi Larson, 612/683-3538
Brown:          Professor G.S. Guralnik, 401/863-2623

BROWN UNIVERSITY INSTALLS FIRST OF TWO CRAY
RESEARCH SUPERCOMPUTERS

EAGAN, Minn., Oct. 26, 1995 -- Cray Research announced today
that Providence, RI.-based Brown University has installed the
first of two CRAY(R) supercomputers to be dedicated to the
university's Department of Physics.  According to Cray,
Brown's physics department is the first university physics
department in the U.S. to acquire a Cray system.  Terms were
not disclosed. 

A large-memory (four gigabyte) CRAY EL98(TM) system was
installed this year, to be followed by a CRAY J916(TM) system
scheduled for installation later next year.  These systems are
the first Cray Research supercomputers acquired by Brown and
will be used by Brown University Professors Guralnik,
Kosterlitz, Marston, Tan and other university physicists and
students who are focusing on problems in high-energy and
condensed matter theoretical physics.  These users will run
simulations on the supercomputers to help them learn more
about the basic structure of matter and its interactions.

"Combined with analytic methods, the computational power of
the Cray supercomputers will make an excellent tool for
understanding how microscopic laws determine the higher
level characteristics of matter," said Dr. Brad Marston,
assistant professor of physics at Brown University.  "The Cray
supercomputers will give physicists at Brown a powerful tool
to develop new ideas because of their ability to run the very
complex simulations that are often needed to make predictions
from current theories and ideas," Marston continued.

While these CRAY systems will be used to perform calculations
testing very abstract ideas, in time, the results could become
important for everyday living.

"When we understand something basic about the universe" said
Dr. Gerald Guralnik, professor of physics at Brown, "it
eventually affects the way we live.  That is how we generated
the technological advances we already have today.  The
knowledge that we obtain could be essential for developing
new energy sources as well as leading to new product
developments in the semiconductor and materials industries." 

The CRAY EL98 and CRAY J916 are Cray's compact, air-cooled
supercomputers priced starting at $250,000 (U.S. list price). 
The supercomputers are designed to operate as powerful
simulation servers for compute-intensive problems that
challenge the capabilities of workstations.  Since its
announcement in September 1994, the current CRAY J90 line
has attracted more than 200 orders, with about 40 percent
coming from new-to-Cray customers including from new
industries. 

Cray Research provides the leading high-performance
computing tools and services to help solve customers' most
challenging problems.

                               ###

---

Source: Brown University Installs First of Two Cray Research Supercomputers

Silicon to Supercomputer

The J90 logic is implemented using application-specific integrated circuit (ASIC) chips fabricated by IBM. There are 10 unique ASICs that are found in the processor and memory modules. A typical J90 system could contain about 230 of these CMOS chips. The photo below shows a processor module with the cover removed. Each module contains 4 scalar/vector processors. The space at the top of the board can be used for optional HIPPI interfaces or Y1 Channels to additional I/O Processors.

A Cray J90 quad processor module.

The ASIC chip types are:

• MBI - DRAM memory interface
• MAD - Memory side of memory crossbar for read data
• MAR - Memory side of memory crossbar for write data
• VA - CPU side of memory crossbar for write data
• VB - CPU side of memory crossbar for read data
• CI - Channel interface (I/O)
• JS - Shared registers for multi-CPU applications
• PC - Scalar processor and processor control
• VU - Vector processor
• MC - Maintenance and clock distribution

There is only one chip (called PC) for each scalar processor and one additional chip (called VU) for each vector processor. There are only 8 chips on each processor module for the CPUs and the rest of the 18 out of 26 chips are used for communication between processors or between the processors and the memory banks. This circuitry is the key to a "balanced" system where the memory bandwidth is great enough to sustain the rate at which the processors can operate on the data.

Power Up

The Cray J916 was featured at our monthly open house this past Saturday. We spent most of the day talking to visitors about the system and showing them the rest of our collection. We did make some progress in the morning before we opened. Dave took some photos while I was working.

Working on the system clock board on the J90 backplane.
The Central Control Unit is in the foreground.

Everything seems to be functional with one exception - there is a system clock PWR FAULT light showing on the Central Control Unit. We suspected a loose cable or board connector and traced the signal paths. We couldn't find the cause of the fault and will need to dig deeper another time.

Jedi vs. the Droids

How does the performance of a 20 year old supercomputer compare to the devices that we use today? Let's compare the Cray J916 to a recent laptop and a smart phone.

According to an archived copy of the Cray J90 Series webpage the vector processors have a theoretical peak performance of 200 mflops each, giving our 8 CPU system 1.6 gflops. But, your mileage may vary depending on the code that is running. One of the standard benchmarks used for supercomputers is LINPACK. Results for a J916 with the same configuration as ours are listed in Performance of Various Computers Using Standard Linear Equations Software by Jack J. Dongarra from June 1995. An 8 CPU system was measured at 1.436 gflops, an efficiency of about 90% of the theoretical peak.

"Just right for you" - it certainly is for us...

Next we'll run Linpack for Android. My Samsung Galaxy Note II has a 1.6 GHz ARM Cortex-A9 with four cores. Running LINPACK multi-threaded gives about 200 mflops, just a little faster than a single J90 processor. So, yes, that is (nearly) a mid-1990s entry level supercomputer in my pocket. At least on paper. We're really just exercising the ability to do floating point calculations, and this is not necessarily a good measure of system throughput on a real problem.

I estimate that the theoretical peak performance of the ARM is about 3 gflops or so, giving well below 10% efficiency. (I'm ignoring the GPU as I have no way to run LINPACK on it to benchmark it.) I should mention that the Android version of LINPACK is based on this Java Version and the low efficiency is in part due to the Java Virtual Machine.

But, overall, the Cray system with a 100 MHz clock speed has roughly 7.5 times the performance of an Android running at 1.6 GHz.

bootp

We are at the point where we can power up the Cray and begin configuring it. Next we need to work on the J90 System Console (or SWS, the Service WorkStation.) This is used to net boot the I/O Processors in the I/O Subsystem, which in turn loads UNICOS into the J90 main memory. The SWS is a Sun SPARCstation 5 running Solaris. We didn't get the original SS5 that came with the Cray so we had to build one.

The SWS with ThinWire Ethernet, FDDI, and graphics.

We had a couple of these lying around, but they weren't in very good shape. We cobbled together a system from the parts which is better configured than what would have been used when the J90 was installed in 1996. This system has the maximum of 256 MB of RAM and a 24 bit S24 TCX graphics card.

There are two Ethernet interfaces. The one on the SBus card is 10BASE2, more informally known as ThinWire. This is only used to connect the SWS to the two I/O Processors in the I/O Subsystem. This allows the IOPs to net boot from the SWS and is also used to configure and manage the system.

System Ready

In a previous post I described the power requirements of the Cray J916 and the importance of the Central Control Unit (CCU) monitoring for faults. This is rather critical as the machine won't function properly unless these hardware status signals check out.

The back of the I/O Subsystem in the Peripheral Cabinet. This is the cleanest cable management system I've ever seen, but tracing cable paths while re-wiring the system was time consuming.

With the system re-cabled we can power it up and begin testing the hardware. We noticed that the CCU contains rechargeable D size batteries. Yes, we received a donated Cray and batteries were included! They were, of course, very dead. The system had been unplugged for a long time.

Under the Covers

What's inside a Cray J90? I'm going to take a pause in the restoration to "pop the hood" and describe what's under the covers. Here we have the two cabinets bolted together with the doors and most of the panels removed. On the right is the Processing Cabinet and on the left is the Peripheral Cabinet.

The Cray J916 with the front doors and many of the filler panels removed.

The Processing Cabinet has a Central Control Unit at the top which has LEDs that display status and fault conditions. Below that is a plenum space for the cooling blower intake and dust filters. Next is the backplane (the Processor and Memory Modules are inserted from the back.) Under that is the blower which exhausts out the back. At the bottom are the 48 volt DC power supplies.

The Peripheral Cabinet is moderately configured with plenty of space for expansion. In the center is the I/O Subsystem VME Chassis which contains two VME backplanes. Each contains a SPARC single-board computer which functions as an I/O Processor (IOP.) The remaining VME slots contain the system interfaces including Ethernet, disk controllers, and the I/O Buffer Boards (IOBB) that transfer data from peripherals to the J90 processors and memory. At the bottom are two disk arrays.

Kilowatts, Control Cables, and Cooling

The Retro-Computing Society of RI is located in the Atlantic Mills in the Olneyville neighborhood of Providence. We have a 1,500 sq. ft. facility in a mixed-use complex that was originally a worsted mill during the industrial revolution. It suits our needs quite well with amenities like a loading dock and a freight elevator. As I described in the previous installment, the system weighs more than a half ton. It was surprisingly easy to move as it has well designed casters that allow it to be smoothly rolled around. Now that it has arrived we need a plan to plug it in. Our electrical panel was a bit under-powered for running the Cray.

The back of the J916 with the two cabinets bolted together. Each cabinet has an AC power entry box which we had removed to prevent damage to the power cord during the move.

The system requires three-wire, single-phase, at 240 volts. A fully configured Processor Cabinet could draw a maximum of 4,200 watts while a full Peripheral Cabinet could draw up to of 3,600 watts. Our J916 is a moderately configured system which is about one half full. There is plenty of space for expansion to add additional drive bays, for example. There is a helpful Electrical Requirements Worksheet in the Preparing for a System Installation manual. For this configuration we calculate that both cabinets draw a total of about 2,000 watts or so. The property manager scheduled an electrician to upgrade our service and install the receptacles. We also made other changes to make it easier to power up some of our other machines that we haven't been able to run recently. The work was completed on June 30, 2014.

Customer Name (if not confidential)

It was about four years ago when I first learned that there was still a Cray in the Department of Physics at Brown University. The machine was in the same building as my office; for years I had no idea it was sitting idle just a few floors above me. My colleague Prof. Ian Dell'Antonio facilitated the donation to the Retro-Computing Society of RI. He uses the high energy theory cluster for research on observational cosmology and gravitational lensing.

My other computer is a Cray...

In my previous installment I described the Theory Computing Cluster machine room where the Cray J916 had been used for high energy theoretical physics research at Brown University. It was installed in 1996. I'm not sure exactly when they stopped using it. I was told that it was difficult to program and had been unused for some time. Supercomputers tend to have a rather short shelf life.


We first moved the Cray to Brown's Science Center where it was on display for a few years. Before it could be safely moved I had to "split" the cabinets. There is one Peripheral Cabinet housing the I/O Subsystem and disk arrays. The second cabinet is the mainframe, or Processing Cabinet, which contains the CPUs and memory. The Cray documentation sometimes uses the archaic definition of mainframe to refer to the primary frame (or cabinet) that contains the central processor. The Oxford English Dictionary cites a usage from Honeywell in 1964 and gives this definition:

mainframe, n.

2. Computing. Originally: the central processing unit and primary memory of a computer. Now usually: any large or general-purpose computer, esp. one supporting numerous peripherals or subordinate computers.

The Theory Cluster

The use of high performance computers has had a tremendous impact on the progress of science. Theses machines have enabled us to advance our understanding of everything from elementary particles to the large scale structure of the universe. The fastest systems of any era are referred to as supercomputers. For many years supercomputing was synonymous with the machines designed by Seymour Cray at Control Data Corporation and later at Cray Research.

Supercomputers have always been very large and expensive. They require a large amount of electrical power and exotic cooling systems. They are typically a shared resource only used at large government research laboratories and academic institutions. By the late 1980s a new class of minisupercomputer was introduced. With a price starting at less than one million dollars these smaller air-cooled systems could exclusively be used by a research group or academic department.

In the mid 1990s the Brown University Department of Physics was the first physics department in the U.S. to acquire a Cray system. In Augusts 1995 a Cray EL98 was installed. This was followed in late 1996 with the installation of a Cray J916. They were used for high-energy and condensed matter theoretical physics. Details of the research are at the Computational High Energy Physics group page.

The Theory Cluster Machines webpage of the High Energy Physics Group at Brown University. The page was created in late 1995 and includes a publicity photo of the Cray EL98 that had just been installed. The snapshot was captured using NCSA X Mosaic on a SPARCstation 5 running Solaris.