Computer Taxonomies

In 1966[1], Michael Flynn developed a taxonomy of computer systems based on the notions of streams of instructions and data.

It was his observation that one could describe, at a very coarse level, the degree of parallelism of a computer based on these two parameters: either the instruction or the data stream could be either serial or parallel.

He begins by defining instruction and data streams:

Instruction stream

sequence of instructions performed by machine

Data stream

sequence of data called for by instruction stream (including input and partial results)

SISD (Single Instruction Single Data - IBM 704)

This is the classical computer: it executes a single instruction at a time, and executes a single operation on the data.

Confluent SISD (CDC 6600)

overlap execution of serial instructions. Instruction decoder is bottleneck (Flynn mentions that it would be possible to decode multiple instructions per cycle give either more complex decoder or restrictions on programming practices. This has happened now...). While Flynn's illustrations show pipelined execution, he really means more like dedicated CDC-style functional units.

SIMD (Single Instruction Multiple Data - ILLIAC IV)

A single instruction operates on multiple data elements.

MISD (Multiple Instruction, Single Data)

Flynn's original definition has fallen from use. Assume high bandwidth dedicated execution units as in confluent SISD. Now assume some number of virtual machines dispatching instructions to these execution units. Hmm, P4 does this....

MIMD (Multiple Instruction Multiple Data)

This is the opposite extreme from SISD: a computer system in which there are several instructions streams operating simultaneously (and in large part independently), and each of the instruction streams operating on its own data stream.

As some examples, most computers today present an SISD model to the programmer, but are implemented as pipelines. Low-end microcontrollers (like the HC11) are SISD. Crays present an SIMD model in their vector instructions, but is implemented as pipelines. The Goodyear Massively Parallel Processor and the Connection Machine (CM-1 and CM-2) were both SIMD.

A later extension to his classification includes MSIMD computers: ensembles of multiple SIMD machines.

We also frequently refer to SPMD (Single Program Multiple Data) machines. In this case the computer itself is actually MIMD, but we execute copies of a single program on all the nodes. This makes the problem of managing parallel processing a lot simpler (imagine trying to program each of Google's 10,000 machines with a different program!).

We can also further subdivide the categories. In particular, there are a lot of variations on MIMD machines:

Distributed Memory

A machine with a pile o' PCs communicating using a fast network. Clusters meeting this description, typically "Beowulf" clusters running Linux, have become the dominant model of very high performance computing today.

Global Memory

A with several processors all sharing a common memory using a bus.

Message Passing

Processes do not share memory; instead, they communicate data as needed by sending a message under programmer control.

Shared Variable

Processes communicate by simply reading and writing shared locations.

Distributed memory is typically associated with message passing, and shared memory with shared variables - but these associations can be reversed!

Programming with shared variables is typically easier than programming with message passing - you can just put the data where you want, and read it when you want. On the other hand, message passing always has exactly one process in control of any data element, making debugging much less difficult (a slightly different statement than easier!).

References