An Overview of Computer System Performance Evaluation
- CSCI 6175 Seminar in Computer Science

Xiannong Meng
Department of Computer Science
The University of Texas - Pan American
Edinburg, TX 78539-2999
meng@panam.edu

Fall 1997

Opening Questions

• Is a 200 M Hz processor necessarily ``faster'' than a 133 M Hz processor?
• Or is a 1 BIPS processor necessarily ``faster'' than a 700 MIPS (0.7 BIPS) processor?
• Will a faster network necessarily send data through easier? (for example 100 M bps Ethernet vs. 10 M bps Ethernet)

The answer to all three questions are ``not necessarily''. The first two depend on issues such as what kind of instructions are in question, how many instructions can be in pipeline, etc.

The answer to last question depends on what kind of network protocol is in use. For a bus-based protocol such as Ethernet, faster speed may actually send less data through.

Computer systems performance evaluation is a collection of theories, practices, sometimes ``arts'' to be used in analyzing computer systems which include processors, primary memory, secondary memory, system bus, networks, and others.

Outline of the Presentation

• Basic concepts
• The question of workload
• Performance metrics
• Three types of techniques
  • Measurement
  • Simulation
  • Analytical Study
• What others are doing
• My work

The Question of Workload

• The term test workload denotes any workload used in performance studies.
• To make any performance analysis valid, one has to consider what kind of workload to be used in the study.
• A real workload is one observed on a system being used for normal operations. Problem: can't repeat, thus not suitable for study.
• A synthetic workload is one whose characteristics are similar to those of real workload and can be applied repeatedly in a controlled manner.

 
Table: Gibson Instruction Mix (quoted from Raj Jain's book page 49)

Performance Metrics

Typical performance metrics include the following.

Delay :

e.g. how long would it take to transfer data from disk to memory?

Response time :

e.g. how long would it take for a job to get started?

Throughput :

e.g. how many jobs can be processed in a given period of time?

Utilization :

e.g. how much time was the CPU used? How much time was it idle?

Three Types Techniques

Typically, three different types of performance evaluation techniques can be used (or a mix of these).

Measurement:

Using software or hardware actually measure predefined performance metrics.

Issues

• Interference
• Statistical significance
• Results tend to be more accurate, more meaningful, more directly applicable
• Expensive, difficult to repeat

Common benchmark programs

Sieve:

Find all prime numbers in a given range, computation intensive

Ackermann's function:

Calculate a particular numeric value, computation intensive

Some values of A(x,y): ,

Whetstone:

Whetstone consists of a set of 11 modules designed to match observed dynamic frequency of operations used in 949 ALGOL programs. (later it was translated into other programming languages). The measurement unit is KWIPS (Kilo Whetstone Instructions Per Second).

LINPACK:

Developed at Argonne National Lab in 1983, this benchmark program consists of a number of programs that solve dense systems of linear equations. The measurement unit is MFLOPS (Million Floating Point Instructions per Second).

Dhrystone:

Developed at Siemens in 1984, this set contains many procedure calls and is considered to represent systems programming environment. Available in C, Pascal, and Ada. The measurement unit is Dhrystone Instructions Per Second (DIPS).

Lawrence Livermore Loops:

This contains a set of 24 separate tests dominated by large scientific calculations that are largely vectorizable. The measurement unit is MFLOPS.

Debit-Credit Benchmark:

While above mentioned benchmarks are computation-oriented, this is a transaction process oriented. It represents a distributed banking network. The measurement unit is Transactions Per Second (TPS).

SPEC Benchmark Suite:

Most recent, more widely used benchmark program. The recent version include SPEC CPU95 (including integer, floating etc.) and SPEC WEB96. The measurement unit is SPECmark.

Simulation:

Here we are mainly concerned with statistical simulation. A simulation is the process of designing a mathematical or logical model of a real system and then conducting computer-based experiments with the model to describe, explain, and predict the behavior of the real system.

Issues:

• Easy to vary the parameters
• Easy to repeat
• How to model the real system accurately
• Statistical significance
• How to interpret results (analysis of simulation results)

Types of simulation:

Monte Carlo Simulation:

A simulation without time axis involved. For example, Buffon's needle experiment.

Trace-Driven simulation:

A simulation using a trace as its input. A trace is a time-ordered record of events on a real system.

Discrete-Event Simulation:

A simulation using a discrete-state model of the system.

Analytical Study:

Analytical performance study uses mathematical model (often a queueing model) to represent the system under study. The solution for the queueing model represents the characteristics of a real system.

Issues:

• Simple to perform (paper and pencil)
• Powerful
• Less accurate
• Sometimes a solution to the queueing model is difficult to come up

Ethernet example: maximum throughput of Ethernet can be represented as follows (see Lam80).

where

and is the propagation delay (time for signal to travel in wire, often taken as per 1,000 meters) and T is the packet transmitting time.

For a Ethernet of 500 m length, 1500 bytes packet length

What Others Are Doing

From recent SIGMETRICS conference

• Network measurement and modeling: wide-area network, WWW
• Network resource management: queue management for rate based congestion control, TCP over ATM
• Parallel computer systems: performance debugging shared memory parallel programs
• Network support for video
• Network performance modeling
• Memory management: informed multiprocess prefetching and caching, adaptive page replacement based on memory reference behavior

My Work

• Queueing model and simulation of metropolitan area network with multimedia traffic (remote past)
• Queueing model and simulation of load sharing and fault-tolerance computing systems (remote past)
• Measured performance of distributed simulations (recent past)
• Measured performance comparison of TCP/IP communications in C++ and Java (recent)
• Simulation and (hopefully) queueing model of telephone center where customers can take vacation (David Lung)
• Web page access pattern (to do)
• Web caching pattern (to do)
• Queuing model for WWW (to do)

Any takers?

• About this document ...