# Analysis of parallel algorithms

This article discusses the analysis of parallel algorithms. Like in the analysis of "ordinary", sequential, algorithms, one is typically interested in asymptotic bounds on the resource consumption (mainly time spent computing), but the analysis is performed in the presence of multiple processor units that cooperate to perform computations. Thus, one can determine not only how many "steps" a computation takes, but also how much faster it becomes as the number of processors goes up. Interestingly, the analysis approach works by first suppressing (or abstracting away) the number of processors. The next background paragraph explains how the abstraction of the number of processors first emerged.

A so-called work-time (WT) (sometimes called work-depth, or work-span) framework was originally introduced by Shiloach and Vishkin [1] for conceptualizing and describing parallel algorithms. In the WT framework, a parallel algorithm is first described in terms of parallel rounds. For each round, the operations to be performed are characterized, but several issues can be suppressed. For example, the number of operations at each round need not be clear, processors need not be mentioned and any information that may help with the assignment of processors to jobs need not be accounted for. Second, the suppressed information is provided. The inclusion of the suppressed information is, in fact, guided by the proof of a scheduling theorem due to Brent [2], which is explained later in this article. The WT framework is useful since while it can greatly simplify the initial description of a parallel algorithm, inserting the details suppressed by that initial description is often not very difficult. For example, the WT framework was adopted as the basic presentation framework in the parallel algorithms books (for the Parallel random-access machine PRAM model) [3] and [4], as well as in the class notes [5]. The overview below explains how the WT framework can be used for analyzing more general parallel algorithms, even when their description is not available within the WT framework.

## Overview

Suppose computations are executed on a machine that has p processors. Let T_{p} denote the time that expires between the start of the computation and its end. Analysis of the computation's running time focuses on the following notions:

- The
*work*of a computation executed by p processors is the total number of primitive operations that the processors perform.[6] Ignoring communication overhead from synchronizing the processors, this is equal to the time used to run the computation on a single processor, denoted*T*_{1}. - The
*depth*or*span*is the length of the longest series of operations that have to be performed sequentially due to data dependencies (the*critical path*). The depth may also be called the*critical path length*of the computation.[7] Minimizing the depth/span is important in designing parallel algorithms, because the depth/span determines the shortest possible execution time.[8] Alternatively, the span can be defined as the time*T*_{∞}spent computing using an idealized machine with an infinite number of processors.[9] - The
*cost*of the computation is the quantity pT_{p}. This expresses the total time spent, by all processors, in both computing and waiting.[6]

Several useful results follow from the definitions of work, span and cost:

*Work law*. The cost is always at least the work:*pT*≥_{p}*T*_{1}. This follows from the fact that p processors can perform at most p operations in parallel.[6][9]*Span law*. A finite number p of processors cannot outperform an infinite number, so that*T*≥_{p}*T*_{∞}.[9]

Using these definitions and laws, the following measures of performance can be given:

*Speedup*is the gain in speed made by parallel execution compared to sequential execution:*S*=_{p}*T*_{1}∕*T*. When the speedup is Ω(_{p}*n*) for input size n (using big O notation), the speedup is linear, which is optimal in simple models of computation because the work law implies that*T*_{1}∕*T*≤_{p}*p*(super-linear speedup can occur in practice due to memory hierarchy effects). The situation*T*_{1}∕*T*=_{p}*p*is called perfect linear speedup.[9] An algorithm that exhibits linear speedup is said to be scalable.[6]*Efficiency*is the speedup per processor,*S*∕_{p}*p*.[6]*Parallelism*is the ratio*T*_{1}∕*T*_{∞}. It represents the maximum possible speedup on any number of processors. By the span law, the parallelism bounds the speedup: if*p*>*T*_{1}∕*T*, then:_{∞}

.[9]

- The
*slackness*is*T*_{1}∕ (*pT*_{∞}). A slackness less than one implies (by the span law) that perfect linear speedup is impossible on p processors.[9]

## Execution on a limited number of processors

Analysis of parallel algorithms is usually carried out under the assumption that an unbounded number of processors is available. This is unrealistic, but not a problem, since any computation that can run in parallel on N processors can be executed on *p* < *N* processors by letting each processor execute multiple units of work. A result called *Brent's law* states that one can perform such a "simulation" in time T_{p}, bounded by[10]

or, less precisely,[6]

An alternative statement of the law bounds T_{p} above and below by

- .

showing that the span (depth) *T*_{∞} and the work *T*_{1} together provide reasonable bounds on the computation time.[2]

## References

- Shiloach, Yossi; Vishkin, Uzi (1982). "An
*O*(*n*^{2}log*n*) parallel max-flow algorithm".*Journal of Algorithms*.**3**(2). pp. 128–146. doi:10.1016/0196-6774(82)90013-X. - Brent, Richard P. (1974-04-01). "The Parallel Evaluation of General Arithmetic Expressions".
*Journal of the ACM*.**21**(2): 201–206. CiteSeerX 10.1.1.100.9361. doi:10.1145/321812.321815. ISSN 0004-5411. - JaJa, Joseph (1992).
*An Introduction to Parallel Algorithms*. Addison-Wesley. ISBN 978-0-201-54856-3. - Keller, Jorg; Kessler, Cristoph W.; Traeff, Jesper L. (2001).
*Practical PRAM Programming*. Wiley-Interscience. ISBN 978-0-471-35351-5. - Vishkin, Uzi (2009).
*Thinking in Parallel: Some Basic Data-Parallel Algorithms and Techniques, 104 pages*(PDF). Class notes of courses on parallel algorithms taught since 1992 at the University of Maryland, College Park, Tel Aviv University and the Technion. - Casanova, Henri; Legrand, Arnaud; Robert, Yves (2008).
*Parallel Algorithms*. CRC Press. p. 10. CiteSeerX 10.1.1.466.8142. - Blelloch, Guy (1996). "Programming Parallel Algorithms" (PDF).
*Communications of the ACM*.**39**(3): 85–97. CiteSeerX 10.1.1.141.5884. doi:10.1145/227234.227246. - Michael McCool; James Reinders; Arch Robison (2013).
*Structured Parallel Programming: Patterns for Efficient Computation*. Elsevier. pp. 4–5. - Cormen, Thomas H.; Leiserson, Charles E.; Rivest, Ronald L.; Stein, Clifford (2009) [1990].
*Introduction to Algorithms*(3rd ed.). MIT Press and McGraw-Hill. pp. 779–784. ISBN 0-262-03384-4. - Gustafson, John L. (2011). "Brent's Theorem".
*Encyclopedia of Parallel Computing*. pp. 182–185. doi:10.1007/978-0-387-09766-4_80. ISBN 978-0-387-09765-7.