What is Parallel Computing?
Put simply, parallel computing occurs when multiple computer actions are occurring at the same time. The devices performing the simultaneous actions can be individual execution units within CPUs or GPUs, the processors themselves, and even whole computers. The important part is that there is a division of work happening between the processing components that ultimately speeds up the rate at which the overall goal is achieved.
<\p>
In this image, you can clearly see parallelism at work. On the left is the serial workflow, in which items 1, 2, and 3 are processed sequentially (or serially), one after the other. On the right is the parallel workflow in which all 3 items are being completed at the same time, thus saving 2 items' worth of processing time compared to the serial workflow.
We can classify the type of parallelism a job is utilizing based on how it is divided among processing components:
- Task parallel jobs are those that have each processing component running a different task from one another, which may all operate on the same or different sets of input data. An example of this would be a process that calculates the average and median of a data set at the same time: one processing element can iterate through the array and sum all of its items, while the other can sort the array in order to determine the median.
- Data parallel jobs essentially run the same process or algorithm on different segments of the input data set at the same time in order to complete the process faster. A common example of this is a parallel sum reduction in which multiple processing elements sum segments of a data set before being combined in the end, reducing the time it takes to sum the whole data set.
Many parallel computing jobs are a combination of both task and data parallelism, meaning that multiple tasks are being performed on multiple parts of a data set at once.
In addition to the way that the job is divided, we can also classify a job's parallelism based on the way processing units running the job store relevant values and communicate among themselves:
-
Shared address space parallelism means that the units that are completing the job all have access to a centralized data store and can communicate simply by saving and retrieving values from the centralized location.
This is commonly the case in multicore (multi-CPU) computers and GPUs due to the reduced amount of time spent communicating between devices enabled by the shared memory, and is the primary type of address space parallelism this series is concerned with. -
Distributed address space parallelism means that the processing units working on the job are entirely data-independent of one another. If two processors need to communicate or share data, they do so by explicitly passing data to one another, a process known as message passing.
This is primarily used in multicomputer parallelism, in which multiple whole computers in a network are linked together to collaborate on a job, because the computers used do not have any shared memory.
In the next tutorial, you will learn about GPUs and how they are built for parallelism at the hardware level.