View source for Parallel computing

[[File:IBM Blue Gene P supercomputer.jpg|thumb|right|230px|[[Super computer]]s are used to exploit parallelism]]
'''Parallel computing''' is a form of [[computing|computation]] in which many [[Instruction (computer science)|instructions]] are carried out simultaneously (termed "in parallel"),<ref>Almasi, G.S. and A. Gottlieb (1989). [http://portal.acm.org/citation.cfm?id=1011116.1011127 ''Highly Parallel Computing'']. Benjamin-Cummings publishers, Redwood City, CA.</ref> depending on the [[theory]] that large problems can often be divided into smaller ones, and then solved [[Concurrency (computer science)|concurrently]] ("in parallel").

There are several different forms of parallel computing: 
#[[Bit-level parallelism]],
#[[Instruction level parallelism|Instruction-level parallelism]],
#[[Data parallelism]], 
#[[Task parallelism]].

It has been used for many years, mainly in [[high performance computing|high-performance computing]], with a great increase in its use in recent years, due to the physical [[constraint]]s preventing [[frequency scaling]]. Parallel computing has become the main model in [[computer architecture]], mainly in the form of [[Multi-core processor]]s.<ref name="View-Power">Asanovic, Krste et al. (December 18, 2006). [http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-183.pdf "The Landscape of Parallel Computing Research: A View from Berkeley"] (PDF). University of California, Berkeley. Technical Report No. UCB/EECS-2006-183.  "Old [conventional wisdom]: Increasing clock frequency is the primary method of improving processor performance. New [conventional wisdom]: Increasing parallelism is the primary method of improving processor performance&nbsp;... Even representatives from Intel, a company generally associated with the 'higher clock-speed is better' position, warned that traditional approaches to maximizing performance through maximizing clock speed have been pushed to their limit."</ref> However, in recent years, [[power consumption]] by parallel computers has become a concern.<ref>Asanovic et al. Old [conventional wisdom]: Power is free, but transistors are expensive. New [conventional wisdom] is [that] power is expensive, but transistors are "free".</ref>

Parallel computers can be classified according to the level at which the hardware supports parallelism—with [[Multi-core processor|multi-core]] and [[Symmetric multiprocessing|multi-processor]] computers having multiple [[Central processing unit|processing elements]] inside a single machine, while [[Computer cluster|clusters]], [[Blade server|blades]], [[Massive parallel processing|MPPs]], and [[Grid computing|grids]] use multiple computers to work on the same task.

[[Parallel algorithm|Parallel computer programs]] are more difficult to [[Computer program|program]] than sequential ones,<ref>[[David A. Patterson (scientist)|Patterson, David A.]] and [[John L. Hennessy]] (1998). ''Computer Organization and Design'', Second Edition, Morgan Kaufmann Publishers, p.&nbsp;715. {{ISBN|1558604286}}.</ref> because concurrency introduces several new classes of potential [[software bug]]s, of which [[race condition]]s and [[Deadlock|dead locks]] are the most common. however many [[Parallel programming model|parallel programming languages]] have been created to simplify parallel computers programming. But still [[Computer networking|communication]] and [[Synchronization (computer science)|synchronization]] between the different subtasks is difficult while achieving good parallel program performance.

== Parallel Computing Media ==
<gallery widths='160px' heights='100%' mode='traditional' caption=''>
File:Parallelism_vs_concurrency.png|Parallelism vs concurrency

File:Gustafson.png|A graphical representation of [[Gustafson's law]]

File:Taiwania 3 Supercomputer.jpg|Taiwania 3 of [[Taiwan]], a parallel supercomputing device that joined [[COVID-19]] research

File:Nopipeline.png|A canonical processor without [[Instruction pipelining|pipeline]]. It takes five clock cycles to complete one instruction and thus the processor can issue subscalar performance ({{nobreak|1=IPC = 0.2 < 1}}).

File:Fivestagespipeline.png|A canonical five-stage [[Instruction pipelining|pipelined]] processor. In the best case scenario, it takes one clock cycle to complete one instruction and thus the processor can issue scalar performance ({{nobreak|1=IPC = 1}}).

File:Superscalarpipeline.svg|A canonical five-stage [[Instruction pipelining|pipelined]] processor with two execution units. In the best case scenario, it takes one clock cycle to complete two instructions and thus the processor can issue superscalar performance ({{nobreak|1=IPC = 2 > 1}}).

File:Numa.svg|A logical view of a [[non-uniform memory access]] (NUMA) architecture. Processors in one directory can access that directory's memory with less latency than they can access memory in the other directory's memory.
</gallery>
== References ==
{{reflist|2}}
== Other websites ==
{{wikibooks|Distributed Systems}}
* {{dmoz|Computers/Parallel_Computing/}}
* [http://www.llnl.gov/computing/tutorials/parallel_comp/ Lawrence Livermore National Laboratory: Introduction to Parallel Computing]{{Webarchive|url=https://web.archive.org/web/20130610122229/https://computing.llnl.gov/tutorials/parallel_comp/ |date=2013-06-10 }}
* [http://www-unix.mcs.anl.gov/dbpp/ Designing and Building Parallel Programs, by Ian Foster]
*[http://wotug.ukc.ac.uk/parallel/ Internet Parallel Computing Archive] {{Webarchive|url=https://web.archive.org/web/20021012122919/http://wotug.ukc.ac.uk/parallel/ |date=2002-10-12 }}
*[http://dsonline.computer.org/portal/site/dsonline/index.jsp?pageID=dso_level1_home&path=dsonline/topics/parallel&file=index.xml&xsl=generic.xsl Parallel processing topic area at IEEE Distributed Computing Online] {{Webarchive|url=https://web.archive.org/web/20110928211245/http://dsonline.computer.org/portal/site/dsonline/index.jsp?pageID=dso_level1_home&path=dsonline%2Ftopics%2Fparallel&file=index.xml&xsl=generic.xsl |date=2011-09-28 }}
*[http://www.new-npac.org/projects/cdroms/cewes-1998-05/copywrite/pcw/book.html Parallel Computing Works Free On-line Book] {{Webarchive|url=https://web.archive.org/web/20110727120355/http://www.new-npac.org/projects/cdroms/cewes-1998-05/copywrite/pcw/book.html |date=2011-07-27 }}
*[http://ark.cdlib.org/ark:/13030/ft0f59n73z/ Frontiers of Supercomputing Free On-line Book Covering topics like algorithms and industrial applications]


{{Tech-stub}}

[[Category:Computer science]]