All new 6.0.1 modifications to existing files.

Author: spophale

Chap_SIMD.tex

@@ -11,21 +11,21 @@
 Loops without loop-carried backward dependences (or with dependences preserved using
 \kcode{ordered simd}) are candidates for vectorization by the compiler for 
 execution with SIMD units. In addition, with state-of-the-art vectorization 
-technology and \kcode{declare simd} directive extensions for function vectorization
-in the OpenMP 4.5 specification, loops with function calls can be vectorized as well. 
+technology and \kcode{declare_simd} directive extensions for function vectorization
+in the OpenMP Specification, loops with function calls can be vectorized as well. 
 The basic idea is that a scalar function call in a loop can be replaced by a vector version 
 of the function, and the loop can be vectorized simultaneously by combining a loop 
 vectorization (\kcode{simd} directive on the loop) and a function 
-vectorization (\kcode{declare simd} directive on the function).
+vectorization (\kcode{declare_simd} directive on the function).
 
 A \kcode{simd} construct states that SIMD operations be performed on the
-data within the loop.  A number of clauses are available to provide
+data within the loop. A number of clauses are available to provide
 data-sharing attributes (\kcode{private}, \kcode{linear}, \kcode{reduction} and 
 \kcode{lastprivate}).  Other clauses provide vector length preference/restrictions 
 (\kcode{simdlen} / \kcode{safelen}), loop fusion (\kcode{collapse}), and data 
 alignment (\kcode{aligned}).
 
-The \kcode{declare simd} directive designates
+The \kcode{declare_simd} directive designates
 that a vector version of the function should also be constructed for 
 execution within loops that contain the function and have a \kcode{simd} 
 directive.  Clauses provide argument specifications (\kcode{linear},

Chap_affinity.tex

@@ -1,74 +1,59 @@
 \cchapter{OpenMP Affinity}{affinity}
 \label{chap:openmp_affinity}
 
-OpenMP Affinity consists of a \kcode{proc_bind} policy (thread affinity policy) and a specification of
-places (``location units'' or \plc{processors} that may be cores, hardware
-threads, sockets, etc.).  
-OpenMP Affinity enables users to bind computations on specific places.
-The placement will hold for the duration of the parallel region. 
-However, the runtime is free to migrate the OpenMP threads 
-to different cores (hardware threads, sockets, etc.) prescribed within a given place, 
-if two or more cores (hardware threads, sockets, etc.) have been assigned to a given place.
-
-Often the binding can be managed without resorting to explicitly setting places.
-Without the specification of places in the \kcode{OMP_PLACES} variable, 
-the OpenMP runtime will distribute and bind threads using the entire range of processors for 
-the OpenMP program, according to the \kcode{OMP_PROC_BIND} environment variable
-or the \kcode{proc_bind} clause.  When places are specified, the OMP runtime
-binds threads to the places according to a default distribution policy, or
-those specified in the \kcode{OMP_PROC_BIND} environment variable or the
-\kcode{proc_bind} clause.
-
-In the OpenMP Specifications document a processor refers to an execution unit that
-is enabled for an OpenMP thread to use.  A processor is a core when there is
-no SMT (Simultaneous Multi-Threading) support or SMT is disabled.  When 
-SMT is enabled, a processor is a hardware thread (HW-thread). (This is the
-usual case; but actually, the execution unit is implementation defined.) Processor
-numbers are numbered sequentially from 0 to the number of cores less one (without SMT), or
-0 to the number HW-threads less one (with SMT). OpenMP places use the processor number to designate
-binding locations (unless an ``abstract name'' is used.) 
-
-
-The processors available to a process may be a subset of the system's
-processors.  This restriction may be the result of a 
-wrapper process controlling the execution (such as \plc{numactl} on Linux systems), 
-compiler options, library-specific environment variables, or default
-kernel settings.  For instance, the execution of multiple MPI processes,
-launched on a single compute node, will each have a subset of processors as
-determined by the MPI launcher or set by MPI affinity environment 
-variables for the MPI library.  %Forked threads within an MPI process
-%(for a hybrid execution of MPI and OpenMP code) inherit the valid 
-%processor set for execution from the parent process (the initial task region) 
-%when a parallel region forks threads.  The binding policy set in 
-%\code{OMP\_PROC\_BIND} or the \code{proc\_bind} clause will be applied to 
-%the subset of processors available to \plc{the particular} MPI process.
-
-%Also, setting an explicit list of processor numbers in the \code{OMP\_PLACES} 
-%variable before an MPI launch (which involves more than one MPI process) will
-%result in unspecified behavior (and doesn't make sense) because the set of 
-%processors in the places list must not contain processors outside the subset 
-%of processors for an MPI process. A separate \code{OMP\_PLACES} variable must
-%be set for each MPI process, and is usually accomplished by launching a script 
-%which sets \code{OMP\_PLACES} specifically for the MPI process. 
-
-Threads of a team are positioned onto places in a compact manner, a 
-scattered distribution, or onto the primary thread's place, by setting the 
-\kcode{OMP_PROC_BIND} environment variable or the \kcode{proc_bind} clause  to 
-\kcode{close}, \kcode{spread}, or \kcode{primary} (\kcode{master} has been deprecated), respectively.  When 
-\kcode{OMP_PROC_BIND} is set to FALSE no binding is enforced; and 
-when the value is TRUE, the binding is implementation defined to 
-a set of places in the \kcode{OMP_PLACES} variable or to places 
-defined by the implementation if the \kcode{OMP_PLACES} variable 
-is not set. 
-
-The \kcode{OMP_PLACES} variable can also be set to an abstract name 
-(\kcode{threads}, \kcode{cores}, \kcode{sockets}) to specify that a place is
-either a single hardware thread, a core, or a socket, respectively. 
-This description of the \kcode{OMP_PLACES} is most useful when the 
-number of threads is equal to the number of hardware thread, cores
-or sockets.  It can also be used with a \kcode{close} or \kcode{spread} 
-distribution policy when the equality doesn't hold.
-
+OpenMP defines \emph{thread affinity} with respect to \emph{places}, where a
+place is an abstraction that represents a set of processors (e.g., one or more
+processor IDs, a hardware thread, a core, a socket, etc.). Thread affinity
+control enables users to assign threads that perform computation in a parallel
+region to specific places, while allowing the runtime implementation to freely
+migrate threads to different execution units within a given place. A thread
+that is assigned to a place for a given parallel region remains bound to that
+place for the duration of that region.
+
+The places available for thread affinity control (referred to as a \emph{place
+partition}) can be set via the \kcode{OMP_PLACES} environment variable. The
+binding of threads to places can be managed explicitly or handled implicitly.
+Without the \kcode{OMP_PLACES} variable being set, the initial place partition
+is implementation defined.  The method by which threads are assigned to places
+for a given parallel region is determined by the specified thread affinity
+policy. This policy can be set via the \kcode{OMP_PROC_BIND} environment
+variable or can be explicitly set for a particular \kcode{parallel} construct
+with the \kcode{proc_bind} clause.
+
+The OpenMP specification document defines a \emph{processor} as a hardware
+execution unit on which one or more OpenMP threads may execute. The actual
+hardware mechanism that a given processor ID represents depends on the
+implementation and architecture. For example, a processor could correspond to a
+core on the device that does not have simultaneous multi-threading (SMT)
+support or for which SMT is disabled. While for an SMT-enabled device, a
+processor could correspond to a hardware thread. Processor IDs are the
+resulting sequential numbering of processors, starting from 0. The initial
+place partition can be defined explicitly with processor IDs or using an
+\emph{abstract name}. For example, \pout{OMP_PLACES="\{0,1\},\{2,3\}"}
+defines two places in the initial place partition, the first place consisting
+of processors 0 and 1 from the device and the second place consisting of
+processors 2 and 3 from the device. Alternatively, \pout{OMP_PLACES="cores"}
+defines there to be one place per core on the host device.
+
+The processors that are available to an OpenMP program process may be a subset
+of the processors on the system. This restriction may be the result of a
+wrapper process controlling the execution (such as \ucode{numactl} on Linux
+systems), compiler options, library-specific environment variables, or default
+kernel settings. For instance, the execution of multiple MPI processes, launched
+on a single compute node, will each have a subset of processors as determined
+by the MPI launcher or set by MPI affinity environment variables for the MPI
+library.
+
+The threads that are under affinity control for a given parallel region include
+the threads assigned to its team and additionally any free-agent threads (see
+Section~\ref{sec:free_agent}) that execute tasks bound to the region. Affinity
+control for threads can be disabled (i.e., allowing threads to migrate freely
+across processors) by setting \kcode{OMP_PROC_BIND} to \vcode{false}. If instead
+\kcode{OMP_PROC_BIND} is \vcode{true}, then threads will bind to places but the
+places to which they bind are implementation defined. Finally, three affinity
+policies that are more prescriptive are available via the environment variable
+or the \kcode{proc_bind} clause: \kcode{spread}, \kcode{close}, and
+\kcode{primary}. These are detailed in the following section. 
 
 % We need an example of using sockets, cores and threads:
 

Chap_data_environment.tex

@@ -1,32 +1,42 @@
 \cchapter{Data Environment}{data_environment}
 \label{chap:data_environment}
-The OpenMP \plc{data environment} contains data attributes of variables and
-objects.  Many constructs (such as \kcode{parallel}, \kcode{simd}, \kcode{task}) 
-accept clauses to control \plc{data-sharing} attributes
-of referenced variables in the construct, where \plc{data-sharing} applies to
-whether the attribute of the variable is \plc{shared}, 
-is \plc{private} storage, or has special operational characteristics 
-(as found in the \kcode{firstprivate}, \kcode{lastprivate}, \kcode{linear}, or \kcode{reduction} clause).
+An OpenMP \emph{data environment} is defined by a set of variables or objects
+and their \emph{data-environment attributes}. Data-environment attributes can
+be divided into \emph{data-sharing attributes} and \emph{data-mapping
+attributes}. 
 
-The data environment for a device (distinguished as a \plc{device data environment})
-is controlled on the host by \plc{data-mapping} attributes, which determine the
-relationship of the data on the host, the \plc{original} data, and the data on the
-device, the \plc{corresponding} data.
+Many constructs (such as \kcode{parallel}, \kcode{simd}, \kcode{task}) 
+accept clauses to control data-sharing attributes of referenced variables in
+the construct, where data-sharing applies to whether the attribute of the
+variable is \emph{shared} or \emph{private}, in addition to other special
+operational characteristics of private (as indicated by the
+\kcode{firstprivate}, \kcode{lastprivate}, \kcode{linear}, or \kcode{reduction}
+clause).
+
+Variables and objects in the data environment for a target device
+(distinguished as a device data environment) have \emph{data-mapping
+attributes} that are controlled by data-mapping constructs (such as
+\kcode{target} or \kcode{target_data}), which determine the relationship of the
+data on the host (the \emph{original} data) and the data on the device (the
+\emph{corresponding} data).
 
 \bigskip
 DATA-SHARING ATTRIBUTES
 
-Data-sharing attributes of variables can be classified as being \plc{predetermined},
-\plc{explicitly determined} or \plc{implicitly determined}.
+Data-sharing attributes of variables can be classified as being \emph{predetermined},
+\emph{explicitly determined} or \emph{implicitly determined}.
 
 Certain variables and objects have predetermined attributes.  
 A commonly found case is the loop iteration variable in associated loops 
-of a \kcode{for} or \kcode{do} construct. It has a private data-sharing attribute.
-Variables with predetermined data-sharing attributes cannot be listed in a data-sharing clause; but there are some
-exceptions (mainly concerning loop iteration variables).
+of a \kcode{for} or \kcode{do} construct. It has a private data-sharing
+attribute. Certain declarative directives can also be used to define variables
+as having special predetermined data-sharing attributes including \emph{threadprivate},
+\emph{groupprivate}, and \emph{device-local}. Variables with predetermined
+data-sharing attributes cannot usually be listed in a data-sharing clause, but there
+are some exceptions (mainly concerning loop iteration variables).
 
 Variables with explicitly determined data-sharing attributes are those that are
-referenced in a given construct and are listed in a data-sharing attribute
+referenced in a given construct and are listed in a data-sharing
 clause on the construct. Some of the common data-sharing clauses are:
 \kcode{shared}, \kcode{private}, \kcode{firstprivate}, \kcode{lastprivate}, 
 \kcode{linear}, and \kcode{reduction}. % Are these all of them?
@@ -38,44 +48,57 @@
 For a complete list of variables and objects with predetermined and
 implicitly determined attributes, please refer to the
 \docref{Data-sharing Attribute Rules for Variables Referenced in a Construct}
-subsection of the OpenMP Specifications document.  
+subsection of the OpenMP Specification document.  
 
 \bigskip
 DATA-MAPPING ATTRIBUTES
 
-The \kcode{map} clause on a device construct explicitly specifies how the list items in
-the clause are mapped from the encountering task's data environment (on the host)
-to the corresponding item in the device data environment (on the device).
-The common \plc{list items} are arrays, array sections, scalars, pointers, and
-structure elements (members). 
-
-Procedures and global variables have predetermined data mapping if they appear
-within the list or block of a \kcode{declare target} directive. Also, a C/C++ pointer
-is mapped as a zero-length array section, as is a C++ variable that is a reference to a pointer.
-% Waiting for response from Eric on this.
+A data-mapping attribute determines the manner in which a variable or object is
+mapped from a data environment of a task (typically on the host device) to a
+device data environment on a different device. The specification of list items
+in a \kcode{map} clause is the main mechanism for controlling the data-mapping
+attributes of data in a device data environment. These list items may include
+variables, including array and structure elements, array sections, as well as
+more general lvalue expressions in C/C++ (such as a dereferenced expression of
+pointer type).
 
-Without explicit mapping, non-scalar and non-pointer variables within the scope of the \kcode{target}
-construct are implicitly mapped with a \plc{map-type} of \kcode{tofrom}.
-Without explicit mapping, scalar variables within the scope of the \kcode{target}
-construct are not mapped, but have an implicit firstprivate data-sharing
-attribute. (That is, the value of the original variable is given to a private
-variable of the same name on the device.) This behavior can be changed with
-the \kcode{defaultmap} clause.
+If a \kcode{map} clause is not explicitly specified for a variable that is
+referenced in a \kcode{target} construct, that variable may still have an
+\emph{implicit} data-mapping attribute (as if it had appeared in a \kcode{map}
+clause). For example, the use of a declare target directive or
+\kcode{defaultmap} clause can result in a variable having an implicit
+data-mapping attribute. Additionally, list items that appear in certain
+data-sharing clauses (e.g., \kcode{reduction}) on a compound target construct
+can imply a data-mapping attribute. Also, non-scalar variables referenced
+inside a \kcode{target} construct that do not otherwise have a predetermined or
+explicit data-sharing or data-mapping attribute will typically be implicitly
+mapped by default, in contrast to scalar variables which are typically given
+an implicit firstprivate attribute (these default implicit attributes can be
+changed with the use of the \kcode{defaultmap} clause). For a complete set of
+rules for implicit data-mapping attributes, refer to the
+\docref{Implicit Data-Mapping Attribute Rules}
+subsection of the OpenMP Specification document.
 
-The \kcode{map} clause can appear on \kcode{target}, \kcode{target data} and 
-\kcode{target enter/exit data} constructs.  The operations of creation and
-removal of device storage as well as assignment of the original list item 
-values to the corresponding list items may be complicated when the list 
-item appears on multiple constructs or when the host and device storage 
-is shared. In these cases the item's reference count, the number of times
-it has been referenced (increment by 1 on entry and decrement by 1 on exit) in nested (structured)
-map regions and/or accumulative (unstructured) mappings, determines the operation.
-Details of the \kcode{map} clause and reference count operation are specified 
-in the \docref{\kcode{map} Clause} subsection of the OpenMP Specifications document.
+The \kcode{map} clause can appear on data-mapping constructs (specifically,
+\kcode{target}, \kcode{target_data}, \kcode{target_enter_data} and
+\kcode{target_exit_data}).  The operations of creation and removal of corresponding
+storage as well as assignment of the original list item values to the
+corresponding list items may be complicated when the list item appears on
+multiple constructs that are executed concurrently. To accomodate this, a
+reference count is maintained to determine which of those operations are
+needed. This can help ensure that corresponding storage is not removed on
+completion of one construct while another construct that has mapped the same
+data still requires it, as well as elide data transfers between devices in
+cases where corresponding storage is not being created or removed (though this
+can be overridden with use of modifiers such as \kcode{delete} or
+\kcode{always}). Details of the \kcode{map} clause and reference count
+operations are specified in the \docref{\kcode{map} Clause} subsection of the
+OpenMP Specification document.
 
 
 %===== Examples Sections =====
 \input{data_environment/threadprivate}
+\input{data_environment/groupprivate}
 \input{data_environment/default_none}
 \input{data_environment/private}
 \input{data_environment/fort_loopvar}