One Sided Communications
Public API
- group MPI One-Sided MPC API
API of the MPI module to perform RMA operations.
Typedefs
-
typedef struct mpc_mpi_win_s *mpc_mpi_win_h
MPI Window opaque handle.
Functions
-
int mpc_mpi_osc_win_create(void **base, const size_t size, const int disp_unit, MPI_Info info, mpc_lowcomm_communicator_t comm, mpc_mpi_win_h *win_p)
Creates a window from a user provided buffer.
- Parameters:
base – [in] Address of the underlying buffer
size – [in] Minimum size requested for the window
disp_unit – [in] Displacement unit to use in the window
info – [in] Info object to store in the window at creation
comm – [in] Communicator associated with the created window
win_p – [out] Pointer to store the resulting handle on the created window
- Returns:
MPI errorcode
-
int mpc_mpi_osc_win_allocate(void **base, const size_t size, const int disp_unit, MPI_Info info, mpc_lowcomm_communicator_t comm, mpc_mpi_win_h *win_p)
Creates a window by allocating a buffer for the user.
- Parameters:
base – [in] Address of the allocated buffer
size – [in] Minimum size requested for the window
disp_unit – [in] Displacement unit to use in the window
info – [in] Info object to store in the window at creation
comm – [in] Communicator associated with the created window
win_p – [out] Pointer to store the resulting handle on the created window
- Returns:
MPI errorcode
-
int mpc_mpi_osc_win_create_dynamic(MPI_Info info, mpc_lowcomm_communicator_t comm, mpc_mpi_win_h *win_p)
Creates an empty dynamic window.
- Parameters:
info – [in] Info object to store in the window at creation
comm – [in] Communicator associated with the created window
win_p – [out] Pointer to store the resulting handle on the created window
- Returns:
MPI errorcode
-
int mpc_mpi_osc_win_free(mpc_mpi_win_h win)
Frees a window.
- Parameters:
win – [in] Handle on the window to free
- Returns:
MPI errorcode
-
int mpc_mpi_osc_win_attach(mpc_mpi_win_h win, void *base, const size_t len)
Attaches a buffer to a dynamic window.
Warning
Trying to attach memory to a created or allocated window will fail
- Parameters:
win – [in] Handle of the window the memory will be attached to
base – [in] Address of the buffer to attach
len – [in] Size of the buffer to attach
- Returns:
MPI errorcode
-
int mpc_mpi_osc_win_detach(mpc_mpi_win_h win, const void *base)
Attaches a buffer to a dynamic window.
Warning
Trying to detach memory from a created or allocated window will fail
- Parameters:
win – [in] Handle of the window the memory will be attached to
base – [in] Address of the buffer to attach
len – [in] Size of the buffer to attach
- Returns:
MPI errorcode
-
int mpc_mpi_osc_put(const void *origin_addr, const int origin_count, mpc_lowcomm_datatype_t origin_dt, const int target, const ptrdiff_t target_disp, const int target_count, mpc_lowcomm_datatype_t target_dt, mpc_mpi_win_h win)
Performs a put one-sided operation.
- Parameters:
origin_addr – [in] Address of the buffer to read from
origin_count – [in] Number of elements to read at the origin
origin_dt – [in] Datatype of the elements to read at the origin
target – [in] Rank of the target task
target_disp – [in] Offset at the target location (multiplied by the disp_unit of the target)
target_count – [in] Number of elements to write at the target
target_dt – [in] Datatype of the elements to write at the target
win – [in] Handle of the window used for the communication
- Returns:
MPI errorcode
-
int mpc_mpi_osc_get(void *origin_addr, const int origin_count, mpc_lowcomm_datatype_t origin_dt, const int target, const ptrdiff_t target_disp, const int target_count, mpc_lowcomm_datatype_t target_dt, mpc_mpi_win_h win)
Performs a get one-sided operation.
- Parameters:
origin_addr – [in] Address of the buffer to write to
origin_count – [in] Number of elements to write at the origin
origin_dt – [in] Datatype of the elements to write at the origin
target – [in] Rank of the target task
target_disp – [in] Offset at the target location (multiplied by the disp_unit of the target)
target_count – [in] Number of elements to read at the target
target_dt – [in] Datatype of the elements to read at the target
win – [in] Handle of the window used for the communication
- Returns:
MPI errorcode
-
int mpc_mpi_osc_accumulate(const void *origin_addr, const int origin_count, mpc_lowcomm_datatype_t origin_dt, const int target, const ptrdiff_t target_disp, const int target_count, mpc_lowcomm_datatype_t target_dt, MPI_Op op, mpc_mpi_win_h win)
Performs an accumulate one-sided operation.
- Parameters:
origin_addr – [in] Address of the buffer to read from
origin_count – [in] Number of elements to read at the origin
origin_dt – [in] Datatype of the elements to read at the origin
target – [in] Rank of the target task
target_disp – [in] Offset at the target location (multiplied by the disp_unit of the target)
target_count – [in] Number of elements to write at the target
target_dt – [in] Datatype of the elements to write at the target
op – [in] Operation to execute
win – [in] Handle of the window used for the communication
- Returns:
MPI errorcode
-
int mpc_mpi_osc_get_accumulate(const void *origin_addr, const int origin_count, mpc_lowcomm_datatype_t origin_dt, void *result_addr, const int result_count, mpc_lowcomm_datatype_t result_dt, const int target_rank, const ptrdiff_t target_disp, const int target_count, mpc_lowcomm_datatype_t target_dt, MPI_Op op, mpc_mpi_win_h win)
Performs a get and accumulate one-sided operation.
- Parameters:
origin_addr – [in] Address of the buffer to read from
origin_count – [in] Number of elements to read at the origin
origin_dt – [in] Datatype of the elements to read at the origin
result_addr – [in] Address of the buffer to write to
result_count – [in] Number of elements to write at the origin
result_dt – [in] Datatype of the elements to write at the origin
target – [in] Rank of the target task
target_disp – [in] Offset at the target location (multiplied by the disp_unit of the target)
target_count – [in] Number of elements to write at the target
target_dt – [in] Datatype of the elements to write at the target
op – [in] Operation to execute
win – [in] Handle of the window used for the communication
- Returns:
MPI errorcode
-
int mpc_mpi_osc_compare_and_swap(const void *origin_addr, const void *compare_addr, void *result_addr, mpc_lowcomm_datatype_t dt, const int target, const ptrdiff_t target_disp, mpc_mpi_win_h win)
Performs an atomic compare and swap one-sided operation.
- Parameters:
origin_addr – [in] Address of the buffer to read from to replace
compare_addr – [in] Address of the buffer to read from to have the compare element
result_addr – [in] Address of the buffer to write to
dt – [in] Datatype of the element
target – [in] Rank of the target task
target_disp – [in] Offset at the target location (multiplied by the disp_unit of the target)
win – [in] Handle of the window used for the communication
- Returns:
MPI errorcode
-
int mpc_mpi_osc_fetch_and_op(const void *origin_addr, void *result_addr, mpc_lowcomm_datatype_t dt, const int target, const ptrdiff_t target_disp, MPI_Op op, mpc_mpi_win_h win)
Performs an atomic fetch and operation one-sided operation.
- Parameters:
origin_addr – [in] Address of the buffer to read from to replace
result_addr – [in] Address of the buffer to write to
dt – [in] Datatype of the element
target – [in] Rank of the target task
target_disp – [in] Offset at the target location (multiplied by the disp_unit of the target)
op – [in] Operation to execute
win – [in] Handle of the window used for the communication
- Returns:
MPI errorcode
-
int mpc_mpi_osc_rput(const void *origin_addr, const int origin_count, mpc_lowcomm_datatype_t origin_dt, const int target, const ptrdiff_t target_disp, const int target_count, mpc_lowcomm_datatype_t target_dt, mpc_mpi_win_h win, MPI_internal_request_t **req)
Performs a put one-sided operation with a request to be completed.
- Parameters:
origin_addr – [in] Address of the buffer to read from
origin_count – [in] Number of elements to read at the origin
origin_dt – [in] Datatype of the elements to read at the origin
target – [in] Rank of the target task
target_disp – [in] Offset at the target location (multiplied by the disp_unit of the target)
target_count – [in] Number of elements to write at the target
target_dt – [in] Datatype of the elements to write at the target
win – [in] Handle of the window used for the communication
req – [out] Request associated with the communication
- Returns:
MPI errorcode
-
int mpc_mpi_osc_rget(void *origin_addr, const int origin_count, mpc_lowcomm_datatype_t origin_dt, const int target, const ptrdiff_t target_disp, const int target_count, mpc_lowcomm_datatype_t target_dt, mpc_mpi_win_h win, MPI_internal_request_t **req)
Performs a get one-sided operation with a request to be completed.
- Parameters:
origin_addr – [in] Address of the buffer to write to
origin_count – [in] Number of elements to write at the origin
origin_dt – [in] Datatype of the elements to write at the origin
target – [in] Rank of the target task
target_disp – [in] Offset at the target location (multiplied by the disp_unit of the target)
target_count – [in] Number of elements to read at the target
target_dt – [in] Datatype of the elements to read at the target
win – [in] Handle of the window used for the communication
req – [out] Request associated with the communication
- Returns:
MPI errorcode
-
int mpc_mpi_osc_raccumulate(const void *origin_addr, const int origin_count, mpc_lowcomm_datatype_t origin_dt, const int target, const ptrdiff_t target_disp, const int target_count, mpc_lowcomm_datatype_t target_dt, MPI_Op op, mpc_mpi_win_h win, MPI_internal_request_t **request)
Performs an accumulate one-sided operation with a request to be completed.
- Parameters:
origin_addr – [in] Address of the buffer to read from
origin_count – [in] Number of elements to read at the origin
origin_dt – [in] Datatype of the elements to read at the origin
target – [in] Rank of the target task
target_disp – [in] Offset at the target location (multiplied by the disp_unit of the target)
target_count – [in] Number of elements to write at the target
target_dt – [in] Datatype of the elements to write at the target
op – [in] Operation to execute
win – [in] Handle of the window used for the communication
req – [out] Request associated with the communication
- Returns:
MPI errorcode
-
int mpc_mpi_osc_rget_accumulate(const void *origin_addr, const int origin_count, mpc_lowcomm_datatype_t origin_dt, void *result_addr, const int result_count, mpc_lowcomm_datatype_t result_dt, const int target_rank, const ptrdiff_t target_disp, const int target_count, mpc_lowcomm_datatype_t target_dt, MPI_Op op, mpc_mpi_win_h win, MPI_internal_request_t **request)
Performs a get and accumulate one-sided operation with a request to be completed.
- Parameters:
origin_addr – [in] Address of the buffer to read from
origin_count – [in] Number of elements to read at the origin
origin_dt – [in] Datatype of the elements to read at the origin
result_addr – [in] Address of the buffer to write to
result_count – [in] Number of elements to write at the origin
result_dt – [in] Datatype of the elements to write at the origin
target – [in] Rank of the target task
target_disp – [in] Offset at the target location (multiplied by the disp_unit of the target)
target_count – [in] Number of elements to write at the target
target_dt – [in] Datatype of the elements to write at the target
op – [in] Operation to execute
win – [in] Handle of the window used for the communication
req – [out] Request associated with the communication
- Returns:
MPI errorcode
-
int mpc_mpi_osc_lock(const int lock_type, const int target, const int mpi_assert, mpc_mpi_win_h win)
Locks a window either to be share or to have exclusive access.
- Parameters:
lock_type – [in] Type of locking to perform (SHARED or EXCLUSIVE)
target – [in] Rank of the target task
mpi_assert – [in] Special bitfield to atler the lock behaviour
win – [in] Handle of the window used
- Returns:
MPI errorcode
-
int mpc_mpi_osc_unlock(const int target, mpc_mpi_win_h win)
Unlocks a window previously locked.
- Parameters:
target – [in] Rank of the target task
win – [in] Handle of the window used
- Returns:
MPI errorcode
-
int mpc_mpi_osc_lock_all(const int mpi_assert, mpc_mpi_win_h win)
Locks all the target in the window for shared access.
- Parameters:
mpi_assert – [in] Special bitfield to atler the lock behaviour
win – [in] Handle of the window used
- Returns:
MPI errorcode
-
int mpc_mpi_osc_unlock_all(mpc_mpi_win_h win)
Unlocks all the target in the window.
Note
The window should be locked with mpc_mpi_osc_lock_all
- Parameters:
win – [in] Handle of the window used
- Returns:
MPI errorcode
-
int mpc_mpi_osc_sync(mpc_mpi_win_h win)
Synchronizes the outstanding communications with the network.
- Parameters:
win – [in] Handle of the window used
- Returns:
MPI errorcode
-
int mpc_mpi_osc_flush(const int target, mpc_mpi_win_h win)
Flushes the outstanding communications.
Note
This routine assess local and distant completion of the communications upon return
- Parameters:
target – [in] Rank of the target task
win – [in] Handle of the window used
- Returns:
MPI errorcode
-
int mpc_mpi_osc_flush_all(mpc_mpi_win_h win)
Flushes the outstanding communications for all targets in the window.
Note
This routine assess local and distant completion of the communications upon return
- Parameters:
win – [in] Handle of the window used
- Returns:
MPI errorcode
-
int mpc_mpi_osc_fence(const int mpi_assert, mpc_mpi_win_h win)
Synchronizes the target in the window.
Note
This routine assess local and distant completion of the communications upon return
- Parameters:
mpi_assert – [in] Special bitfield to atler the lock behaviour
win – [in] Handle of the window used
- Returns:
MPI errorcode
-
int mpc_mpi_osc_start(mpc_lowcomm_group_t *group, const int mpi_assert, mpc_mpi_win_h win)
Opens an access epoch on the specified group.
- Parameters:
group – [in] Group on which the access epoch will be opened
mpi_assert – [in] Special bitfield to atler the lock behaviour
win – [in] Handle of the window used
- Returns:
MPI errorcode
-
int mpc_mpi_osc_complete(mpc_mpi_win_h win)
Closes an access epoch on the specified group.
Note
This routine assess local completion of the communications upon return
- Parameters:
win – [in] Handle of the window used
- Returns:
MPI errorcode
-
int mpc_mpi_osc_post(mpc_lowcomm_group_t *group, const int mpi_assert, mpc_mpi_win_h win)
Opens an exposure epoch on the specified group.
- Parameters:
group – [in] Group on which the exposure epoch will be opened
mpi_assert – [in] Special bitfield to atler the lock behaviour
win – [in] Handle of the window used
- Returns:
MPI errorcode
-
int mpc_mpi_osc_wait(mpc_mpi_win_h win)
Closes an exposure epoch on the specified group.
Note
This routine assess distant completion of the communications upon return
- Parameters:
win – [in] Handle of the window used
- Returns:
MPI errorcode
-
int mpc_mpi_osc_test(mpc_mpi_win_h win, int *flag)
Tries to close an exposure epoch on the specified group.
- Parameters:
win – [in] Handle of the window used
flag – [out] Success status (1 if the epoch was closed)
- Returns:
MPI errorcode
-
MPI_Comm mpc_mpi_osc_win_get_comm(const mpc_mpi_win_h win)
Retrieves the communicator associated with a win.
- Parameters:
win – [in] Handle of the window to retrieve from
- Returns:
A reference on the communicator
-
MPI_Group mpc_mpi_osc_win_get_group(const mpc_mpi_win_h win)
Retrieves the group associated with a win.
- Parameters:
win – [in] Handle of the window to retrieve from
- Returns:
A copy of the group
-
typedef struct mpc_mpi_win_s *mpc_mpi_win_h
Internal API
- group Internal implementation of MPI One-Sided
Internals of the MPI module to perform RMA operations.
Defines
Typedefs
-
typedef enum mpc_mpi_osc_epoch_e mpc_mpi_osc_epoch_t
Possible epoch (access and exposure)
-
typedef struct mpc_mpi_osc_epoch_type_s mpc_mpi_osc_epoch_type_t
Structure keeping the epoch state for both access and exposure.
-
typedef struct mpc_mpi_win_s mpc_mpi_win_t
Internal structure of an MPI window.
Enums
-
enum mpc_mpi_osc_epoch_e
Possible epoch (access and exposure)
Values:
-
enumerator NONE_EPOCH
No epoch opened.
-
enumerator FENCE_EPOCH
Fence epoch opened.
-
enumerator POST_WAIT_EPOCH
Post wait epoch opened (between a post and a wait)
-
enumerator START_COMPLETE_EPOCH
Start complete epoch opened (between a start and a complete)
-
enumerator PASSIVE_EPOCH
Passive epoch opened (between lock and unlock)
-
enumerator PASSIVE_ALL_EPOCH
Passive all epoch opened (between lock_all and unlock_all)
-
enumerator NONE_EPOCH
Functions
-
static void __osc_min_op_long(const void *in, void *out, size_t length, mpc_lowcomm_datatype_t dt)
Reduction operation performing a min on long int.
This function is used to check the difference in displacement unit in the window creation across the tasks.
- Parameters:
in – [in] Array of the reference longs
out – [out] Arary of long to be compared
length – [in] Size (in elements) of in and out
dt – [in] Datatype of the arrays (UNUSED)
-
static int __osc_win_setup_after_creation(mpc_mpi_win_t *win, const size_t size, int disp_unit, MPI_Info info, const int flavor)
Setup the MPI specific operations on the MPI Window after LowComm window creation.
This function setup the MPI specific objects and characteristics (MPI_Info, epoch, disp_unit, flavor, model…). The most significant part is the gathering of all the disp_unit of the tasks.
- Parameters:
win – [out] MPI Window to fill
size – [in] Size of the Window underlying buffer
disp_unit – [in] Displacement unit of the current task for this window
info – [in] MPI Info object to associate with the newly created window
flavor – [in] MPI Creation Flavor of the window
- Returns:
MPI errorcode
-
static int __osc_create_dt_blocks(const void *const typebuf, const int count, const mpc_lowcomm_datatype_t dt, int **blocks_p, intptr_t **disps_p, int *intblockct_p)
Creates contiguous data blocks from a non-contiguous datatype.
A non-contiguous datatype can be describe as contiguous blocks of various sizes displaced from one another.
- Parameters:
typebuf – [in] Buffer containing the data of type dt
count – [in] Number of element of type dt in typebuf
dt – [in] Datype of the elements in typebuf
blocks_p – [out] Pointer to store an array of block size
disps_p – [out] Pointer to store an array of displacement for the blocks
intblockct_p – [out] Pointer to store the number of blocks
- Returns:
MPI errorcode
-
static int __osc_discontig_common(void *origin_addr, const int origin_count, const mpc_lowcomm_datatype_t origin_dt, const bool is_orig_dt_contig, const int target, const uint64_t target_disp, const int target_count, const mpc_lowcomm_datatype_t target_dt, const bool is_target_dt_contig, const bool do_get, mpc_lowcomm_window_t *win, mpc_lowcomm_request_t *request)
Perform a RMA operation on a non-contiguous datatype.
This function handles both put and get type of RMA.
- Parameters:
origin_addr – [in] Buffer address at the origin
origin_count – [in] Number of elements in origin_addr
origin_dt – [in] Datatype of the elements in origin_addr
is_orig_dt_contig – [in] Boolean on whether the origin datatype is contiguous or not
target – [in] Rank of the target task
target_disp – [in] Remote offset at the target
target_count – [in] Number of elements to put/get from the target
target_dt – [in] Datatype of the elements at the target
is_target_dt_contig – [in] Boolean on whether the origin datatype is contiguous or not
do_get – [in] Whether to perform a get (true) or a put (false)
win – [in] Window on which the operation is performed
request – [out] Request associated with the ongoing communication
- Returns:
MPI errorcode
-
static int __osc_put_common(const void *const origin_addr, const int origin_count, mpc_lowcomm_datatype_t origin_dt, const int target, const ptrdiff_t target_disp, const int target_count, mpc_lowcomm_datatype_t target_dt, mpc_mpi_win_t *win, mpc_lowcomm_request_t *request)
Common entry point to perform a put operation.
This function unwraps the datatypes and compute the correct offset based on the datatype lb and target_disp. It performs a put directly if both datatypes are contiguous otherwise it relies on mpc_osc_discontig_common.
- Parameters:
origin_addr – [in] Buffer address at the origin
origin_count – [in] Number of elements in origin_addr
origin_dt – [in] Datatype of the elements in origin_addr
target – [in] Rank of the target task
target_disp – [in] Remote offset at the target
target_count – [in] Number of elements to put/get from the target
target_dt – [in] Datatype of the elements at the target
win – [in] Window on which the operation is performed
request – [out] Request associated with the ongoing communication
- Returns:
MPI errorcode
-
static int __osc_get_common(void *origin_addr, const int origin_count, mpc_lowcomm_datatype_t origin_dt, const int target, const ptrdiff_t target_disp, const int target_count, mpc_lowcomm_datatype_t target_dt, mpc_mpi_win_t *win, mpc_lowcomm_request_t *request)
Common entry point to perform a get operation.
This function unwraps the datatypes and compute the correct offset based on the datatype lb and target_disp. It performs a get directly if both datatypes are contiguous otherwise it relies on mpc_osc_discontig_common.
- Parameters:
origin_addr – [in] Buffer address at the origin
origin_count – [in] Number of elements in origin_addr
origin_dt – [in] Datatype of the elements in origin_addr
target – [in] Rank of the target task
target_disp – [in] Remote offset at the target
target_count – [in] Number of elements to put/get from the target
target_dt – [in] Datatype of the elements at the target
win – [in] Window on which the operation is performed
request – [out] Request associated with the ongoing communication
- Returns:
MPI errorcode
-
int osc_win_attr_ht_release(struct mpc_common_hashtable *atht)
Release the attributes hashtable of a window.
- Parameters:
atht – [in] Hashtable to release
- Returns:
MPI errorcode
-
static inline void __osc_win_keyval_ht_init_once()
Initiates the global keyval hashtable is not done already.
-
static inline uint64_t __osc_get_per_rank_keyval_key(const int keyval)
Retrieves the per rank keyval.
- Parameters:
keyval – [in] Global keyval identifier
- Returns:
Keyval for the current task
-
static struct osc_win_keyval_s *__osc_win_keyval_init(MPI_Win_copy_attr_function *copy_fn, MPI_Win_delete_attr_function *delete_fn, void *extra_state)
Allocates and initializes a new keyval.
- Parameters:
copy_fn – [in] Copy function to associate with the keyval
delete_fn – [in] Deletion function to associate with the keyval
extra_state – [in] Additional information stored in the keyval
- Returns:
Allocated and initialized keyval
-
static struct osc_win_attr_s *__osc_win_attr_init(int keyval, void *value, struct osc_win_keyval_s *keyval_pl, mpc_mpi_win_t *win)
Allocates and initializes a new attribute.
- Parameters:
keyval – [in] Unique identifier of the attribute
value – [in] Value stored in the attribute
keyval_pl – [in] Keyval used by this attribute
win – [in] Window associated with the attribute
- Returns:
Allocated and initialized attribute
Variables
-
static struct mpc_common_hashtable __osc_win_keyval_ht
Shared keyval hashtable.
-
static mpc_common_spinlock_t __osc_win_keyval_ht_lock = MPC_COMMON_SPINLOCK_INITIALIZER
Global keyval hashtable lock.
-
static int __osc_win_keyval_ht_init_done = 0
State of the keyval hashtable (1 for initialized)
-
static OPA_int_t __win_attr_counter
Global attributes counter.
-
struct mpc_mpi_osc_epoch_type_s
Structure keeping the epoch state for both access and exposure.
Public Members
-
mpc_mpi_osc_epoch_t access
Epoch state for window access.
-
mpc_mpi_osc_epoch_t exposure
Epoch state for window exposure.
-
mpc_mpi_osc_epoch_t access
-
struct mpc_mpi_win_s
Internal structure of an MPI window.
Public Members
-
mpc_lowcomm_window_t *win_module
Lowcomm window.
-
char win_name[MPI_MAX_OBJECT_NAME]
Name of the window.
-
int flavor
Creation flavor of the window.
-
int model
Model of the window.
-
size_t size
Size of the underlying buffer.
-
mpc_mpi_osc_epoch_type_t epoch
Epoch states for access and exposure.
-
OPA_int_t lock_count
Number of lock on the window.
-
int disp_unit
Displacement unit if it is the same across tasks.
-
int *disp_units
Displacement units of the all the tasks.
-
struct mpc_common_hashtable attrs
Attributes of the window.
-
mpc_lowcomm_window_t *win_module
-
struct osc_win_keyval_s
Internal structure of an osc keyval.
-
struct osc_win_attr_s
Internal structure of an osc attribute.
Public Members
-
int keyval
Unique identifier of the attribute.
-
void *value
Information stored in the attribute.
-
struct osc_win_keyval_s keyval_pl
Associated keyval structure.
-
mpc_mpi_win_t *win
Associated window.
-
int keyval
-
typedef enum mpc_mpi_osc_epoch_e mpc_mpi_osc_epoch_t