One Sided communication

LowComm One-Sided communications

SUMMARY:

Here are described the internal of the One-Sided communications Lowcomm top level API.

CONTENTS:

lowcomm_one_sided.c : Communication routines
lowcomm_one_sided_common.c : Common routines within this submodule
lowcomm_one_sided_helpers.c : Getter routines
lowcomm_one_sided_internal.h : Definition of all the internal structure and common routines
lowcomm_one_sided_sync.c : Synchronization routines
lowcomm_one_sided_window.c : Window management routines

This representation is temporary before the integration of UBCL

Public API

group MPC LowComm one sided communications API

Typedefs

typedef struct mpc_lowcomm_window_s mpc_lowcomm_window_t: Opaque object representing a window.

typedef enum mpc_lowcomm_window_flag_e mpc_lowcomm_window_flag_t: Window creation flags.

typedef enum mpc_lowcomm_atomic_op_e mpc_lowcomm_atomic_op_t: LOWCOMM atomics operation identifiers.

typedef enum mpc_lowcomm_accumulate_op_e mpc_lowcomm_accumulate_op_t: LOWCOMM accumulate operation identifiers.

typedef struct mpc_lowcomm_osc_win_attr_s mpc_lowcomm_osc_win_attr_t: Attributes of a window that can be retrieved.

Enums

enum mpc_lowcomm_window_flag_e

Window creation flags.

Values:

enumerator MPC_LOWCOMM_WINDOW_FLAG_CREATE: Create with a pre existing buffer.

enumerator MPC_LOWCOMM_WINDOW_FLAG_ALLOCATE: Create by allocating a buffer.

enumerator MPC_LOWCOMM_WINDOW_FLAG_DYNAMIC: Create a dynamic window.

enumerator MPC_LOWCOMM_WINDOW_FLAG_SHARED: Create a window shared in memory (implemented)

enum mpc_lowcomm_atomic_op_e

LOWCOMM atomics operation identifiers.

Values:

enumerator LOWCOMM_ATOMIC_OP_FETCH_ADD: Atomic addition.

enumerator LOWCOMM_ATOMIC_OP_FETCH_SUB: Atomic subtraction.

enumerator LOWCOMM_ATOMIC_OP_FETCH_AND: Atomic bitwise and.

enumerator LOWCOMM_ATOMIC_OP_FETCH_OR: Atomic bitwise or.

enumerator LOWCOMM_ATOMIC_OP_FETCH_XOR: Atomic bitwise xor.

enumerator LOWCOMM_ATOMIC_OP_SWAP: Atomic swap.

enumerator LOWCOMM_ATOMIC_OP_CSWAP: Atomic compare and swap.

enum mpc_lowcomm_accumulate_op_e

LOWCOMM accumulate operation identifiers.

Values:

enumerator LOWCOMM_ACCUMULATE_REPLACE: Replace the target buffer with the origin buffer.

enumerator LOWCOMM_ACCUMULATE_MAX: Element wise max operation.

enumerator LOWCOMM_ACCUMULATE_MIN: Element wise min operation.

enumerator LOWCOMM_ACCUMULATE_SUM: Element wise addition operation.

enumerator LOWCOMM_ACCUMULATE_PROD: Element wise product operation.

enumerator LOWCOMM_ACCUMULATE_LAND: Element wise logical AND operation.

enumerator LOWCOMM_ACCUMULATE_BAND: Element wise bitwise AND operation.

enumerator LOWCOMM_ACCUMULATE_LOR: Element wise logical OR operation.

enumerator LOWCOMM_ACCUMULATE_BOR: Element wise bitwise OR operation.

enumerator LOWCOMM_ACCUMULATE_LXOR: Element wise logical XOR operation.

enumerator LOWCOMM_ACCUMULATE_BXOR: Element wise bitwise XOR operation.

enumerator LOWCOMM_ACCUMULATE_UNSUPPORTED: Unsupported operation (used for error handling)

enum mpc_lowcomm_osc_lock_type_t

LOWCOMM Lock types.

Values:

enumerator MPC_LOWCOMM_WINDOW_LOCK_EXCLUSIVE = 301: Exclusive access to distant memory.

enumerator MPC_LOWCOMM_WINDOW_LOCK_SHARED = 302: Shared access to distant memory.

enumerator MPC_LOWCOMM_WINDOW_LOCK_ATOMIC = 666: For internal use.

Functions

int mpc_lowcomm_window_create(void **buffer, const size_t size, mpc_lowcomm_communicator_t comm, mpc_lowcomm_window_flag_t flag, mpc_lowcomm_window_t **window_p)

Creates a window for one sided communications use.

Parameters:

buffer – [in] Underlying buffer of the window It will be set when allocating and unused for dynamic windows
size – [in] Size (in Bytes) of the requested window (unused for dynamic windows)
comm – [in] Comunicator on which the one sided operations will happen
flag – [in] Creation flag to modify the behaviour
window_p – [out] Pointer to store an handle on the created window

Return values:

MPC_LOWCOMM_SUCCESS – on success
an – errorcode otherwise

int mpc_lowcomm_window_free(mpc_lowcomm_window_t **window_p)

Frees a window.

The handle is set to NULL upon success to avoid any misusage of a released window

Parameters:

window_p – [in] Reference on the window handle

Returns:

MPC_LOWCOMM_SUCCESS upon success

int mpc_lowcomm_window_attach(mpc_lowcomm_window_t *window, void *base, const size_t len)

Attaches a buffer to a dynamic window.

Warning

Attaching a buffer to any other type of window is erroneous

Parameters:

window – [in] Handle on the window on which the memory should be attached
base – [in] Address of the buffer to attach
len – [in] Size (in Bytes) of the buffer to attach

Returns:

MPC_LOWCOMM_SUCCESS upon success

int mpc_lowcomm_window_detach(mpc_lowcomm_window_t *window, const void *base)

Detaches a buffer from a dynamic window.

Warning

Detaching a buffer to any other type of window is erroneous

Parameters:

window – [in] Handle on the window from which the memory should be detached
base – [in] Address of the buffer to detach

Returns:

MPC_LOWCOMM_SUCCESS upon success

int mpc_lowcomm_iput(const void *const origin_buffer, const int size, const int target, const intptr_t offset, mpc_lowcomm_window_t *win, mpc_lowcomm_request_t *const req)

Register an non blocking put communication.

If the request is set to MPC_REQUEST_NULL, the completion can be checked with synchronisations only. Otherwise, the req should have been allocated used mpc_lowcomm_request_alloc.

Note

Completion is assessed either by a synchronization routine or by waiting the request

Parameters:

origin_buffer – [in] Address of the buffer to read from
size – [in] Size (in Bytes) of the data to read
target – [in] Rank of the target task
offset – [in] Offset (in Bytes) in the target window to write to
window – [in] Handle on the window to use
req – [out] Request tracking the completion of the operation

Returns:

MPC_LOWCOMM_SUCCESS upon success

int mpc_lowcomm_iget(void *origin_buffer, const int size, const int target, const intptr_t offset, mpc_lowcomm_window_t *win, mpc_lowcomm_request_t *const req)

Register an non blocking get communication.

If the request is set to MPC_REQUEST_NULL, the completion can be checked with synchronisations only. Otherwise, the req should have been allocated used mpc_lowcomm_request_alloc.

Note

Completion is assessed either by a synchronization routine or by waiting the request

Parameters:

origin_buffer – [in] Address of the buffer to write to
size – [in] Size (in Bytes) of the data to write
target – [in] Rank of the target task
offset – [in] Offset (in Bytes) in the target window to read from
window – [in] Handle on the window to use
req – [out] Request tracking the completion of the operation

Returns:

MPC_LOWCOMM_SUCCESS upon success

bool mpc_lowcomm_atomic_dt_is_compatible(const mpc_lowcomm_datatype_t datatype)

Test if a MPC datatype is compatible with atomic operations.

Parameters:

datatype – [in] Datatype to test

Return values:

true – if the datatype is compatible
false – if the datatype is not compatible

int mpc_lowcomm_iatomic(const void *const origin_buffer, void *fetch_buffer, const void *const compare_buffer, const mpc_lowcomm_datatype_t datatype, const int target, const intptr_t offset, const mpc_lowcomm_atomic_op_t op, mpc_lowcomm_window_t *win, mpc_lowcomm_request_t *req)

Performs an distant atomic operation.

The type of operation and the argument usage is decided by the op argument. For more information about the available operations see mpc_lowcomm_atomic_op_t.

Note

Completion is assessed either by a synchronization routine or by waiting the request

Parameters:

origin_buffer – [in] Address of the buffer to read from
fetch_buffer – [in] Address of the buffer to write to
compare_buffer – [in] Address of the buffer containing the operand for comparison (used for CSWAP)
datatype – [in] Datatype of the element communicated
target – [in] Rank of the target task
offset – [in] Offset (in Bytes) in the target window to read from
op – [in] Operation identifier (see mpc_lowcomm_atomic_op_t)
window – [in] Handle on the window to use
req – [out] Request tracking the completion of the operation

Returns:

Lowcomm errorcode

int mpc_lowcomm_iaccumulate(const void *const origin_buffer, void *fetch_buffer, const size_t count, const mpc_lowcomm_datatype_t datatype, const int target, const intptr_t offset, const mpc_lowcomm_accumulate_op_t op, mpc_lowcomm_window_t *win, mpc_lowcomm_request_t *req)

Performs an distant accumulate operation.

The type of operation and the argument usage is decided by the op argument. For more information about the available operations see mpc_lowcomm_accumulate_op_t.

Note

Completion is assessed either by a synchronization routine or by waiting the request

Warning

Depending on the backend this routine may be unsupported. In such cases, it will return MPC_LOWCOMM_NOT_SUPPORTED.

Parameters:

origin_buffer – [in] Address of the buffer to read from
fetch_buffer – [in] Address of the buffer to write to (NULL for accumulate without get)
count – [in] Number of elements of type datatype
datatype – [in] Datatype of the element communicated
target – [in] Rank of the target task
offset – [in] Offset (in Bytes) in the target window to read from
op – [in] Operation identifier (see mpc_lowcomm_accumulate_op_t)
window – [in] Handle on the window to use
req – [out] Request tracking the completion of the operation

Returns:

Lowcomm errorcode

int mpc_lowcomm_osc_lock(const mpc_lowcomm_osc_lock_type_t lock, const int target, mpc_lowcomm_window_t *win, const bool is_no_check)

Locks a target window.

Parameters:

lock – [in] Type of lock to perform (see @mpc_lowcomm_osc_lock_type_t)
target – [in] Rank of the target task
win – [in] Handle on the window to lock
is_no_check – [in] Whether to perform an actual lock or not

Return values:

MPC_LOWCOMM_SUCCESS – upon success
MPC_LOWCOMM_OSC_ALREADY_LOCKED – if the lock is already taken
An – error code otherwise

int mpc_lowcomm_osc_unlock(const int target_rank, mpc_lowcomm_window_t *win, const bool is_atomic)

Unlocks a target window.

Parameters:

target – [in] Rank of the target task
win – [in] Handle on the window to unlock
is_atomic – [in] Whether the lock is of atomic type (special case)

Return values:

MPC_LOWCOMM_SUCCESS – upon success
An – error code otherwise

int mpc_lowcomm_osc_lock_all(mpc_lowcomm_window_t *win, const bool is_no_check)

Locks all windows.

The definition of “all” is based on the communicator embedded in the window.

Parameters:

win – [in] Handle on the window
is_no_check – [in] Whether to perform an actual lock or not

Returns:

MPC_LOWCOMM_SUCCESS upon success

int mpc_lowcomm_osc_unlock_all(mpc_lowcomm_window_t *win)

Unlocks all windows.

The definition of “all” is based on the communicator embedded in the window.

Parameters:

win – [in] Handle on the window

Returns:

MPC_LOWCOMM_SUCCESS upon success

int mpc_lowcomm_osc_sync(mpc_lowcomm_window_t *win)

Synchronises the outstanding communications with the network.

Parameters:

win – [in] Handle on the window to sync

Returns:

MPC_LOWCOMM_SUCCESS upon success

int mpc_lowcomm_osc_start(mpc_lowcomm_group_t *group, mpc_lowcomm_window_t *win, const bool is_no_check)

Opens the window access of the specified group.

Parameters:

group – [in] Group to get access to
win – [in] Handle on the window
is_no_check – [in] Whether to perform an actual lock or not

Returns:

MPC_LOWCOMM_SUCCESS upon success

int mpc_lowcomm_osc_post (mpc_lowcomm_group_t *group, mpc_lowcomm_window_t *win, __UNUSED__ const bool is_no_check)

Opens the window exposure to the specified group.

Parameters:

group – [in] Group to provide access to
win – [in] Handle on the window
is_no_check – [in] Whether to perform an actual lock or not (UNUSED)

Returns:

MPC_LOWCOMM_SUCCESS upon success

int mpc_lowcomm_osc_complete(mpc_lowcomm_window_t *win)

Closes the access to the windows of the specified group.

Parameters:

win – [in] Handle on the window

Returns:

MPC_LOWCOMM_SUCCESS upon success

int mpc_lowcomm_osc_wait(mpc_lowcomm_window_t *win)

Waits for outstanding communication and Closes the exposure.

Parameters:

win – [in] Handle on the window

Returns:

MPC_LOWCOMM_SUCCESS upon success

int mpc_lowcomm_osc_test(mpc_lowcomm_window_t *win, bool *is_completed)

Looks for outstanding communication and Closes the exposure if none are left.

Parameters:

win – [in] Handle on the window
is_completed – [out] Whether all the communications are completed and the exposure closed

Returns:

MPC_LOWCOMM_SUCCESS upon success

int mpc_lowcomm_osc_request_complete(const int status, mpc_lowcomm_request_t *request, const size_t length)

Completes a request given by any of the communication or synchronization routine.

This is useful to mimic a meta communication and still use the same type of request

Parameters:

status – [in] Status on which the request is completed
request – [in] Request to complete
length – [in] Length of the message mimicked, written in request for future retrieving

Returns:

MPC_LOWCOMM_SUCCESS upon success

mpc_lowcomm_communicator_t mpc_lowcomm_osc_win_get_comm(const mpc_lowcomm_window_t *const win)

Retrieves the communicator embedded in the window.

Parameters:

win – [in] Window from which the communicator should be retrieved

Returns:

A reference on the communicator

mpc_lowcomm_group_t *mpc_lowcomm_osc_win_get_group(const mpc_lowcomm_window_t *const win)

Retrieves the group embedded in the window.

Parameters:

win – [in] Window from which the group should be retrieved

Returns:

A copy of the group

mpc_lowcomm_osc_win_attr_t mpc_lowcomm_osc_win_get_attr(const mpc_lowcomm_window_t *const win)

Retrieves the attributes of a window.

Parameters:

win – [in] Window from which the attributes should be retrieved

Returns:

A copy of the attributes

int lowcomm_osc_init(void)

Initializes the one-sided part of lowcomm.

Returns:: An errorcode

int lowcomm_osc_deinit(void)

Deinitializes the one-sided part of lowcomm.

Returns:: An errorcode

Variables

static const int MPC_LOWCOMM_OSC_ALREADY_LOCKED = 15990: Special value to significate that a window is already locked.

Note

It do not have to be an error

struct mpc_lowcomm_osc_win_attr_s

Attributes of a window that can be retrieved.

Public Members

void *base_data: Address of the underlying buffer.

Internal API

group Internal LowComm one sided API

Internals of the LowComm one sided communication API.

Defines

OSC_POST_PEER_MAX 32: Maximum number of peers in an active target mode.

OSC_LOCK_UNLOCK 0x0000000000000000ull: Special value for an unlocked window.

OSC_LOCK_EXCLUSIVE 0x0000000100000000ull: Special value for an exclusivity locked window.

OCS_LCP_RKEY_BUF_SIZE_MAX 1024: Maximum Size of a remote memory key.

OSC_DYNAMIC_WIN_ATTACH_MAX 128: Maximum number of memory attached to a dynamic window.

OSC_STATE_COMPLETION_COUNTER_OFFSET 0: Offset in the state part of the window of the completion counter.

OSC_STATE_POST_COUNTER_OFFSET (OSC_STATE_COMPLETION_COUNTER_OFFSET + sizeof(uint64_t)): Offset in the state part of the window of the post counter.

OSC_STATE_POST_OFFSET (OSC_STATE_POST_COUNTER_OFFSET + sizeof(uint64_t)): Offset in the state part of the window of the posts.

OSC_STATE_ACC_LOCK_OFFSET (OSC_STATE_POST_OFFSET + (OSC_POST_PEER_MAX * sizeof(uint64_t))): Offset in the state part of the window of the accumulate lock.

OSC_STATE_GLOBAL_LOCK_OFFSET (OSC_STATE_ACC_LOCK_OFFSET + sizeof(uint64_t)): Offset in the state part of the window of the lock.

OSC_STATE_REGION_LOCK_OFFSET (OSC_STATE_GLOBAL_LOCK_OFFSET + sizeof(uint64_t)): Offset in the state part of the window of the local lock.

OSC_STATE_DYNAMIC_WIN_COUNT_OFFSET (OSC_STATE_REGION_LOCK_OFFSET + sizeof(uint64_t)): Offset in the state part of the window of the dynamic attached memories count.

OSC_STATE_DYNAMIC_WIN_OFFSET (OSC_STATE_DYNAMIC_WIN_COUNT_OFFSET + sizeof(uint64_t)): Offset in the state part of the window of the dynamic attached memories array.

Typedefs

typedef struct _lowcomm_osc_dynamic_win_s _lowcomm_osc_dynamic_win_t: Dynamic memory description.

typedef struct _lowcomm_osc_module_state_s _lowcomm_osc_module_state_t: Synchronization states specific values.

typedef struct _lowcomm_osc_win_info_s _lowcomm_osc_win_info_t: Description of a window.

typedef struct _lowcomm_osc_pending_post_s _lowcomm_osc_pending_post_t: Description of a pending post operation.

typedef struct _lowcomm_osc_local_dynamic_win_info_s _lowcomm_osc_local_dynamic_win_info_t: Description of an attached memory region.

typedef struct _lowcomm_osc_lock_s _lowcomm_osc_lock_t: Description of a distant lock.

typedef struct mpc_lowcomm_window_s mpc_lowcomm_window_t: Internal structure of a LCP backed window.

Functions

static int __osc_common_get_dynamic_win_info(lcp_task_h task, const int target, const uint64_t remote_addr, mpc_lowcomm_window_t *win, lcp_mem_h *rmem_p)

Get information about a dynamic window from a distant target.

Parameters:

task – [in] Current LCP task
target – [in] Rank of the target task
remote_addr – [in] Address of the buffer in case of dynamic window
win – [in] Window on which the communication happens
rmem_p – [out] Retrieved remote memory

Returns:

MPC_LOWCOMM_SUCCESS upon success

int _lowcomm_osc_perform_atomic(lcp_task_h task, lcp_ep_h ep, const uint64_t value, uint64_t *result, const uint64_t remote_addr, const lcp_mem_h rkey, const lcp_atomic_dt_t atomic_datatype, const lcp_atomic_op_t op, mpc_lowcomm_window_t *win)

Perform a blocking atomic operation assessing the completion at return.

Parameters:

task – [in] Current LCP task
ep – [in] Endpoint to flush
value – [in] Element to write in the distant memory
result – [out] Pointer to store the fetched value (should contain the compare element for cswap)
remote_addr – [in] Address at the target where the operation should happen
rkey – [in] Remote key of the target memory
atomic_datatype – [in] datatype of the element (see lcp_atomic_dt_t)
op – [in] Atomic operation to perform (see lcp_atomic_op_t)
win – [in] Window on which the communication happens

Returns:

MPC_LOWCOMM_SUCCESS upon success

int _lowcomm_osc_perform_flush(lcp_task_h task, lcp_ep_h ep, lcp_mem_h mem, mpc_lowcomm_window_t *win)

Perform a blocking flush assessing all the communication are completed.

If the ep is NULL then, all endpoints will be flushed. If the mem is NULL then, all memories will be flushed. If both are NULL, the entire manager will be flushed (all endpoints and all memories).

Parameters:

task – [in] Current LCP task
ep – [in] Endpoint to flush
mem – [in] Memory to flush
win – [in] Window on which the flush happens

Returns:

MPC_LOWCOMM_SUCCESS upon success

void _lowcomm_osc_get_target_ep(const int target, mpc_lowcomm_window_t *win, lcp_ep_h *ep_p)

Retrieves the LCP endpoint to the target in the window.

Parameters:

target – [in] Rank of the target task
win – [in] Window from which the endpoint should be retrieved
ep_p – [out] Pointer to store the retrieved endpoint

int _lowcomm_osc_get_remote_memory(lcp_task_h task, const int target, const uint64_t remote_addr, mpc_lowcomm_window_t *const win, lcp_mem_h *rmem_p)

Retrieves remote memory key of the target.

Parameters:

task – [in] Current LCP task
target – [in] Rank of the target task
remote_addr – [in] Address of the buffer in case of dynamic window
win – [in] Window on which the communication happens
rmem_p – [out] Retrieved remote memory

Returns:

MPC_LOWCOMM_SUCCESS upon success

int _lowcomm_osc_release_remote_memory(lcp_mem_h *rmem, mpc_lowcomm_window_t *const win)

Release a remote memory created with _lowcomm_osc_get_remote_memory.

This function set the memory handle to NULL to avoid any misusage

Parameters:

rmem – [in] Reference on the remote memory to release
win – [in] Window associated with the retrieval

Returns:

MPC_LOWCOMM_SUCCESS upon success

int _lowcomm_osc_find_attached_region_position(const _lowcomm_osc_dynamic_win_t *regions, const int min_idx, const int max_idx, const uint64_t base, const size_t len, int *insert_idx)

Finds the attached memory corresponding with base of size len.

This function is a dichotomic search implemented in a recursive manner

Parameters:

regions – [in] Array containing the dynamic memories of a window
min_idx – [in] Start index for the search
max_idx – [in] Stop index for the search
base – [in] Address of the buffer associated with the memory to search
len – [in] Size of the memory to find
insert_idx – [out] Pointer to store the first free index in case the memory was not found

Return values:

index – of found memory
-1 – if not found

int _lowcomm_osc_end_exclusive_lock(lcp_task_h task, const int target, const uint64_t lock_offset, mpc_lowcomm_window_t *window)

Locks a window in exclusive mode.

Parameters:

task – [in] Current LCP Task
target – [in] Rank of the target task
lock_offset – [in] Offset (in Bytes) of the lock state in the target window
window – [in] Window to lock

Returns:

MPC_LOWCOMM_SUCCESS upon success

int _lowcomm_osc_start_exclusive_lock(lcp_task_h task, const int target, const uint64_t lock_offset, mpc_lowcomm_window_t *window)

Unlocks a window in exclusive mode.

Parameters:

task – [in] Current LCP Task
target – [in] Rank of the target task
lock_offset – [in] Offset (in Bytes) of the lock state in the target window
window – [in] Window to unlock

Returns:

MPC_LOWCOMM_SUCCESS upon success

static inline bool __sync_need_atomic_lock(const int target, mpc_lowcomm_window_t *window)

Tests if a window is already locked in exclusive mode.

This function is used to know if an atomic lock is necessary as those two type of locks are very similar.

Parameters:

target – [in] Rank of the target task
window – [in] Window to check the if the lock is held

Return values:

true – if an atomic lock is needed (no exclusive lock found)
false – if no atomic lock is needed (exclusive lock found)

static int __sync_start_shared_lock(lcp_task_h task, const int target, mpc_lowcomm_window_t *window)

Locks a window in shared mode.

Parameters:

task – [in] Current LCP Task
target – [in] Rank of the target task
window – [in] Window to lock

Returns:

MPC_LOWCOMM_SUCCESS upon success

static int __sync_end_shared_lock(lcp_task_h task, const int target, mpc_lowcomm_window_t *window)

Unlocks a window in shared mode.

Parameters:

task – [in] Current LCP Task
target – [in] Rank of the target task
window – [in] Window to unlock

Returns:

MPC_LOWCOMM_SUCCESS upon success

static int __sync_start_atomic_lock(lcp_task_h task, const int target, mpc_lowcomm_window_t *window)

Locks a window in atomic mode.

Note

If the window is already locked in exclusive mode, this function does nothing

Parameters:

task – [in] Current LCP Task
target – [in] Rank of the target task
window – [in] Window to lock

Return values:

MPC_LOWCOMM_SUCCESS – upon success
MPC_LOWCOMM_OSC_ALREADY_LOCKED – if the lock is already taken (no need to unlock)
A – LowComm errorcode otherwise

static int __sync_end_atomic_lock(lcp_task_h task, const int target, mpc_lowcomm_window_t *window)

Unlocks a window in atomic mode.

Note

If the window is already locked in exclusive mode, this function does nothing

Parameters:

task – [in] Current LCP Task
target – [in] Rank of the target task
window – [in] Window to unlock

Returns:

MPC_LOWCOMM_SUCCESS upon success

static void __sync_handle_incoming_post(volatile uint64_t *post_ptr, const int size, const int ranks_in_grp_win[static const size], mpc_lowcomm_window_t *window)

Updates the pending posts list on receiving a post from another task.

Parameters:

post_ptr – [in] Some internal state
size – [in] Number of tasks in the group
ranks_in_grp_win – [in] Translation table between tasks in group and global task ranks
window – [in] Window for which the post are handled

int lowcomm_osc_progress()

Progression function for the LowComm one-sided subpart.

This function is meant to be registered in mpc_common_progress

Returns:: lcp_progress errorcode

static int _win_init(const mpc_lowcomm_window_flag_t flavor, mpc_lowcomm_communicator_t comm, void **base, const size_t size, mpc_lowcomm_window_t **win_p)

Creates and initializes a window.

Parameters:

flavor – [in] Type of creation to perform
comm – [in] Communicator to embed in the new window
base – [in] Pointer to the address of the underlying memory buffer
size – [in] Requested size (in Bytes) of the underlying memory buffer
win_p – [out] Pointer to store an handle on the newly created window

Returns:

MPC_LOWCOMM_SUCCESS upon success

Variables

static const lcp_atomic_dt_t mpc2lcp_atomic_dt_table[]: Translation table from MPC datatypes to LCP atomic datatypes.

static const lcp_atomic_op_t mpc2lcp_atomic_op_table [] ={[LOWCOMM_ATOMIC_OP_FETCH_ADD] = LCP_ATOMIC_OP_ADD,[LOWCOMM_ATOMIC_OP_FETCH_SUB] = LCP_ATOMIC_OP_SUB,[LOWCOMM_ATOMIC_OP_FETCH_AND] = LCP_ATOMIC_OP_AND,[LOWCOMM_ATOMIC_OP_FETCH_OR] = LCP_ATOMIC_OP_OR,[LOWCOMM_ATOMIC_OP_FETCH_XOR] = LCP_ATOMIC_OP_XOR,[LOWCOMM_ATOMIC_OP_SWAP] = LCP_ATOMIC_OP_SWAP,[LOWCOMM_ATOMIC_OP_CSWAP] = LCP_ATOMIC_OP_CSWAP,}: Translation table from MPC Atomic operations to LCP atomic operations.

static int osc_is_initialized = 0: Whether the One Sided part of LowComm is already initialized.

static lcp_manager_h osc_mngr = NULL: Specific manager for the one-sided communications.

static mpc_common_spinlock_t osc_win_lock = MPC_COMMON_SPINLOCK_INITIALIZER: Mutex lock for accessing the global state of the submodule.

struct _lowcomm_osc_dynamic_win_s

Dynamic memory description.

Public Members

uint64_t base: Address of the base of the buffer.

size_t size: Size (in Bytes) of the memory region.

size_t rkey_len: Size (in Bytes) of the remote memory key.

char rkey_buf[OCS_LCP_RKEY_BUF_SIZE_MAX]: Remote memory key.

struct _lowcomm_osc_module_state_s

Synchronization states specific values.

Public Members

volatile uint64_t completion_counter: Number of complete received.

volatile uint64_t post_counter: Number of post received.

volatile uint64_t post_state[OSC_POST_PEER_MAX]: State of each post.

volatile uint64_t acc_lock: State of the accumulate lock.

volatile uint64_t global_lock: State of the global lock.

volatile uint64_t region_lock: State of the regional lock.

volatile uint64_t dynamic_win_count: Number memory regions attached.

volatile _lowcomm_osc_dynamic_win_t dynamic_wins[OSC_DYNAMIC_WIN_ATTACH_MAX]: Memory regions.

struct _lowcomm_osc_win_info_s

Description of a window.

Public Members

uint64_t addr: Address of the described memory.

lcp_mem_h rkey: Remote key of the described memory.

struct _lowcomm_osc_pending_post_s

Description of a pending post operation.

Public Members

int rank: Rank of the source task.

mpc_list_elem_t elem: Structure to make the pending post a queue.

struct _lowcomm_osc_local_dynamic_win_info_s

Description of an attached memory region.

Public Members

lcp_mem_h memh: Memory describer.

volatile int refcnt: Reference counting to avoid duplicates.

struct _lowcomm_osc_lock_s

Description of a distant lock.

Public Members

bool is_no_check: Does the lock should not really lock.

mpc_lowcomm_osc_lock_type_t type: Type of lock (see mpc_lowcomm_osc_lock_type_t)

int target: Rank of the target task.

struct mpc_lowcomm_window_s

Internal structure of a LCP backed window.

Public Members

lcp_manager_h mngr: Communication manager.

lcp_context_h ctx: Communication context.

mpc_lowcomm_communicator_t comm: Communicator.

mpc_lowcomm_window_flag_t win_flag: Window creation flag.

void *base_data: Address of the underlying user buffer.

lcp_mem_h lkey_data: Local memory key of the user buffer memory.

_lowcomm_osc_win_info_t *rdata_win_info: Remote windows information about user buffer.

_lowcomm_osc_module_state_t *state: Address of the state values.

lcp_mem_h lkey_state: Local memory key of the state values memory.

_lowcomm_osc_win_info_t *rstate_win_info: Remote windows information about state.

struct mpc_common_hashtable outstanding_locks: Locks currently held.

int lock_all_is_no_check: Does the lock all really lock.

int post_count: Number of pending posts.

mpc_list_elem_t pending_posts: Queue with the pending posts.

mpc_lowcomm_group_t *start_group: Group with granted access through start.

mpc_lowcomm_group_t *post_group: Group with granted exposure through post.

int *start_grp_ranks: Translation table between start group ranks.

_lowcomm_osc_local_dynamic_win_info_t *local_dynamic_win_infos