LowComm Protocol layer (LCP)

group LowComm Protocol API

Description of the LCP API.

LCP Context is the communication manager.

Its primary functions are: list, initiate, configure and instantiate the communication interfaces (rails) available. There can be a single communication context per application.

Typedefs

typedef struct lcp_dt_ops lcp_dt_ops_t

Datatype operations.

There used to support packing/unpacking of derived MPI datatypes.

typedef struct lcp_context_param lcp_context_param_t

LCP context parameters.

It is passed to the lcp_context_create routine to configure communication context initialization.

Enums

LCP context field mask.

It is used to specify which parameters has been specified in lcp_context_param_t.

Values:

enumerator LCP_CONTEXT_DATATYPE_OPS = MPC_BIT(0): datatype mask

enumerator LCP_CONTEXT_PROCESS_UID = MPC_BIT(1): process uid mask

enumerator LCP_CONTEXT_NUM_TASKS = MPC_BIT(2): task number mask

enumerator LCP_CONTEXT_REQUEST_SIZE = MPC_BIT(3): upper layer request size

enumerator LCP_CONTEXT_REQUEST_CB = MPC_BIT(4): request init callback

Functions

int lcp_context_create(lcp_context_h *ctx_p, lcp_context_param_t *param)

Context creation.

List, init and instantiate communication interfaces available. Based on the components (tcp, ptl, ofi,…), list available devices, open interfaces based on user configuration.

Parameters:

param – [in] lcp_context_param_t Parameters for context creation.
ctx_p – [out] Pointer to context handle that will be allocated by the routine.

Returns:

Error code returned.

int lcp_context_fini(lcp_context_h ctx)

Context release.

Release and deallocate all the communication interface. Call at the end of the execution when no more communication are needed.

Parameters:

ctx – [in] Context handle to release.

Returns:

Error code returned.

lcp_task_h lcp_context_task_get(lcp_context_h ctx, int tid)

Get task handle.

Get the task handle based on the task identifier (TID). The TID is the MPI rank in the MPI_COMM_WORLD communicator.

Parameters:

ctx – [in] Context handle.
tid – [in] Task identifier or MPI comm world rank.

Returns:

lcp_task_h task handle for the specified tid.

int lcp_progress(lcp_manager_h mngr)

Communication progress.

Progress all outstanding communications.

Parameters:

ctx – [in] Context handle.

Returns:

Error code returned.

struct lcp_dt_ops

Datatype operations.

There used to support packing/unpacking of derived MPI datatypes.

Public Members

void (*pack)(void *context, void *buffer)

void (*unpack)(void *context, void *buffer)

struct lcp_context_param

LCP context parameters.

It is passed to the lcp_context_create routine to configure communication context initialization.

Public Members

uint32_t field_mask: context field mask

uint64_t process_uid: local process UID

int num_tasks: local process UID

lcp_dt_ops_t dt_ops

int request_size: datatype operations (pack/unpack)

lcp_request_init_callback_func_t request_init

LCP endpoints are containers for connection and communication management.

They are virtual communication channels associated with each process in an MPI program. There can be only one endpoint per MPC UNIX process. It supports multirail (data striping and multiplexing) transparently.

Functions

int lcp_ep_create(lcp_manager_h mngr, lcp_ep_h *ep_p, uint64_t uid, unsigned flags)

Create LCP Endpoint.

Instantiate a protocol endpoint. Upon return, connection will have been started and the endpoint can be used directly with communication primitives.

Parameters:

ctx – [in] Manager handle.
ep_p – [out] Pointer to endpoint handle that will be allocated by the routine.
uid – [in] Process identifier or MPC UNIX identifier.
flags – [in] Flags to configure creation (unused for now).

Returns:

Error code returned.

lcp_ep_h lcp_ep_get(lcp_manager_h mngr, uint64_t uid)

Get LCP Endpoint.

Get a protocol endpoint based on UNIX process identifier (UID).

Parameters:

ctx – [in] Manager handle.
uid – [in] Process identifier or MPC UNIX identifier.

Returns:

lcp_ep_h endpoint handle for the specified UID.

int lcp_ep_get_or_create(lcp_manager_h mngr, uint64_t uid, lcp_ep_h *ep_p, unsigned flags)

Get LCP Endpoint or create it if it does not exist.

Get a protocol endpoint based on UNIX process identifier (UID), creates it if it to does exist yet.

Parameters:

ctx – [in] Manager handle.
uid – [in] Process identifier or MPC UNIX identifier.
ep_p – [out] Pointer to endpoint handle that will be allocated by the routine.
flags – [in] Flags to configure creation (unused for now).

Returns:

Error code returned.

LCP Tasks are containers for task data (ex: tag matching list).

Any communication primitives has the local task as argument.

Functions

int lcp_task_create(lcp_context_h ctx, int tid, lcp_task_h *task_p)

Create LCP Task.

Instantiate an LCP Task that will contain all communication related information that should be own by a MPI task. There are as many tasks as MPI ranks, each task is identified by its global MPI rank.

Parameters:

ctx – [in] Context handle.
tid – [in] Task identifier or MPI comm world rank.
task_p – [out] Pointer to task handle that will be allocated by the routine.

Returns:

Error code returned.

LCP Communication primitives.

Typedefs

typedef struct lcp_manager_param lcp_manager_param_t

Manager parameters.

Specifies the parameters that have been specified in lcp_manager_param_t.

typedef struct lcp_tag_recv_info lcp_tag_recv_info_t

Tag receive information.

Information to be returned to user whenever a request as been matched. The structure is passed from upper layer and field by LCP with tag information (length, tag, source and found in case of probing). For example, this can be used in MPI_Status.

typedef struct lcp_tag_info lcp_tag_info_t

Tag information.

Information to be returned to user whenever a request as been matched. The structure is passed from upper layer and field by LCP with tag information (length, tag, source and found in case of probing). For example, this can be used in MPI_Status.

typedef struct lcp_request_param lcp_request_param_t

Request parameters.

Specifies a set of fields used to characterize how a request should be handled by the communication primitives. It is primarily used to specify user callback and to pass tag receive information (lcp_tag_recv_info_t).

typedef struct lcp_am_recv_param lcp_am_recv_param_t

LCP Active Message receive parameters.

Parameters that are returned to the user through param argument of lcp_am_callback_t user callback. The callback has previously been defined by the user and set to the local task through lcp_am_set_handler_callback.

Enums

Manager fields.

Specifies fields present in lcp_manager_param_t during manager creation.

Values:

enumerator LCP_MANAGER_ESTIMATED_EPS = MPC_BIT(0)

enumerator LCP_MANAGER_COMM_MODEL = MPC_BIT(1)

Manager instantiation flags.

Specifies a set of flags that are proposed to improve performances in some situation.

Values:

enumerator LCP_MANAGER_TSC_MODEL = MPC_BIT(0): Init interface Two-Sided Communication capabilities.

enumerator LCP_MANAGER_OSC_MODEL = MPC_BIT(1): Init interface One-Sided Communication capabilities.

Request field masks.

Specifies the parameters that have been specified in lcp_request_param_t.

Values:

enumerator LCP_REQUEST_TRY_OFFLOAD = MPC_BIT(0): Try offload send mask.

enumerator LCP_REQUEST_USER_DATA = MPC_BIT(1): User data mask.

enumerator LCP_REQUEST_USER_REQUEST = MPC_BIT(2): User request mask.

enumerator LCP_REQUEST_TAG_SYNC = MPC_BIT(3): Sync request mask.

enumerator LCP_REQUEST_AM_SYNC = MPC_BIT(4): AM sync request mask.

enumerator LCP_REQUEST_SEND_CALLBACK = MPC_BIT(5): Send callback mask.

enumerator LCP_REQUEST_RECV_CALLBACK = MPC_BIT(6): Recv callback mask.

enumerator LCP_REQUEST_REPLY_BUFFER = MPC_BIT(7): Result buffer for Atomics.

enumerator LCP_REQUEST_USER_MEMH = MPC_BIT(8): User-provided local Memory handle.

enumerator LCP_REQUEST_USER_EPH = MPC_BIT(9): User-provided Endpoint handle.

enum lcp_dt_type

Datatype type.

Specifies if the datat in the request is contiguous or not (derived)

Values:

enumerator LCP_DATATYPE_CONTIGUOUS = MPC_BIT(0)

enumerator LCP_DATATYPE_DERIVED = MPC_BIT(1)

LCP Active Message flags.

Flags identifying the type of message returned by the callback lcp_am_callback_t.

Values:

enumerator LCP_AM_EAGER = MPC_BIT(0): Message received is of type eager.

enumerator LCP_AM_RNDV = MPC_BIT(1): Message received is of type rendez-vous.

Functions

int lcp_task_associate(lcp_task_h task, lcp_manager_h mngr)

LCP Task associate to manager.

Associates a task to a communication context, or lcp_manager_h. Internal structures are instantiated to enable communication context isolation. Type of structure instantiated may depend on the communication model of the manager. For example, matching queues are only created for two-sided communication model.

Parameters:

task – [in] Task to be associated.
mngr – [in] Manager handle to be associated to.

Returns:

Error code returned.

int lcp_task_dissociate(lcp_task_h task, lcp_manager_h mngr)

LCP Task dissociate from manager.

Remove all data structure used within the communication context, or lcp_manager_h.

Parameters:

task – [in] Task to be dissociated.
mngr – [in] Manager handle to be dissociated to.

Returns:

Error code returned.

int lcp_manager_create(lcp_context_h ctx, lcp_manager_h *mngr_p, lcp_manager_param_t *params)

Create LCP Manager.

An LCP manager act as a communication context that groups all outstanding communications. It can be used to insulate different communication contexts such as MPI windows or global contexts to allow fine grain progression, endpoint flush, etc… Endpoints lcp_ep_h are local to managers. An LCP manager is shared between a group of LCP Tasks.

Parameters:

ctx – [in] Context handle.
mngr_p – [out] Pointer to manager handle that will be allocated by the routine.
params – [in] Manager parameters.

Returns:

Error code returned.

int lcp_manager_fini(lcp_manager_h mngr)

Delete LCP Manager.

Deletes resources associated to a LCP manager.

Parameters:

mngr – [in] Manager handle to be deleted.

Returns:

Error code returned.

void *lcp_request_alloc(lcp_task_h task)

Initialize a request.

Returns an initialized pointer to a user request. Within the upper layer, it is sometimes needed to initialize fields of upper layer request before actually sending the request. The user is then responsible to free the request using lcp_request_free

Parameters:

task – [in] Task handle of the task allocating the request.
request – [out] Request handle containing the upper layer request.

Returns:

Error code returned.

void lcp_request_free(void *request)

Free a request.

Free a previously allocated request from lcp_request_alloc

Parameters:

request – [in] Request handle to be freed.

Returns:

Error code returned.

int lcp_request_check_status(void *request)

Check request status.

Check the status of an LCP request.

Parameters:

request – [in] Request handle to be checked.

Returns:

Error code returned.

lcp_status_ptr_t lcp_tag_send_nb(lcp_ep_h ep, lcp_task_h task, const void *buffer, size_t count, lcp_tag_info_t *tag_info, const lcp_request_param_t *param)

LCP send tag communication.

Communication primitive to send a tag message to the corresponding lcp_tag_recv_nb(). The routine is non-blocking and thus returns directly, however request may be completed later depending on how the message is sent. request is considered completed when buffer can be safely reused by the user. Actual completion is performed by setting the mpc_lowcomm_request_t COMPLETION_FLAG.

Parameters:

ep – [in] Endpoint handle to the destination.
task – [in] Task handle of the source task.
buffer – [in] Address of source buffer.
count – [in] Size of source buffer.
request – [in] Lowcomm request.
param – [in] Request parameters lcp_request_param_t.

Returns:

Error code returned or pointer to request. Use lcp_request_check_status to check completion.

lcp_status_ptr_t lcp_tag_recv_nb(lcp_manager_h mngr, lcp_task_h task, void *buffer, size_t count, lcp_tag_info_t *tag_info, int32_t src_mask, int32_t tag_mask, const lcp_request_param_t *param)

LCP receive tag communication.

Communication primitive to receive a tag message coming from lcp_tag_send_nb(). The routine is non-blocking and thus returns directly, request completion depends on whether it has been matched. Completion is performed internally by setting the mpc_lowcomm_request_t COMPLETION_FLAG, it can be done directly within the routine or later by a call to lcp_progress().

Parameters:

task – [in] Task handle of the source task.
buffer – [in] Address of buffer where data is copied.
count – [in] Size of source buffer.
request – [in] Lowcomm request.
param – [in] Request parameters lcp_request_param_t.

Returns:

Error code returned or pointer to request. Use lcp_request_check_status to check completion.

int lcp_tag_probe_nb(lcp_manager_h mngr, lcp_task_h task, const int src, const int tag, const uint64_t comm, lcp_tag_recv_info_t *recv_info)

LCP probe tag.

Communication primitive to probe internal matching lists to check if a message has been received. Information is filled within recv_info.

Parameters:

task – [in] Task handle of the source task.
src – [in] MPI rank.
tag – [in] MPI tag.
comm – [in] Communicator identifier.
recv_info – [in] Receive information container that will be filled in case a match is found.

Returns:

Error code returned.

int lcp_am_set_handler_callback(lcp_manager_h mngr, lcp_task_h task, uint8_t am_id, void *arg, lcp_am_callback_t cb, unsigned flags)

LCP register user handler for a specific AM identifier.

Register a user handler with a user specified AM identifier. Handler will be called when a message, that has been sent through lcp_am_send_nb with the specified AM ID, has been received. The user may use the opaque arg pointer to pass custom data.

Parameters:

task – [in] Task handle of the source task.
am_id – [in] Unique handler identifier.
arg – [in] Opaque pointer to user data.
cb – [in] User-defined handler callback.
flags – [in] Flags (unused).

Returns:

Error code returned.

lcp_status_ptr_t lcp_am_send_nb(lcp_ep_h ep, lcp_task_h task, int32_t dest_tid, uint8_t am_id, void *hdr, size_t hdr_size, const void *buffer, size_t count, const lcp_request_param_t *param)

LCP send AM communication.

Send data using the Active Message datapath. A user-defined handler must have been set prior to calling this routine, see lcp_am_set_handler_callback.

In case of thread-based communication, we must specify the destination rank through dest_tid.

Parameters:

ep – [in] Endpoint to destination.
task – [in] Task handle of the local task.
dest_tid – [in] Unique handler identifier.
am_id – [in] Unique handler identifier.
hdr – [in] Pointer to user-defined header.
hdr_size – [in] Size of user-defined header.
buffer – [in] Pointer to buffer.
count – [in] Size of buffer.
param – [in] Request parameters lcp_request_param_t.

Returns:

Error code returned or pointer to request. Use lcp_request_check_status to check completion.

lcp_status_ptr_t lcp_am_recv_nb(lcp_manager_h mngr, lcp_task_h task, void *data_ctnr, void *buffer, size_t count, const lcp_request_param_t *param)

LCP receive AM communication.

This routine may be called to provide to LCP the necessary user buffer to receive the data in case it is sent through the rendez-vous protocol. This routine may thus be used in conjunction with LCP_AM_RNDV flags that would have been set within the user callback through lcp_am_recv_param_t. To be notified for completion, user must set the completion callback in lcp_request_param_t.

Parameters:

task – [in] Task handle of the source task.
data_ctnr – [in] Opaque pointer that must be data from lcp_am_callback_t.
buffer – [in] Buffer on which received data will be written.
count – [in] Size of buffer.
param – [in] Request parameters lcp_request_param_t.

Returns:

Error code returned.

lcp_status_ptr_t lcp_put_nb(lcp_ep_h ep, lcp_task_h task, void *buffer, size_t length, uint64_t remote_addr, lcp_mem_h rkey, const lcp_request_param_t *param)

LCP Put RMA communication.

This routine exposes RMA put capabilities with the same semantics.

User may provide a callback through lcp_request_param_t so that it can be notified when the request has completed. As a return value, a pointer to the upper layer is provided which can be used to check progression of the communication.

Parameters:

ep – [in] Endpoint to destination.
task – [in] Task handle of the source task.
buffer – [in] Buffer from which data will be sent.
length – [in] Length of data in buffer.
remote_addr – [in] remote address to put the data.
rkey – [in] Remote memory key handle.
param – [in] Request parameters lcp_request_param_t.

Returns:

Pointer to upper layer request.

lcp_status_ptr_t lcp_get_nb(lcp_ep_h ep, lcp_task_h task, void *buffer, size_t length, uint64_t remote_addr, lcp_mem_h rkey, const lcp_request_param_t *param)

LCP Get RMA communication.

This routine exposes RMA get capabilities with the same semantics.

User may provide a callback through lcp_request_param_t so that it can be notified when the request has completed. As a return value, a pointer to the upper layer is provided which can be used to check progression of the communication.

Parameters:

ep – [in] Endpoint to destination.
task – [in] Task handle of the source task.
buffer – [in] Buffer from which data will be sent.
length – [in] Length of data in buffer.
remote_addr – [in] Remote address from which data is read.
rkey – [in] Remote memory key handle.
param – [in] Request parameters lcp_request_param_t.

Returns:

Error code returned or pointer to pending request.

lcp_status_ptr_t lcp_flush_nb(lcp_manager_h mngr, lcp_task_h task, const lcp_request_param_t *param)

LCP Flush RMA communication.

This routine flushes all outstanding RMA operations. Once the operation is complete, all RMA operations issued before this call are locally and remotely completed.

User may provide a callback through lcp_request_param_t so that it can be notified when the request has completed.

Parameters:

mngr – [in] Communication manager.
task – [in] Task handle of the source task.
param – [in] Request parameters lcp_request_param_t.

Returns:

Error code returned or pointer to pending request.

lcp_status_ptr_t lcp_atomic_op_nb(lcp_ep_h ep, lcp_task_h task, const void *buffer, lcp_atomic_dt_t atomic_datatype, uint64_t remote_addr, lcp_mem_h rkey, lcp_atomic_op_t op_type, const lcp_request_param_t *param)

LCP Atomic RMA communication.

This routine exposes RMA Atomic capabilities with the same semantics.

User may provide a callback through lcp_request_param_t so that it can be notified when the request has completed. As a return value, a pointer to the upper layer is provided which can be used to check progression of the communication. The reply_buffer field shall contain the compare operand for the compare and swap operation on call.

Parameters:

ep – [in] Endpoint to destination.
task – [in] Task handle of the source task.
buffer – [in] Buffer from which data will be sent.
atomic_datatype – [in] Datatype of the data to use.
remote_addr – [in] Remote address from which data is read or to which it is written.
rkey – [in] Remote memory key handle.
op_type – [in] Type of atomic operation to perform
param – [in] Request parameters lcp_request_param_t.

Returns:

Error code returned or pointer to pending request.

struct lcp_manager_param

Manager parameters.

Specifies the parameters that have been specified in lcp_manager_param_t.

Public Members

unsigned field_mask

int estimated_eps: Estimated number of endpoints.

unsigned flags: communication model.

struct lcp_tag_recv_info

Tag receive information.

Information to be returned to user whenever a request as been matched. The structure is passed from upper layer and field by LCP with tag information (length, tag, source and found in case of probing). For example, this can be used in MPI_Status.

Public Members

size_t length: Length of the data received by the matched request.

int32_t tag: Tag of the matched request.

int32_t src: Source of the matched request.

int32_t dest: Destination of the matched request.

unsigned found: Has request been found in matching list or not.

unsigned flags: Flags containing info from LCP (truncation, …)

struct lcp_tag_info

Tag information.

Information to be returned to user whenever a request as been matched. The structure is passed from upper layer and field by LCP with tag information (length, tag, source and found in case of probing). For example, this can be used in MPI_Status.

Public Members

int32_t tag: Tag of the matched request.

int32_t src: Source of the matched request.

int32_t dest: Source of the matched request.

uint64_t src_uid: Source UID.

uint64_t dest_uid: Destination UID.

uint16_t comm_id: Communicator ID.

unsigned found: Has request been found in matching list or not.

struct lcp_request_param

Request parameters.

Specifies a set of fields used to characterize how a request should be handled by the communication primitives. It is primarily used to specify user callback and to pass tag receive information (lcp_tag_recv_info_t).

Public Members

uint32_t field_mask: Flags to indicate which parameter is used.

lcp_send_callback_func_t send_cb: Completion callback for send requests.

lcp_tag_recv_callback_func_t recv_cb: Completion callback for recv requests.

void *reply_buffer: Location for returned value with atomic operations.

lcp_datatype_t datatype: Contiguous or non-contiguous data.

void *request: Pointer to mpc_lowcomm_request for completion.

lcp_mem_h mem: Memory handle for RMA requests.

lcp_ep_h ep: Endpoint to be flushed.

struct lcp_am_recv_param

LCP Active Message receive parameters.

Parameters that are returned to the user through param argument of lcp_am_callback_t user callback. The callback has previously been defined by the user and set to the local task through lcp_am_set_handler_callback.

Public Members

uint32_t flags

lcp_ep_h reply_ep

Atomic interface of the LCP layer.

Typedefs

typedef union lcp_atomic_reply_data_u lcp_atomic_reply_data_t: Union of the possible replies types.

typedef struct lcp_atomic_reply lcp_atomic_reply_t: Structured data to describe the reply to a fetch-like atomic operation.

typedef struct lcp_atomic_proto lcp_atomic_proto_t: Interface functions for the atomics.

Enums

Values:

enumerator LCP_ATO_PROTO_RMA: Use RMA channel to send atomics.

enumerator LCP_ATO_PROTO_SW: Use the software atomics.

Functions

static void lcp_atomic_complete(lcr_completion_t *comp)

Completes an atomic operation.

Parameters:

comp – [in] Completion function to call on completion

static int lcp_atomic_post(lcp_request_t *req)

Performs an atomic operation without fetch.

Parameters:

req – [in] Request describing the operation to perform

Return values:

MPC_LOWCOMM_SUCCESS – upon success
errorcode – otherwise

static int lcp_atomic_fetch(lcp_request_t *req)

Performs an atomic operation with fetch.

Parameters:

req – [in] Request describing the operation to perform

Return values:

MPC_LOWCOMM_SUCCESS – upon success
errorcode – otherwise

static int lcp_atomic_cswap(lcp_request_t *req)

Performs an atomic compare and swap operation.

Parameters:

req – [in] Request describing the operation to perform

Return values:

MPC_LOWCOMM_SUCCESS – upon success
errorcode – otherwise

static inline int lcp_atomic_select_proto_channel(lcp_ep_h ep, const lcp_atomic_proto_t **ato_proto_p, lcp_chnl_idx_t *cc)

Selects the type of atomic to perform (software or RMA)

Parameters:

ep – [in] Endpoint describing the capabilities
ato_proto_p – [in] Interface with the function to use
cc – [in] Index of the chosen channel

Returns:

MPC_LOWCOMM_SUCCESS

lcp_status_ptr_t lcp_atomic_op_nb(lcp_ep_h ep, lcp_task_h task, const void *buffer, lcp_atomic_dt_t atomic_datatype, uint64_t remote_addr, lcp_mem_h rkey, lcp_atomic_op_t op_type, const lcp_request_param_t *param)

LCP Atomic RMA communication.

This routine exposes RMA Atomic capabilities with the same semantics.

User may provide a callback through lcp_request_param_t so that it can be notified when the request has completed. As a return value, a pointer to the upper layer is provided which can be used to check progression of the communication. The reply_buffer field shall contain the compare operand for the compare and swap operation on call.

Parameters:

ep – [in] Endpoint to destination.
task – [in] Task handle of the source task.
buffer – [in] Buffer from which data will be sent.
atomic_datatype – [in] Datatype of the data to use.
remote_addr – [in] Remote address from which data is read or to which it is written.
rkey – [in] Remote memory key handle.
op_type – [in] Type of atomic operation to perform
param – [in] Request parameters lcp_request_param_t.

Returns:

Error code returned or pointer to pending request.

static ssize_t lcp_atomic_pack(void *dest, void *data)

Packs a software atomic operation into an atomic header.

Parameters:

dest – [out] Header of the communication containing the necessary information
data – [in] Request describing the data to pack into the atomic header

Returns:

Size of the packed header in bytes

static ssize_t lcp_atomic_reply_pack(void *dest, void *data)

Packs a reply to a software atomic operation into an atomic header.

Parameters:

dest – [out] Header of the communication containing the necessary information
data – [in] Request describing the data to pack into the atomic header

Returns:

Size of the packed header in bytes

int lcp_atomic_sw(lcp_request_t *req)

Generic function to send a software atomic operation.

Parameters:

req – [in] Request containing the necessary information for the communication

Return values:

MPC_LOWCOMM_SUCCESS –
MPC_LOWCOMM_ERROR –

static int lcp_atomic_reply(lcp_manager_h mngr, lcp_task_h task, lcp_atomic_reply_t *ato_reply)

Generic function to send the reply of a completed fetch-like atomic operation.

Parameters:

mngr – [in] Manager used for the communication
task – [in] Task identifier of the initiator, destination of communication
ato_reply – [in] Packed header containing the data to communicate

Return values:

MPC_LOWCOMM_SUCCESS – upon success
errorcode – otherwise

static int lcp_atomic_handler(void *arg, void *data, size_t length, unsigned flags)

Executes the requested atomic operation with the provided data.

This function handles local atomic operations which have been provided through an Active Message or shm like communication.

Parameters:

arg – [in] Manager used for the communication
data – [in] Header of the communication containing the necessary information
length – [in] Size of the datatype in bytes (UNUSED)
flags – [in] (UNUSED)

Return values:

MPC_LOWCOMM_SUCCESS – on success
MPC_LOWCOMM_ERROR – otherwise

static int lcp_atomic_reply_handler(void *arg, void *data, size_t length, unsigned flags)

Handles the response of fetch-like atomic operations.

This function replies with the data to the remote initiator.

Parameters:

arg – [in] Manager used for the communication (UNUSED)
data – [in] Header of the communication containing the necessary information
length – [in] Size of the datatype in bytes (UNUSED)
flags – [in] (UNUSED)

Returns:

MPC_LOWCOMM_SUCCESS

LCP_DEFINE_AM (LCP_AM_ID_ATOMIC, lcp_atomic_handler, 0)

LCP_DEFINE_AM (LCP_AM_ID_ATOMIC_REPLY, lcp_atomic_reply_handler, 0)

static inline const char *lcp_ato_sw_decode_op(lcp_atomic_op_t op_type)

Decodes the LCP operation types into strings for logging purposes.

Parameters:

op_type – [in] Operation to decode

Returns:

String containing the name of the operation

Variables

static const lcr_atomic_op_t lcp_atomic_op_table [] ={[LCP_ATOMIC_OP_ADD] = LCR_ATOMIC_OP_ADD,[LCP_ATOMIC_OP_AND] = LCR_ATOMIC_OP_AND,[LCP_ATOMIC_OP_OR] = LCR_ATOMIC_OP_OR,[LCP_ATOMIC_OP_XOR] = LCR_ATOMIC_OP_XOR,[LCP_ATOMIC_OP_SWAP] = LCR_ATOMIC_OP_SWAP,[LCP_ATOMIC_OP_CSWAP] = LCR_ATOMIC_OP_CSWAP,}: Translation table from LCP to LCR atomic ops.

static const lcr_atomic_dt_t lcp_atomic_dt_table [] ={[LCP_ATOMIC_DT_FLOAT] = LCR_ATOMIC_DT_FLOAT,[LCP_ATOMIC_DT_DOUBLE] = LCR_ATOMIC_DT_DOUBLE,[LCP_ATOMIC_DT_INT8] = LCR_ATOMIC_DT_INT8,[LCP_ATOMIC_DT_INT16] = LCR_ATOMIC_DT_INT16,[LCP_ATOMIC_DT_INT32] = LCR_ATOMIC_DT_INT32,[LCP_ATOMIC_DT_INT64] = LCR_ATOMIC_DT_INT64,[LCP_ATOMIC_DT_UINT8] = LCR_ATOMIC_DT_UINT8,[LCP_ATOMIC_DT_UINT16] = LCR_ATOMIC_DT_UINT16,[LCP_ATOMIC_DT_UINT32] = LCR_ATOMIC_DT_UINT32,[LCP_ATOMIC_DT_UINT64] = LCR_ATOMIC_DT_UINT64,}: Translation table from LCP to LCR atomic datatypes.

const lcp_atomic_proto_t ato_rma_proto = {.send_cswap = lcp_atomic_cswap, .send_fetch = lcp_atomic_fetch, .send_post = lcp_atomic_post,}: Interface functions using the RMA atomics.

mpc_common_spinlock_t lcp_atomic_global_lock = MPC_COMMON_SPINLOCK_INITIALIZER: Lock used for the fallback when atomic operation is not available.

const lcp_atomic_proto_t ato_sw_proto = {.send_fetch = lcp_atomic_sw, .send_cswap = lcp_atomic_sw, .send_post = lcp_atomic_sw,}: Interface functions using the software atomics.

static const size_t lcp_atomic_dt_size [] ={[LCP_ATOMIC_DT_FLOAT] = sizeof(float),[LCP_ATOMIC_DT_DOUBLE] = sizeof(double),[LCP_ATOMIC_DT_INT8] = sizeof(int8_t),[LCP_ATOMIC_DT_INT16] = sizeof(int16_t),[LCP_ATOMIC_DT_INT32] = sizeof(int32_t),[LCP_ATOMIC_DT_INT64] = sizeof(int64_t),[LCP_ATOMIC_DT_UINT8] = sizeof(uint8_t),[LCP_ATOMIC_DT_UINT16] = sizeof(uint16_t),[LCP_ATOMIC_DT_UINT32] = sizeof(uint32_t),[LCP_ATOMIC_DT_UINT64] = sizeof(uint64_t),}: Extent of the atomic datatypes.

const lcp_atomic_proto_t ato_sw_proto: Interface functions using the software atomics.

const lcp_atomic_proto_t ato_rma_proto: Interface functions using the RMA atomics.

union lcp_atomic_reply_data_u

Union of the possible replies types.

Public Members

int8_t i8: 1 byte signed integer

int16_t i16: 2 bytes signed integer

int32_t i32: 4 bytes signed integer

int64_t i64: 8 bytes signed integer

uint8_t u8: 1 byte unsigned integer

uint16_t u16: 2 bytes unsigned integer

uint32_t u32: 4 bytes unsigned integer

uint64_t u64: 8 bytes unsigned integer

float f32: 4 bytes floating point number

double f64: 8 bytes floating point number

struct lcp_atomic_reply

Structured data to describe the reply to a fetch-like atomic operation.

Public Members

uint64_t dest_uid: UID of the initiator.

uint64_t msg_id: ID of the operation.

lcp_atomic_reply_data_t result: Fetched data.

struct lcp_atomic_proto

Interface functions for the atomics.

Public Members

lcp_send_func_t send_fetch: Fetch operation.

lcp_send_func_t send_cswap: Compare and swap operation.

lcp_send_func_t send_post: Atomic operation without fetching.

LCP Memory provides abstraction to interact with memory registration on the Network Interface Cards.

It is primarily used for one-sided communication primitives.

Typedefs

typedef struct lcp_mem_attr lcp_mem_attr_t

LCP memory attributes.

Specifies a set of attributes associated to a memory region. They can be retrieved by a call to lcp_mem_query.

Enums

LCP memory attributes.

Memory attributes.

Values:

enumerator LCP_MEM_SIZE_FIELD = MPC_BIT(0)

enumerator LCP_MEM_ADDR_FIELD = MPC_BIT(1)

enumerator LCP_MEM_PACK_KEY_SIZE = MPC_BIT(2)

enumerator LCP_MEM_PACK_KEY_ALLOC = MPC_BIT(3)

Functions

int lcp_mem_provision(lcp_manager_h mngr, lcp_mem_h *mem_p, lcp_mem_param_t *params)

LCP provision memory.

Provision memory by allocating if specified in the parameters and by registering to a transport and get a memory handle from it. This call be used for MPI 3 Windows allocation calls for example.

Parameters:

mngr – [in] Manager handle.
mem_p – [in] Pointer handle to registered memory.
buffer – [in] Buffer from which data will be registered.
length – [in] Length of data in buffer.
flags – [in] Flags to specify kind of registration.

Returns:

Error code returned.

int lcp_mem_deprovision(lcp_manager_h mngr, lcp_mem_h mem)

LCP deregister memory.

Deregister memory from a NIC.

Parameters:

mngr – [in] Manager handle.
mem – [in] Memory handle to be deregistered.

Returns:

Error code returned.

int lcp_mem_query(lcp_mem_h mem, lcp_mem_attr_t *mem_attr)

LCP query registered memory.

Query attributes of memory that has been registered.

Parameters:

ctx – [in] Context handle.
mem – [in] Memory handle to be deregistered.

Returns:

Error code returned.

int lcp_mem_pack(lcp_manager_h mngr, lcp_mem_h mem, void **rkey_buf_p, int *rkey_len)

LCP pack memory key.

Pack a memory key so it can be sent to a remote peer.

It is allocated by LCP and must be released appropriately, see lcp_mem_release_rkey_buf.

Parameters:

mngr – [in] Manager handle.
mem – [in] Memory handle to be packed.
rkey_buf_p – [in] Buffer where memory key data will be packed (allocated by LCP).
rkey_len – [in] Length of packed memory key in bytes.

Returns:

Error code returned.

int lcp_mem_unpack(lcp_manager_h mngr, lcp_mem_h *mem_p, void *src, size_t size)

LCP unpack memory key.

Allocate a memory key and unpack a memory key so it can be used for RMA communication.

Parameters:

mngr – [in] Manager handle.
mem_p – [in] Pointer where memory key will be unpacked.
src – [in] Buffer where memory key data has been packed
size – [in] Size of the packed key in bytes.

Returns:

Error code returned.

void lcp_mem_release_rkey_buf(void *rkey_buffer)

LCP release memory key.

Release packed memory key created through lcp_mem_pack.

Parameters:

rkey_buffer – [in] Pointer to packed memory key.

Returns:

Error code returned.

struct lcp_mem_attr

LCP memory attributes.

Specifies a set of attributes associated to a memory region. They can be retrieved by a call to lcp_mem_query.

Public Members

uint32_t field_mask

void *address

size_t size