MESSIP: Message-Passing over TCP/IP

Currently, still under development.
This is yet highly experimental, not stable,
and provided only as reference.

Status – Future Directions

Download

Introduction

MESSIP implements Message Passing over TCP/IP, using BSD socket interface programming.
Three kinds of messages can be sent between Processes or Threads:

· Synchronous Messages (blocking messages), from 0 to 64k in size,

· Asynchronous Messages (non blocking, buffered messages), from 0 to 64k in size,

· Timer messages (one shot and repetitive)

Concepts

Message Passing consist in the exchange of bytes from one task (Process or Thread) to another (Process or Thread), whatever if these tasks are on the same node or on another node over a TCP/IP network.

The task sending the message is known as the client and the task receiving the message is known as the server. In theory, clients and servers do not need a third agent to communicate between themselves, at least for Synchronous Messages (e.g. messages that do not require to be buffered). In practice, to initiate the communication between clients and servers, it’s necessary to have a third task, which is called ‘messip-mgr’ (the messip manager). Only one ‘messip_mgr’ is necessary on a given network. Further versions of messip will enable to have several redundant managers.

A library (shared) provides the messip API functions that enable to talk to either the messip manager or to the other tasks (either clients or servers).

Each process which wants to use Message Passing must first connect to the messip manager, using the next API function: messip_connect. If no IP address and no port is specified (preferred method), then a text file /etc/messip is used. If this file does not exist, then the default IP name ‘localhost’ is used, and the default port 9000 is used.

In order to receive messages, a server must first create a channel (see API function: messip_channel_create). A channel is identified by a name, which must be unique over the whole network. It’s perfectly fine to create several channels for a given process, but because you can receive messages on one specific channel and this operation is (usually) blocking (i.e. you’ll usually wait until you get a message over this channel), in practice a thread will create and manage only a channel at a time. In the future, it will be possible to receive messages on several channels at the same time. Note that you could poll, but it’s often not very efficient.

Prior to send any message to a server (whatever it is a Synchronous or an Asynchronous Message), a client must find the channel. That means that the server must know the name that identifies the channel. In order to be able to further communicate with the server, that’s the only information that the client must know about the server. The API function to be used is: messip_channel_connect.

Both API functions messip_channel_create and messip_channel_connect involve communication with the messip manager (messip-mgr).

A server typically waits for incoming messages, using the next API function: messip_receive. This is normally a blocking function (but you can specify a timeout value), that means that you exit from this function on these events:

· a message has been received (either Synchronous or Asynchronous): each Synchronous message has to be replied, unless you want to keep the client blocked and waiting for an answer.

· timeout has occurred

· a timer has expired

· a client which was connected has either died or closed the connection to the channel.

A client sends either Synchronous or Asynchronous Messages:

· a Synchronous Message is blocking: that means the client will wait until the message is received by the server and processed. The server has to reply to the client in order for the client to be unblocked. Note that you can specify a timeout. Synchronous Messaging do not involve the Messip manager (messip_server agent), only the server and the client talk together. Sending a Synchronous message is performed with the next API function: messip_send

· an Asynchronous Message is non blocking. That means that a client can’t ignore the state of the server. Because of this, there is no reply from the server. Therefore, you can’t assume that the message has been received by the server. When the server has created the channel (where he receive messages), he has specified a parameter which is the maximum number of messages that can be buffered. Asynchronous Message are slower than Synchronous Message because they are copied twice: from the client to the Messip manager, then from the Messip manager to the server.

A simple example

This is a very simple example, with 2 processes:

· the 1st process create a named channel, in order to receive messages, then wait messages (with a 30 seconds timeout);

· the 2nd process send two Synchronous Messages over this channel (i.e. to the 1st process).

// This is the server

#include <stdio.h>

#include <stdlib.h>

#include <assert.h>

#include "messip.h"

int

main( int argc, char *argv[] )

{

messip_cnx_t *cnx;

messip_channel_t *ch;

int32_t type;

int index;

// Connexion to the messip manager (use of /etc/messip)

messip_init();

if ( !(cnx = messip_connect( NULL, "p1", MESSIP_NOTIMEOUT )) )

return -1;

// Create a channel, in order to receive messages on it

ch = messip_channel_create( cnx, “one” , MESSIP_NOTIMEOUT, 0 );

assert( ch != NULL );

// Receive messages

for (;;) {

index = messip_receive( ch,

&type, rec_buff, sizeof(rec_buff),
30000 ); // 30 seconds timeout

if (index == MESSIP_MSG_TIMEOUT)

break;

messip_reply( ch, index, "Bonjour", 8 );

}

return 0;

}

// This is the client

#include <stdio.h>

#include <stdlib.h>

#include <assert.h>

#include "messip.h"

int

main( int argc, char *argv[] )

{

messip_cnx_t *cnx;

messip_channel_t *ch;

char rec_buff[80];

int32_t answer;

// Connexion to the messip manager (use of /etc/messip)

messip_init();

if ( !(cnx = messip_connect( NULL, "p2", MESSIP_NOTIMEOUT )) )

return -1;

// Locate the channel where to send message to

ch = messip_channel_connect( cnx, “one”, MESSIP_NOTIMEOUT );

assert( ch != NULL );

// Send two messages

messip_send( ch,
    100, "Un", 3,    // Type=100
    &answer, rec_buff, sizeof( rec_buff ),
    5000 );            // 5 seconds timeout

messip_send( ch,
    1000, "Deux", 4,   // Type=1000
    &answer, rec_buff, sizeof( rec_buff ),
    5000 );            // 5 seconds timeout

return 0;

}

Implementation notes

Resources needed:

· The messip manager (messip-mgr) dynamically creates a new thread to handle each client (whatever it’s a server or a client). There is a new client on each new connection made with messip_connect()

· Also, for the messip manager (messip_server), there is a new thread created for each client to handle Asynchronous Messages (the thread is created on the 1st Asynchronous Message).

· If a server use a timer managed by the library, then a new thread is created when the timer is triggered.

· Both the server and the library must be linked with ‘librt’ that provides some ‘real-time’ extension to Linux, and also with ‘pthread’ (linux).

Performances

Local

			Synchronous Messages	Asynchronous Messages
Machine	Length (bytes) of Send Messages	Length (bytes) of Reply Message	Number of exchanges / seconds	Number of exchanges / seconds
Machine 1	0	0	49,450	21,880
Machine 1	100	20	47,700	20,570
Machine 1	200	50	47,730
Machine 1	1,000	200	45,100	20,710
Machine 1	10,000	1,000	30,000	30,000

Network

				Synchronous Messages	Asynchronous Messages
Machine	Machine	Length (bytes) of Send Messages	Length (bytes) of Reply Message	Number of exchanges / seconds	Number of exchanges / seconds
Machine	Machine	Length (bytes) of Send Messages	Length (bytes) of Reply Message	Number of exchanges / seconds	Messip_mgr and server on the same machine	Messip_mgr and server not on the same machine
M1	M2	0	0	9,820	9,836	9,019
M1	M2	100	20	8,085	8,063	7,730
M1	M2	1,000	200	3,234	3,254	3,570

Notes

· Machine 1: Desktop Intel(R) Pentium(R) 4 CPU 2.26GHz

· Machine 2: Laptop Intel(R) Pentium(R) 4 CPU 2.40GHz

· System: Fedora 0.95, kernel 2.6.0-test8

· Network: 100 Mb

Documentation

Doxygen generated documentation for the library is here.

Doxygen generated documentation for the manager is here.

Please note that only 15% is done so far, the documentation is still a work in progress.

API Functions

Function	Type	Prototype	Description
messip_init	Server and Client	void messip_init( void );	Library initializations
messip_connect	Server and Client	messip_cnx_t * messip_connect(char *mgr_ref, messip_it_t id, int msec_timeout)	Connection to the messip manager (messip_mgr)
messip_disconnect	Server and Client	int messip_disconnect( messip_cnx_t *cnx, long msec_timeout)	Disconnection from the messip manager (messip_mgr)
messip_channel_create	Server	messip_channel_t * messip_channel_create( messip_cnx_t cnx, const char name, long msec_timeout, int maxnb_msg_buffered )	Create a channel when clients will be able to send messages over.
messip_channel_delete	Server	int messip_channel_delete( messip_channel_t *ch, long msec_timeout )	Delete a channel that was previously created. Only channel without active cconnection can be successfully deleted.
messip_channel_connect	Client	messip_channel_t * messip_channel_connect( messip_cnx_t cnx, const char name, long msec_timeout )	Enable a client to connect to the channel owned by a server, in order to send messages.
messip_channel_disconnect	Client	int messip_channel_disconnect( messip_channel_t *ch, long msec_timeout )	Enable a client to disconnect from a channel.
messip_ ping	Client	int messip_ ping( messip_channel_t *ch, long msec_timeout )	Ping the server to check if he can currently process messages.
messip_receive	server	int messip_receive( messip_channel_t ch, int32_t type, void *buffer, int maxlen, long msec_timeout )	Enable a server to receive messages: - Synchronous - Asynchronous - Timer - Disconnection of a client - Death of a client
messip_reply	server	int messip_reply( messip_channel_t ch, int index, int32_t answer, void reply_buffer, int reply_len, long msec_timeout )	Enable a server to reply to a client. Note that until the server has replied to the client, the client is blocked, unless the client has specified a timeout.
messip_send	client	int messip_send( messip_channel_t ch, int32_t type, void send_buffer, int send_len, int32_t answer, void reply_buffer, int reply_maxlen, long msec_timeout )	Enable a client to send a Synchronous Message to a server. The client will be blocked until the server has replied.
messip_buffered_send	client	int messip_buffered_send( messip_channel_t ch, int32_t type, void send_buffer, int send_len, long msec_timeout )	Enable a client to send an Asynchronous Message to a server. The server will buffer these messages, until the maximum number of buffered messages is reached. There is no reply from the server, therefore the client does not wait, unless the maximum number of buffered messages is reached.
messip_timer_create	server	timer_t messip_timer_create( messip_channel_t *ch, int32_t type, long msec_1st_shot, long msec_rep_shot, long msec_timeout )	Arm a timer, either one-shot or repetitive. Note that a new thread is created into the client process when the time is triggered.
messip_timer_delete	server	int messip_timer_delete( messip_channel_t *ch, timer_t timer_id )	Delete a timer previously created.

Utilities

Function	Description
messip-mgr	MessIP manager.

Examples

Several examples (C code) are provided to demonstrate and to test the use of the library:

Code	Description
p.c	A minimum example to demonstrate the very basic messaging using a channel: the server create a channel, wait for a message w/o timeout, and reply back with another msg. the client connect to the channel and send a message w/o timeout.
p.c	A more complex example involving one server and two clients: the server create a channel, then wait messages from clients (p2 and p3). The server does not reply immediately to the clients and keep them blocked. the clients connect to the channel and send a msg.
p.c	Shows the use of timeout and how to know when a client has disconnected from a channel: the server (p1) create a channel, wait a message with a 10" timeout. It displays if a client has disconnected from the channel. the client (p2) send a message to the channel, then exit (which has the side effect to disconnect the client from the channel where it was previously attached).
p.c	Shows how to use Asynchronous messages (non blocking, buffered messages).
p.c	This is a performance test program: the client sends messages to the server for 10 seconds. Both the length of the messages (send and reply) and the duration are configurable from the command-line.
p.c	This example shows how to use a timer (messages sent automatically using ca timer).
p.c
p.c	Messages sent/received using several channels.
p.c	Shows what do to when the message sent by the client exceeds the size of the receiving buffer on the server side.

How to build the server and the library

After your have downloaded the sources code (from SVN), just go into the main directory, and run make:

    ~/sf/messip$ make

To compile the examples: