POSIX thread (pthread) librariesThe POSIX thread libraries are a standards based thread API for C/C++. It allows one to spawn a new concurrent process flow. It is most effective on multi-processor or multi-core systems where the process flow can be scheduled to run on another processor thus gaining speed through parallel or distributed processing. Threads require less overhead than "forking" or spawning a new process because the system does not initialize a new system virtual memory space and environment for the process. While most effective on a multiprocessor system, gains are also found on uniprocessor systems which exploit latency in I/O and other system functions which may halt process execution. (One thread may execute while another is waiting for I/O or some other system latency.) Parallel programming technologies such as MPI and PVM are used in a distributed computing environment while threads are limited to a single computer system. All threads within a process share the same address space. A thread is spawned by defining a function and its arguments which will be processed in the thread. The purpose of using the POSIX thread library in your software is to execute software faster.

Thread Basics: Thread operations include thread creation, termination, synchronization

(joins,blocking), scheduling, data management and process interaction. A thread does not maintain a list of created threads, nor does it know the

thread that created it. All threads within a process share the same address space. Threads in the same process share:

o Process instructions o Most data o open files (descriptors) o signals and signal handlers o current working directory o User and group id

Each thread has a unique: o Thread ID o set of registers, stack pointer o stack for local variables, return addresses o signal mask o priority o Return value: errno

pthread functions return "0" if OK.

Thread Creation and Termination:

Example: pthread1.c

#include <stdio.h>#include <stdlib.h>#include <pthread.h>

void *print_message_function( void *ptr );

main(){ pthread_t thread1, thread2; char *message1 = "Thread 1"; char *message2 = "Thread 2"; int iret1, iret2;

/* Create independent threads each of which will execute function */

iret1 = pthread_create( &thread1, NULL, print_message_function, (void*) message1); iret2 = pthread_create( &thread2, NULL, print_message_function, (void*) message2);

/* Wait till threads are complete before main continues. Unless we */ /* wait we run the risk of executing an exit which will terminate */ /* the process and all threads before the threads have completed. */

pthread_join( thread1, NULL); pthread_join( thread2, NULL);

printf("Thread 1 returns: %d\n",iret1); printf("Thread 2 returns: %d\n",iret2); exit(0);}

void *print_message_function( void *ptr ){ char *message; message = (char *) ptr; printf("%s \n", message);}

Compile:

C compiler: cc -lpthread pthread1.c or

C++ compiler: g++ -lpthread pthread1.c

Run: ./a.out Results:

Thread 1

Thread 2Thread 1 returns: 0Thread 2 returns: 0

Details:

In this example the same function is used in each thread. The arguments are different. The functions need not be the same.

Threads terminate by explicitly calling pthread_exit, by letting the function return, or by a call to the function exit which will terminate the process including any threads.

Function call: pthread_create - create a new thread

int pthread_create(pthread_t * thread, const pthread_attr_t * attr, void * (*start_routine)(void *), void *arg);

Arguments:

o thread - returns the thread id. (unsigned long int defined in bits/pthreadtypes.h)

o attr - Set to NULL if default thread attributes are used. (else define members of the struct pthread_attr_t defined in bits/pthreadtypes.h) Attributes include:

detached state (joinable? Default: PTHREAD_CREATE_JOINABLE. Other option: PTHREAD_CREATE_DETACHED)

scheduling policy (real-time? PTHREAD_INHERIT_SCHED,PTHREAD_EXPLICIT_SCHED,SCHED_OTHER)

scheduling parameter inheritsched attribute (Default: PTHREAD_EXPLICIT_SCHED Inherit

from parent thread: PTHREAD_INHERIT_SCHED) scope (Kernel threads: PTHREAD_SCOPE_SYSTEM User threads:

PTHREAD_SCOPE_PROCESS Pick one or the other not both.) guard size stack address (See unistd.h and bits/posix_opt.h

_POSIX_THREAD_ATTR_STACKADDR) stack size (default minimum PTHREAD_STACK_SIZE set in pthread.h),

o void * (*start_routine) - pointer to the function to be threaded. Function has a single argument: pointer to void.

o *arg - pointer to argument of function. To pass multiple arguments, send a pointer to a structure.

Function call: pthread_join - wait for termination of another thread

int pthread_join(pthread_t th, void **thread_return);

Arguments:

o th - thread suspended until the thread identified by th terminates, either by calling pthread_exit() or by being cancelled.

o thread_return - If thread_return is not NULL, the return value of th is stored in the location pointed to by thread_return.

Function call: pthread_exit - terminate the calling thread

void pthread_exit(void *retval);

Arguments:

o retval - Return value of thread.

This routine kills the thread. The pthread_exit function never returns. If the thread is not detached, the thread id and return value may be examined from another thread by using pthread_join. Note: the return pointer *retval, must not be of local scope otherwise it would cease to exist once the thread terminates.

[C++ pitfalls]: The above sample program will compile with the GNU C and C++ compiler g++. The following function pointer representation below will work for C but not C++. Note the subtle differences and avoid the pitfall below:

void print_message_function( void *ptr );......iret1 = pthread_create( &thread1, NULL, (void*)&print_message_function, (void*) message1);......

Thread Synchronization:

The threads library provides three synchronization mechanisms:

mutexes - Mutual exclusion lock: Block access to variables by other threads. This enforces exclusive access by a thread to a variable or set of variables.

joins - Make a thread wait till others are complete (terminated). condition variables - data type pthread_cond_t

Mutexes:Mutexes are used to prevent data inconsistencies due to operations by multiple threads upon the same memory area performed at the same time or to prevent

race conditions where an order of operation upon the memory is expected. A contention or race condition often occurs when two or more threads need to perform operations on the same memory area, but the results of computations depends on the order in which these operations are performed. Mutexes are used for serializing shared resources such as memory. Anytime a global resource is accessed by more than one thread the resource should have a Mutex associated with it. One can apply a mutex to protect a segment of memory ("critical region") from other threads. Mutexes can be applied only to threads in a single process and do not work between processes as do semaphores.

Example threaded function:

Without Mutex With Mutex

view source

print?1 int counter=0; 2   3 /* Function C */4 void functionC() 5 {

6   7    counter++

8   9 }

view source

print?

01 /* Note scope of variable and mutex are the same */

02 pthread_mutex_t mutex1 = PTHREAD_MUTEX_INITIALIZER;

03 int counter=0; 04   05 /* Function C */06 void functionC() 07 { 08    pthread_mutex_lock( &mutex1 ); 09    counter++ 10    pthread_mutex_unlock( &mutex1 ); 11 }

Possible execution sequence

Thread 1 Thread 2 Thread 1 Thread 2

counter = 0

counter = 0

counter = 0 counter = 0

counter = 1

counter = 1

counter = 1 Thread 2 locked out.Thread 1 has exclusive use of variable counter

counter = 2

If register load and store operations for the incrementing of variable counter occurs with unfortunate timing, it is theoretically possible to have each thread increment and overwrite the same variable with the same value. Another possibility is that thread two would first increment counter locking out thread one until complete and then thread one would increment it to 2.

Sequence Thread 1 Thread 2

1 counter = 0 counter=0

2Thread 1 locked out.Thread 2 has exclusive use of variable counter

counter = 1

3 counter = 2

Code listing: mutex1.c

#include <stdio.h>#include <stdlib.h>#include <pthread.h>

void *functionC();pthread_mutex_t mutex1 = PTHREAD_MUTEX_INITIALIZER;int counter = 0;

main(){ int rc1, rc2; pthread_t thread1, thread2;

/* Create independent threads each of which will execute functionC */

if( (rc1=pthread_create( &thread1, NULL, &functionC, NULL)) ) { printf("Thread creation failed: %d\n", rc1); }

if( (rc2=pthread_create( &thread2, NULL, &functionC, NULL)) ) { printf("Thread creation failed: %d\n", rc2); }

/* Wait till threads are complete before main continues. Unless we */ /* wait we run the risk of executing an exit which will terminate */ /* the process and all threads before the threads have completed. */

pthread_join( thread1, NULL); pthread_join( thread2, NULL);

exit(0);}

void *functionC(){ pthread_mutex_lock( &mutex1 ); counter++; printf("Counter value: %d\n",counter); pthread_mutex_unlock( &mutex1 );}

Compile: cc -lpthread mutex1.c Run: ./a.out Results:

Counter value: 1Counter value: 2

When a mutex lock is attempted against a mutex which is held by another thread, the thread is blocked until the mutex is unlocked. When a thread terminates, the mutex does not unless explicitly unlocked. Nothing happens by default.

Man Pages: pthread_mutex_lock() - acquire a lock on the specified mutex variable. If

the mutex is already locked by another thread, this call will block the calling thread until the mutex is unlocked.

pthread_mutex_unlock() - unlock a mutex variable. An error is returned if mutex is already unlocked or owned by another thread.

pthread_mutex_trylock() - attempt to lock a mutex or will return error code if busy. Useful for preventing deadlock conditions.

Joins:A join is performed when one wants to wait for a thread to finish. A thread calling routine may launch multiple threads then wait for them to finish to get the results. One waits for the completion of the threads with a join.

Sample code: join1.c

#include <stdio.h>#include <pthread.h>

#define NTHREADS 10void *thread_function(void *);pthread_mutex_t mutex1 = PTHREAD_MUTEX_INITIALIZER;int counter = 0;

main(){ pthread_t thread_id[NTHREADS]; int i, j;

for(i=0; i < NTHREADS; i++) { pthread_create( &thread_id[i], NULL, thread_function, NULL ); }

for(j=0; j < NTHREADS; j++)

{ pthread_join( thread_id[j], NULL); } /* Now that all threads are complete I can print the final result. */ /* Without the join I could be printing a value before all the threads */ /* have been completed. */

printf("Final counter value: %d\n", counter);}

void *thread_function(void *dummyPtr){ printf("Thread number %ld\n", pthread_self()); pthread_mutex_lock( &mutex1 ); counter++; pthread_mutex_unlock( &mutex1 );}

Compile: cc -lpthread join1.c Run: ./a.out Results:

Thread number 1026Thread number 2051Thread number 3076Thread number 4101Thread number 5126Thread number 6151Thread number 7176Thread number 8201Thread number 9226Thread number 10251Final counter value: 10

Man Pages: pthread_create() - create a new thread pthread_join() - wait for termination of another thread pthread_self() - return identifier of current thread

Condition Variables:

A condition variable is a variable of type pthread_cond_t and is used with the appropriate functions for waiting and later, process continuation. The condition variable mechanism allows threads to suspend execution and relinquish the processor until some condition is true. A condition variable must always be

associated with a mutex to avoid a race condition created by one thread preparing to wait and another thread which may signal the condition before the first thread actually waits on it resulting in a deadlock. The thread will be perpetually waiting for a signal that is never sent. Any mutex can be used, there is no explicit link between the mutex and the condition variable.

Man pages of functions used in conjunction with the condition variable:

Creating/Destroying: o pthread_cond_init o pthread_cond_t cond = PTHREAD_COND_INITIALIZER; o pthread_cond_destroy

Waiting on condition: o pthread_cond_wait - unlocks the mutex and waits for the condition

variable cond to be signaled. o pthread_cond_timedwait - place limit on how long it will block.

Waking thread based on condition: o pthread_cond_signal - restarts one of the threads that are waiting

on the condition variable cond. o pthread_cond_broadcast - wake up all threads blocked by the

specified condition variable.

Example code: cond1.c

#include <stdio.h>#include <stdlib.h>#include <pthread.h>

pthread_mutex_t count_mutex = PTHREAD_MUTEX_INITIALIZER;pthread_cond_t condition_var = PTHREAD_COND_INITIALIZER;

void *functionCount1();void *functionCount2();int count = 0;#define COUNT_DONE 10#define COUNT_HALT1 3#define COUNT_HALT2 6

main(){ pthread_t thread1, thread2;

pthread_create( &thread1, NULL, &functionCount1, NULL); pthread_create( &thread2, NULL, &functionCount2, NULL);

pthread_join( thread1, NULL); pthread_join( thread2, NULL);

printf("Final count: %d\n",count);

exit(0);

}

// Write numbers 1-3 and 8-10 as permitted by functionCount2()

void *functionCount1(){ for(;;) { // Lock mutex and then wait for signal to relase mutex pthread_mutex_lock( &count_mutex );

// Wait while functionCount2() operates on count // mutex unlocked if condition varialbe in functionCount2() signaled. pthread_cond_wait( &condition_var, &count_mutex ); count++; printf("Counter value functionCount1: %d\n",count);

pthread_mutex_unlock( &count_mutex );

if(count >= COUNT_DONE) return(NULL); }}

// Write numbers 4-7

void *functionCount2(){ for(;;) { pthread_mutex_lock( &count_mutex );

if( count < COUNT_HALT1 || count > COUNT_HALT2 ) { // Condition of if statement has been met. // Signal to free waiting thread by freeing the mutex. // Note: functionCount1() is now permitted to modify "count". pthread_cond_signal( &condition_var ); } else { count++; printf("Counter value functionCount2: %d\n",count); }

pthread_mutex_unlock( &count_mutex );

if(count >= COUNT_DONE) return(NULL); }

}

Compile: cc -lpthread cond1.c Run: ./a.out Results:

Counter value functionCount1: 1Counter value functionCount1: 2Counter value functionCount1: 3Counter value functionCount2: 4Counter value functionCount2: 5Counter value functionCount2: 6Counter value functionCount2: 7Counter value functionCount1: 8Counter value functionCount1: 9Counter value functionCount1: 10Final count: 10

Note that functionCount1() was halted while count was between the values COUNT_HALT1 and COUNT_HALT2. The only thing that has been ensures is that functionCount2 will increment the count between the values COUNT_HALT1 and COUNT_HALT2. Everything else is random.

The logic conditions (the "if" and "while" statements) must be chosen to insure that the "signal" is executed if the "wait" is ever processed. Poor software logic can also lead to a deadlock condition.

Note: Race conditions abound with this example because count is used as the condition and can't be locked in the while statement without causing deadlock.

Thread Scheduling:

When this option is enabled, each thread may have its own scheduling properties. Scheduling attributes may be specified:

during thread creation by dynamically by changing the attributes of a thread already created by defining the effect of a mutex on the thread's scheduling when creating

a mutex by dynamically changing the scheduling of a thread during synchronization

operations.

The threads library provides default values that are sufficient for most cases.

Thread Pitfalls: Race conditions: While the code may appear on the screen in the order

you wish the code to execute, threads are scheduled by the operating system and are executed at random. It cannot be assumed that threads

are executed in the order they are created. They may also execute at different speeds. When threads are executing (racing to complete) they may give unexpected results (race condition). Mutexes and joins must be utilized to achieve a predictable execution order and outcome.

Thread safe code: The threaded routines must call functions which are "thread safe". This means that there are no static or global variables which other threads may clobber or read assuming single threaded operation. If static or global variables are used then mutexes must be applied or the functions must be re-written to avoid the use of these variables. In C, local variables are dynamically allocated on the stack. Therefore, any function that does not use static data or other shared resources is thread-safe. Thread-unsafe functions may be used by only one thread at a time in a program and the uniqueness of the thread must be ensured. Many non-reentrant functions return a pointer to static data. This can be avoided by returning dynamically allocated data or using caller-provided storage. An example of a non-thread safe function is strtok which is also not re-entrant. The "thread safe" version is the re-entrant version strtok_r.

Mutex Deadlock: This condition occurs when a mutex is applied but then not "unlocked". This causes program execution to halt indefinitely. It can also be caused by poor application of mutexes or joins. Be careful when applying two or more mutexes to a section of code. If the first pthread_mutex_lock is applied and the second pthread_mutex_lock fails due to another thread applying a mutex, the first mutex may eventually lock all other threads from accessing data including the thread which holds the second mutex. The threads may wait indefinitely for the resource to become free causing a deadlock. It is best to test and if failure occurs, free the resources and stall before retrying.

...pthread_mutex_lock(&mutex_1);while ( pthread_mutex_trylock(&mutex_2) ) /* Test if already locked */{ pthread_mutex_unlock(&mutex_1); /* Free resource to avoid deadlock */ ... /* stall here */ ... pthread_mutex_lock(&mutex_1);}count++;pthread_mutex_unlock(&mutex_1);pthread_mutex_unlock(&mutex_2);...

The order of applying the mutex is also important. The following code segment illustrates a potential for deadlock:

void *function1(){ ... pthread_mutex_lock(&lock1); // Execution step 1 pthread_mutex_lock(&lock2); // Execution step 3 DEADLOCK!!! ... ... pthread_mutex_lock(&lock2); pthread_mutex_lock(&lock1); ...}

void *function2(){ ... pthread_mutex_lock(&lock2); // Execution step 2 pthread_mutex_lock(&lock1); ... ... pthread_mutex_lock(&lock1); pthread_mutex_lock(&lock2); ...}

main(){ ... pthread_create(&thread1, NULL, function1, NULL); pthread_create(&thread2, NULL, function2, NULL); ...}

If function1 acquires the first mutex and function2 acquires the second, all resources are tied up and locked.

Condition Variable Deadlock: The logic conditions (the "if" and "while" statements) must be chosen to insure that the "signal" is executed if the "wait" is ever processed.

Process:

fork():

The fork() system call will spawn a new child process which is an identical process to the parent except that has a new system process ID. The process is copied in memory from the parent and a new process structure is assigned by

the kernel. The return value of the function is which discriminates the two threads of execution. A zero is returned by the fork function in the child's process.

The environment, resource limits, umask, controlling terminal, current working directory, root directory, signal masks and other process resources are also duplicated from the parent in the forked child process.

Example:

#include <iostream>#include <string>

// Required by for routine#include <sys/types.h>#include <unistd.h>

#include <stdlib.h> // Declaration for exit()

using namespace std;

int globalVariable = 2;

main(){ string sIdentifier; int iStackVariable = 20;

pid_t pID = fork(); if (pID == 0) // child { // Code only executed by child process

sIdentifier = "Child Process: "; globalVariable++; iStackVariable++; } else if (pID < 0) // failed to fork { cerr << "Failed to fork" << endl; exit(1); // Throw exception } else // parent { // Code only executed by parent process

sIdentifier = "Parent Process:"; }

// Code executed by both parent and child. cout << sIdentifier; cout << " Global variable: " << globalVariable; cout << " Stack variable: " << iStackVariable << endl;}

Compile: g++ -o ForkTest ForkTest.cpp Run: ForkTest

Parent Process: Global variable: 2 Stack variable: 20Child Process: Global variable: 3 Stack variable: 21

[Potential Pitfall]: Some memory duplicated by a forked process such as file pointers, will cause intermixed output from both processes. Use the wait() function so that the processes do not access the file at the same time or open unique file descriptors. Some like stdout or stderr will be shared unless synchronized using wait() or some other mechanism. The file close on exit is another gotcha. A terminating process will close files before exiting. File locks set by the parent process are not inherited by the child process.

[Potential Pitfall]: Race conditions can be created due to the unpredictability of when the kernel scheduler runs portions or time slices of the process. One can use wait(). the use of sleep() does not guarentee reliability of execution on a heavily loaded system as the scheduler behavior is not predictable by the application.

Note on exit() vs _exit(): The C library function exit() calls the kernel system call _exit() internally. The kernel system call _exit() will cause the kernel to close descriptors, free memory, and perform the kernel terminating process clean-up. The C library function exit() call will flush I/O buffers and perform aditional clean-up before calling _exit() internally. The function exit(status) causes the executable to return "status" as the return code for main(). When exit(status) is called by a child process, it allows the parent process to examine the terminating status of the child (if it terminates first). Without this call (or a call from main() to return()) and specifying the status argument, the process will not return a value.

#include <stdlib.h>

void exit(int status);

#include <unistd.h>

void _exit(int status);

Man Pages:

fork - create a child process

vfork():

The vfork() function is the same as fork() except that it does not make a copy of the address space. The memory is shared reducing the overhead of spawning a new process with a unique copy of all the memory. This is typically used when using fork() to exec() a process and terminate. The vfork() function also executes the child process first and resumes the parent process when the child terminates.

#include <iostream>#include <string>

// Required by for routine#include <sys/types.h>#include <unistd.h>

using namespace std;

int globalVariable = 2;

main(){ string sIdentifier; int iStackVariable = 20;

pid_t pID = vfork(); if (pID == 0) // child { // Code only executed by child process

sIdentifier = "Child Process: "; globalVariable++; iStackVariable++; cout << sIdentifier; cout << " Global variable: " << globalVariable; cout << " Stack variable: " << iStackVariable << endl; _exit(0); } else if (pID < 0) // failed to fork { cerr << "Failed to fork" << endl; exit(1); // Throw exception } else // parent { // Code only executed by parent process

sIdentifier = "Parent Process:"; }

// executed only by parent

cout << sIdentifier; cout << " Global variable: " << globalVariable; cout << " Stack variable: " << iStackVariable << endl;

exit(0);}

Compile: g++ -o VForkTest VForkTest.cpp Run: VForkTest

Child Process: Global variable: 3 Stack variable: 21Parent Process: Global variable: 3 Stack variable: 21

Note: The child process executed first, updated the variables which are shared between the processes and NOT unique, and then the parent process executes using variables which the child has updated.

[Potential Pitfall]: A deadlock condition may occur if the child process does not terminate, the parent process will not proceed.

Man Pages:

vfork - create a child process and block parent _exit - - terminate the current process

clone():

The function clone() creates a new child process which shares memory, file descriptors and signal handlers with the parent. It implements threads and thus launches a function as a child. The child terminates when the parent terminates. See the YoLinux POSIX threads tutorial

Man Pages:

clone - create a child process

wait():

The parent process will often want to wait until all child processes have been completed. this can be implemented with the wait() function call.

wait(): Blocks calling process until the child process terminates. If child process has already teminated, the wait() call returns immediately. if the calling process has multiple child processes, the function returns when one returns.

waitpid(): Options available to block calling process for a particular child process not the first one.

Code snipet: #include <sys/wait.h>

...

pid_t pID = <I>set to child process id with call to fork OR:</I> // If set <-1, wait for any child process whose process group ID = abs(pID) // If = -1, wait for any child process. Same as wait(). // If = 0, wait for any child process whose process group ID is same as calling process. // If > 0, wait for the child whose process ID = pID.

... int childExitStatus;

pid_t ws = waitpid( pID, &childExitStatus, WNOHANG);

if( WIFEXITED(childExitStatus) ) { // Child process exited thus exec failed. // LOG failure of exec in child process. cout << "Result of waitpid: Child process exited thus exec failed." << endl; }

OR

... int childExitStatus;

pid_t ws = waitpid( pID, &childExitStatus, 0);

if( !WIFEXITED(childExitStatus) ){ cerr << "waitpid() exited with an error: Status= " << WEXITSTATUS(childExitStatus) << endl;}else if( WIFSIGNALED(childExitStatus) ){ cerr << "waitpid() exited due to a signal: " << WTERMSIG(childExitStatus) << endl;}

Notes: See man page for options: WNOHANG, WUNTRACED. See man page for return macros: WIFEXITED(), WEXITSTATUS(),

WIFSIGNALED(), WTERMSIG(), WIFSTOPPED(), WSTOPSIG().

See man page for errors: ECHILD, EINVAL, EINTR. (Also see sample of error processing below.)

Man Pages:

wait / waitpid - wait for process termination

Set system group ID and process ID:

Avoids orphaned process group when parent terminates. When parent dies, this will be a zombie. (No parent process. Parent=1) Instead, create a new process group for the child. Later process the group is terminated to stop all spawned processes. Thus all subsequent processes should be of this group if they are to be terminated by the process group id. Process group leader has the same process id and group process id. If not changed then the process group is that of the parent. Set the process group id to that of the child process.

#include <sys/types.h>#include <unistd.h>

#ifdef __gnu_linux__ pid_t pgid = setpgid(child_pID, child_pID);#endif

... ...

if( pgid < 0) { cout << "Failed to set process group ID" << endl; _exit(0); // If exec fails then exit forked process.

Use the setgid call to set the group id of the current process. Requires root access.

The macro testing for __gnu_linux__ is for cross platform support as man other OS's use a different system call.

Man Pages:

setpgid/getpgid setpgrp/getpgrp - set process group setsid - creates a session and sets the process group ID getuid/geteuid - get user identity setgid - set group identity getgid/getegid - get group (real/effective) identity

setreuid/setregid - set real user or group identity

Kill all processes in a process group:This is the real reason to set up a process group. One may kill all the processes in the process group without having to keep track of how many processes have been forked and all of their process id's.

See /usr/include/bits/signum.h for list of signals.

...

int killReturn = killpg( pID, SIGKILL); // Kill child process group

if( killReturn == ESRCH) // pid does not exist{ cout << "Group does not exist!" << endl;}else if( killReturn == EPERM) // No permission to send signal{ cout << "No permission to send signal!" << endl;}else cout << "Signal sent. All Ok!" << endl;

...

Man Pages: killpg - send signal to a process group kill - send signal to a process signal (2) - ANSI C signal handling signal (7) - List of available signals sigaction - POSIX signal handling functions. pause (2) - wait for signal raise (3) - send a signal to a current process

system() and popen():

The system() call will execute an OS shell command as described by a character command string. This function is implemented using fork(), exec() and waitpid(). The command string is executed by calling /bin/sh -c command-string. During execution of the command, SIGCHLD will be blocked, and SIGINT and SIGQUIT will be ignored. The call "blocks" and waits for the task to be performed before continuing.

#include <stdio.h>#include <stdlib.h>main(){

system("ls -l"); printf("Command done!");

}

The statement "Command done!" will not print untill the "ls -l" command has completed.

The popen() call opens a process by creating a pipe, forking, and invoking the shell (bourne shell on Linux). The advantage to using popen() is that it will allow one to interrogate the results of the command issued.

This example opens a pipe which executes the shell command "ls -l". The results are read and printed out.

#include <stdio.h>main(){ FILE *fpipe; char *command="ls -l"; char line[256];

if ( !(fpipe = (FILE*)popen(command,"r")) ) { // If fpipe is NULL perror("Problems with pipe"); exit(1); }

while ( fgets( line, sizeof line, fpipe)) { printf("%s", line); } pclose(fpipe);}

#include <stdio.h>main(){ FILE *fpipe; char *command="ls -l"; char line[256];

if ( !(fpipe = (FILE*)popen(command,"r")) ) { // If fpipe is NULL perror("Problems with pipe"); exit(1); }

while ( fgets( line, sizeof line, fpipe)) { printf("%s", line); } pclose(fpipe);}

The second argument to popen:

r: Read from stdin (command results) w: write to stdout (command) For stderr: command="ls -l 2>&1","w");

Man Pages:

system - execute a shell command popen - process I/O

exec() functions and execve():

The exec() family of functions will initiate a program from within a program. They are also various front-end functions to execve().

The functions return an integer error code. (0=Ok/-1=Fail).

execl() and execlp():

The function call "execl()" initiates a new program in the same environment in which it is operating. An executable (with fully qualified path. i.e. /bin/ls) and arguments are passed to the function. Note that "arg0" is the command/file name to execute.

int execl(const char *path, const char *arg0, const char *arg1, const

char *arg2, ... const char *argn, (char *) 0);

#include <unistd.h>main(){ execl("/bin/ls", "/bin/ls", "-r", "-t", "-l", (char *) 0);}

Where all function arguments are null terminated strings. The list of arguments is terminated by NULL.

The routine execlp() will perform the same purpose except that it will use environment variable PATH to determine which executable to process. Thus a fully qualified path name would not have to be used. The first argument to the function could instead be "ls". The function execlp() can also take the fully qualified name as it also resolves explicitly.

Man Pages:

execl / execlp - execute

execv() and execvp():

This is the same as execl() except that the arguments are passed as null terminated array of pointers to char. The first element "argv[0]" is the command name.

int execv(const char *path, char *const argv[]);

example:

#include <unistd.h>main(){ char *args[] = {"/bin/ls", "-r", "-t", "-l", (char *) 0 };

execv("/bin/ls", args);

}The routine execvp() will perform the same purpose except that it will use environment variable PATH to determine which executable to process. Thus a fully qualified path name would not have to be used. The first argument to the function could instead be "ls". The function execvp() can also take the fully qualified name as it also resolves explicitly.

Man Pages:

execv / execvp - execute

execve():

The function call "execve()" executes a process in an environment which it assigns.

Set the environment variables:

Assignment:

char *env[] = { "USER=user1", "PATH=/usr/bin:/bin:/opt/bin", (char *) 0 };

Read from file:

#include <iostream>#include <fstream>

#include <string>#include <vector>

// Required by for routine#include <sys/types.h>#include <unistd.h>

using namespace std;

// Class definition:

class CReadEnvironmentVariablesFile{ public: CReadEnvironmentVariablesFile() { m_NumberOfEnvironmentVariables=0; }; ~CReadEnvironmentVariablesFile(); char **ReadFile(string& envFile); private: int m_NumberOfEnvironmentVariables; char **m_envp;};

Call execve:

string getErrMsg(int errnum);

main(){ string envFile("environment_variables.conf"); CReadEnvironmentVariablesFile readEnvFile; char **Env_envp = readEnvFile.ReadFile(envFile);

// Command to execute char *Env_argv[] = { "/bin/ls", "-l", "-a", (char *) 0 };

pid_t pID = fork(); if (pID == 0) // child { // This version of exec accepts environment variables. // Function call does not return on success.

int execReturn = execve (Env_argv[0], Env_argv, Env_envp);

cout << "Failure! execve error code=" << execReturn << endl;

cout << getErrMsg(execReturn) << endl;

_exit(0); // If exec fails then exit forked process. } else if (pID < 0) // failed to fork { cerr << "Failed to fork" << endl; }

else // parent { cout << "Parent Process" << endl; }}

Handle errors:

string getErrMsg(int errnum){

switch ( errnum ) {

#ifdef EACCES case EACCES : { return "EACCES Permission denied"; }#endif

#ifdef EPERM case EPERM : { return "EPERM Not super-user"; }#endif

#ifdef E2BIG case E2BIG : { return "E2BIG Arg list too long"; }#endif

#ifdef ENOEXEC case ENOEXEC : { return "ENOEXEC Exec format error"; }#endif

#ifdef EFAULT case EFAULT : { return "EFAULT Bad address"; }#endif

#ifdef ENAMETOOLONG case ENAMETOOLONG : { return "ENAMETOOLONG path name is too long "; }

#endif

#ifdef ENOENT case ENOENT : { return "ENOENT No such file or directory"; }#endif

#ifdef ENOMEM case ENOMEM : { return "ENOMEM Not enough core"; }#endif

#ifdef ENOTDIR case ENOTDIR : { return "ENOTDIR Not a directory"; }#endif

#ifdef ELOOP case ELOOP : { return "ELOOP Too many symbolic links"; }#endif

#ifdef ETXTBSY case ETXTBSY : { return "ETXTBSY Text file busy"; }#endif

#ifdef EIO case EIO : { return "EIO I/O error"; }#endif

#ifdef ENFILE case ENFILE : { return "ENFILE Too many open files in system"; }#endif

#ifdef EINVAL case EINVAL : { return "EINVAL Invalid argument"; }#endif

#ifdef EISDIR case EISDIR : { return "EISDIR Is a directory"; }#endif

#ifdef ELIBBAD case ELIBBAD : { return "ELIBBAD Accessing a corrupted shared lib"; }#endif default : { std::string errorMsg(strerror(errnum)); if ( errnum ) return errorMsg; } }}

Man Pages:

strerror / strerror_r - return string describing error code errno - number of last error perror - print a system error message

Data File: environment_variables.conf

PATH=/usr/bin:/binMANPATH=/opt/manLANG=CDISPLAY=:0.0

Man Pages:

execve - execute with given environment

Note: Don't mix malloc() and new. Choose one form of memory allocation and stick with it.

Malloc:

..

...

int ii;

m_envp = (char **) calloc((m_NumberOfEnvironmentVariables+1), sizeof(char **));

... // Allocate arrays of character strings. int ii; for(ii=0; ii < NumberOfEnvironmentVariables; ii++) { // NULL terminated m_envp[ii] = (char *) malloc(vEnvironmentVariables[ii].size()+1); strcpy( m_envp[ii], vEnvironmentVariables[ii].c_str()); }

// must terminate with null m_envp[ii] = (char*) 0;

return m_envp;}...

Free:

...

// Free array's of characters

for(ii=0; ii < m_NumberOfEnvironmentVariables; ii++){ free(m_envp[ii]);}

// Free array of pointers.

free(m_envp);

...

SocketSocket programming and the C BSD APIC and C++ sockets programming with examples of the BSD API on the Linux platform

Sockets are an inter-process network communication implementation using a Internet Protocol (IP) stack on an Ethernet transport. Sockets are language and protocol independent and available to "C", Perl, Python, Ruby and Java (and more) programmers. The "C" language BSD API is used on Linux, all popular variants of Unix, Microsoft Windows (NT,2000,XP,... and later) and even

embedded OSs like VxWorks. It is by far the most popular implementation of inter-process network communication.

Sockets allow one process to communicate with another whether it is local on the same computer system or remote over the network. Many other higher level protocols are built upon sockets technology.

The sockets API provides many configuration options so we will try and cover the socket API components and then give examples of a few implementations. It would be very difficult to cover all variations of its use.

Sockets utilize the following standard protocols:

Protocol Description

IP Internet Protocol provides network routing using IP addressing eg 192.168.1.204

UDP User Datagram Protocol - IP with ports to distinguish among processes running on same host. No data verification.

TCP Transmission Control Protocol - IP with ports to distinguish among processes running on same host. Connection oriented, stream transfer, full duplex, reliable with data verification.

BSD socket API:

Typically one configures a socket server to which a socket client may attach and communicate. The IP protocol layer will also require that the domain name or IP addresses of the communicating processes be made known as well. Within the IP protocol it is also important to provide the mechanism used: TCP or UDP.

The BSD is a "C" programming API. Examples shown are compiled using the GNU C++ compiler on Linux:

Basic steps in using a socket: Socket include files:

Include File Description

sys/types.h Types used in sys/socket.h and netinet/in.h

netinet/in.h Internet domain address structures and functions

netinet/tcp.h Socket option macro definitions, TCP headers, enums, etc

sys/socket.h Structures and functions used for socket API.i accept(), bind(), connect(), listen(), recv(), send(),

setsockopt(), shutdown(), etc ...

netdb.h Used for domain/DNS hostname lookup

sys/select.h Used by the select(), pselect() functions and defines FD_CLR, FD_ISSET, FD_SET, FD_ZERO macros

sys/time.h select() uses argument of type struct timeval and pselect() uses struct timespec defined by this include file.

arpa/inet.h Definitions for internet operations. Prototypes functions such as htonl(), htons(), ntohl(), ntohs(), inet_addr(), inet_ntoa(), etc ...

unistd.h Defines constants and types

errno.h Defines sytem error numbers

Create the socket instance:

Open a socket using TCP: Basic declarations and call to "socket".

#include <iostream>#include <sys/types.h> // Types used in sys/socket.h and netinet/in.h#include <netinet/in.h> // Internet domain address structures and functions#include <sys/socket.h> // Structures and functions used for socket API#include <netdb.h> // Used for domain/DNS hostname lookup#include <unistd.h>#include <errno.h>

using namespace std;

main(){ int socketHandle;

// create socket

if((socketHandle = socket(AF_INET, SOCK_STREAM, IPPROTO_IP)) < 0) { close(socketHandle); exit(EXIT_FAILURE); }

... ...}

Socket function prototype:

int socketHandle = socket(int socket_family, int socket_type, int protocol);

Choose socket communications family/domain:

o Internet IPV4: AF_INET o Internet IPV6: AF_INET6 o Unix path name (communicating processes are on the same

system): AF_UNIX

Choose socket type:

o TCP: SOCK_STREAM o UDP: SOCK_DGRAM o Raw protocol at network layer: SOCK_RAW

Choose socket protocol: (See /etc/protocols)

o Internet Protocol (IP): 0 or IPPROTO_IP o ICMP: 1 o ...

Also see:

o socket man page o protocols man page

Configure the socket as a client or server:

Comparison of sequence of BSD API calls:

Socket Server Socket Client

socket() socket()

bind() gethostbyname()

listen()

accept() connect()

recv()/send() recv()/send()

close() close()

This is specific to whether the application is a socket client or a socket server.

o Socket server: bind(): bind the socket to a local socket address. This

assigns a name to the socket. listen(): listen for connections on a socket created with

"socket()" and "bind()" and accept incoming connections.

This is used for TCP and not UDP. Zero is returned on success.

accept(): accept a connection on a socket. Accept the first connection request on the queue of pending connections, create a new connected socket with mostly the same properties as defined by the call to "socket()", and allocate a new file descriptor for the socket, which is returned. The newly created socket is no longer in the listening state. Note this call blocks until a client connects.

... ...#define MAXHOSTNAME 256 ... ...

struct sockaddr_in socketAddress; char sysHost[MAXHOSTNAME+1]; // Hostname of this computer we are running on struct hostNamePtr *hPtr; int portNumber = 8080;

bzero(&socketInfo, sizeof(sockaddr_in)); // Clear structure memory

// Get system information

gethostname(sysHost, MAXHOSTNAME); // Get the name of this computer we are running on if((hPtr = gethostbyname(sysHost)) == NULL) { cerr << "System hostname misconfigured." << endl; exit(EXIT_FAILURE); }

// Load system information into socket data structures

socketInfo.sin_family = AF_INET; // Use any address available to the system. This is a typical configuration for a server. // Note that this is where the socket client and socket server differ. // A socket client will specify the server address to connect to. socketInfo.sin_addr.s_addr = htonl(INADDR_ANY); // Translate long integer to network byte order. socketInfo.sin_port = htons(portNumber); // Set port number

// Bind the socket to a local socket address

if( bind(socketHandle, (struct sockaddr *) &socketInfo, sizeof(struct sockaddr_in)) < 0) { close(socketHandle); perror("bind"); exit(EXIT_FAILURE);

}

listen(socketHandle, 1);

int socketConnection; if( (socketConnection = accept(socketHandle, NULL, NULL)) < 0) { close(socketHandle); exit(EXIT_FAILURE); }

... ...

// read/write to socket here

o Socket functions: bind():

Function prototype:

int bind(int sockfd, struct sockaddr *my_addr, socklen_t addrlen);

Bind arguments:

int sockfd: Socket file descriptor. Returned by call to "socket".

struct sockaddr: Socket information structure socklen_t addrlen: Size of structure Returns 0: Sucess, -1: Failure and errno may be set.

Also see the bind man page

listen(): Function prototype:

int listen(int s, int backlog);

Listen arguments:

int s: Socket file descriptor. Returned by call to "socket". Identifies a bound but unconnected socket.

int backlog: Set maximum length of the queue of pending connections for the listening socket. A reasonable value is 10. Actual maximum permissible: SOMAXCONN Example: int iret = listen(socketHandle, SOMAXCONN); The include file sys/socket.h will include /usr/include/bits/socket.h which defines the default value for SOMAXCONN as 128. The actual value set for the operating system: cat

/proc/sys/net/core/somaxconn In kernels before 2.4.25, this limit was a hard coded value and thus would require a kernel recompile with the SOMAXCONN value as defined in /usr/include/linux/socket.h

For very heavy server use, modify the system limit in the proc file and set "backlog" to the same value (eg. 512).

Returns 0: Sucess, -1: Failure and errno may be set.

Also see the listen man page

accept(): Function prototype:

int accept(int s, struct sockaddr *addr, socklen_t *addrlen);

Accept arguments:

int s: Socket file descriptor. Returned by call to "socket".

struct sockaddr *addr: Pointer to a sockaddr structure. This structure is filled in with the address of the connecting entity.

socklen_t *addrlen: initially contains the size of the structure pointed to by addr; on return it will contain the actual length (in bytes) of the address returned. When addr is NULL nothing is filled in.

Returns: Success: non-negative integer, which is a

descriptor of the accepted socket. Argument "addrlen" will have a return value.

Fail: -1, errno may be set

Also see the accept man page

o [Potential Pitfall]: If you get the following message: o bind: Address already in use

o The solution is to choose a different port or kill the process which is using the port and creating the conflict. You may have to be root to see all processes with netstat.

o netstat -punta | grep 8080

o tcp 0 0 :::8080 :::* LISTEN

o Socket client: connect(): initiate a connection with a remote entity on a

socket. Zero is returned on success. Support both TCP (SOCK_STREAM) and UDP (SOCK_DGRAM). For SOCK_STREAM, an

actual connection is made. For SOCK_DGRAM the address is the address to which datagrams are sent and received.

...

...

struct sockaddr_in remoteSocketInfo;struct hostent *hPtr;int socketHandle;char *remoteHost="dev.megacorp.com";int portNumber = 8080;

bzero(&remoteSocketInfo, sizeof(sockaddr_in)); // Clear structure memory

// Get system information

if((hPtr = gethostbyname(remoteHost)) == NULL){ cerr << "System DNS name resolution not configured properly." << endl; cerr << "Error number: " << ECONNREFUSED << endl; exit(EXIT_FAILURE);}

// Load system information for remote socket server into socket data structures

memcpy((char *)&remoteSocketInfo.sin_addr, hPtr->h_addr, hPtr->h_length);remoteSocketInfo.sin_family = AF_INET;remoteSocketInfo.sin_port = htons((u_short)portNumber); // Set port number

if( (connect(socketHandle, (struct sockaddr *)&remoteSocketInfo, sizeof(sockaddr_in)) < 0){ close(socketHandle); exit(EXIT_FAILURE);}

...

...

o Connect function prototype: int connect(int sockfd, const struct sockaddr *serv_addr, socklen_t addrlen);

Connect arguments: (Same as server's bind() arguments) int sockfd: Socket file descriptor. Returned by call to

"socket". struct sockaddr: Socket information structure socklen_t addrlen: Size of structure Returns 0: Sucess, -1: Failure and errno may be set.

Zero is returned upon success and on error, -1 and errno is set appropriately. Also see the connect man page

o The sockaddr_in data structure: /usr/include/linux/in.h

/* Internet address. */struct in_addr { __u32 s_addr; /* Defined as 32 or 64 bit address (system dependent) */};

/* Structure describing an Internet (IP) socket address. */#define __SOCK_SIZE__ 16 /* sizeof(struct sockaddr) */struct sockaddr_in { sa_family_t sin_family; /* Address family */ unsigned short int sin_port; /* Port number */ struct in_addr sin_addr; /* Internet address */

/* Pad to size of `struct sockaddr'. */ unsigned char __pad[__SOCK_SIZE__ - sizeof(short int) - sizeof(unsigned short int) - sizeof(struct in_addr)];};

o Note:

IP addresses: Note the specific IP address could be specified:

#include <arpa/inet.h> // IP from string conversion socketInfo.sin_addr.s_addr = inet_addr("127.0.0.1"); cout << inet_ntoa(socketInfo.sin_addr) << endl;

or bind to all network interfaces available:

socketInfo.sin_addr.s_addr = htonl(INADDR_ANY);

Port numbers: Note the specific port can be specified: socketInfo.sin_port = htons(8080); cout << ntohs(socketInfo.sin_port) << endl;

or let the system define one for you:

socketInfo.sin_port = htons(INADDR_ANY);

Read from or write to socket:

Use send() and recv(), or write() and read(), or sendto() and recvfrom() to read/write to/from a socket.

o TCP recv() or UDP recvfrom(): receive a message from a socket

char *pcIpAddress;unsigned short shPort;

...

...

if (iSocketType == SOCK_STREAM){ rc = recv(socketHandle, (char*) _pcMessage, (int) _iMessageLength, 0); if ( rc == 0 ) { cerr << "ERROR! Socket closed" << endl; } else if (rc == -1) { cerr << "ERROR! Socket error" << endl; closeSocket(); }}else if (iSocketType == SOCK_DGRAM){ int iLength; struct sockaddr_in stReceiveAddr; iLength = (int) sizeof(struct sockaddr_in); memset((void*) &stReceiveAddr, 0, iLength);

rc = recvfrom(socketHandle, (char*) _pcMessage, (int) _iMessageLength, 0, (struct sockaddr *) &stReceiveAddr, (socklen_t*) &iLength)) if{ rc == 0 ) { cerr << "ERROR! Socket closed" << endl; } else if (rc == -1) { cerr << "ERROR! Socket error" << endl; closeSocket(); }}

pcIpAddress = inet_ntoa(stReceiveAddr.sin_addr);shPort = ntohs(stReceiveAddr.sin_port);

cout << "Socket Received: " << _iNumRead << " bytes from " << pcIpAddress << ":" << shPort << endl;

...

...

read(): read a specific number of bytes from a file descriptorint rc = 0; // Actual number of bytes read by function read()int count = 0; // Running total "count" of bytes readint numToRead = 32; // Number of bytes we want to read each passchar buf[512];

...

...

while(bcount < numToRead){ // rc is the number of bytes returned. if( (rc = read(socketHandle, buf, numToRead - count)) > 0); { count += rc; buf += rc; // Set buffer pointer for next read } else if(rc < 0) { close(socketHandle); exit(EXIT_FAILURE); }

cout << "Number of bytes read: " << count << endl; cout << "Received: " << buf << endl;}

...

...

send(): send a message from a socket. Used only when in a connected state. The only difference between send and write is the presence of flags. With zero flags parameter, send is equivalent to write.#include <string.h>

... ...

char buf[512]; strcpy(buf,"Message to send");

... ...

send(socketHandle, buf, strlen(buf)+1, 0); ...

...TCP send() or UDP sendto():int iSocketType;int iBytesSent = 0;

char *pMessage = "message to send";int iMessageLength = 16; // number of bytes (includes NULL termination)sockaddr pSendAddress;

...

...

if (iSocketType == SOCK_STREAM){ if ((iBytesSent = send(socketHandle, (char*) pMessage, (int) iMessageLength, 0)) < 0 ) { cerr << "Send failed with error " << errno << endl; close(socketHandle); }}else if (iSocketType == SOCK_DGRAM){ if ((iBytesSent = sendto(socketHandle, (char*) pMessage, (int) iMessageLength, 0, (struct sockaddr*) pSendAddress, (int) sizeof(struct sockaddr_in))) < 0 ) { cerr << "Sendto failed with error " << errno << endl; close(); }}else{ // Failed - Socket type not defined}

...

...

Close the socket when done:#include <unistd.h>

...

close(socketHandle);

...

Socket Server:

Simple Socket Server:

#include <iostream>#include <sys/types.h>#include <sys/socket.h>#include <netdb.h>#define MAXHOSTNAME 256

using namespace std;

main(){ struct sockaddr_in socketInfo; char sysHost[MAXHOSTNAME+1]; // Hostname of this computer we are running on struct hostent *hPtr; int socketHandle; int portNumber = 8080;

bzero(&socketInfo, sizeof(sockaddr_in)); // Clear structure memory

// Get system information

gethostname(sysHost, MAXHOSTNAME); // Get the name of this computer we are running on if((hPtr = gethostbyname(sysHost)) == NULL) { cerr << "System hostname misconfigured." << endl; exit(EXIT_FAILURE); }

// create socket

if((socketHandle = socket(AF_INET, SOCK_STREAM, 0)) < 0) { close(socketHandle); exit(EXIT_FAILURE); }

// Load system information into socket data structures

socketInfo.sin_family = AF_INET; socketInfo.sin_addr.s_addr = htonl(INADDR_ANY); // Use any address available to the system socketInfo.sin_port = htons(portNumber); // Set port number

// Bind the socket to a local socket address

if( bind(socketHandle, (struct sockaddr *) &socketInfo, sizeof(socketInfo)) < 0) { close(socketHandle); perror("bind"); exit(EXIT_FAILURE); }

listen(socketHandle, 1);

int socketConnection; if( (socketConnection = accept(socketHandle, NULL, NULL)) < 0) { exit(EXIT_FAILURE); } close(socketHandle);

int rc = 0; // Actual number of bytes read char buf[512];

// rc is the number of characters returned. // Note this is not typical. Typically one would only specify the number // of bytes to read a fixed header which would include the number of bytes // to read. See "Tips and Best Practices" below.

rc = recv(socketConnection, buf, 512, 0); buf[rc]= (char) NULL; // Null terminate string

cout << "Number of bytes read: " << rc << endl; cout << "Received: " << buf << endl;}

Forking Socket Server:

In order to accept connections while processing previous connections, use fork() to handle each connection.

Use establish() and get_connection() to allow multiple connections.

File: serverFork.cpp #include <iostream>#include <sys/types.h>#include <sys/socket.h>#include <sys/wait.h>#include <netdb.h>#include <errno.h>#include <unistd.h>#include <signal.h>#include <netinet/in.h>#define MAXHOSTNAME 256using namespace std;

// Catch signals from child processesvoid handleSig(int signum){ while(waitpid(-1, NULL, WNOHANG) > 0);}

main(){ struct sockaddr_in socketInfo; char sysHost[MAXHOSTNAME+1]; // Hostname of this computer we are running on struct hostent *hPtr; int socketHandle; int portNumber = 8080;

signal(SIGCHLD, handleSig);

bzero(&socketInfo, sizeof(sockaddr_in)); // Clear structure memory // Get system information

gethostname(sysHost, MAXHOSTNAME); // Get the name of this computer we are running on if((hPtr = gethostbyname(sysHost)) == NULL) { cerr << "System hostname misconfigured." << endl; exit(EXIT_FAILURE); }

// create socket

if((socketHandle = socket(AF_INET, SOCK_STREAM, 0)) < 0) { close(socketHandle); exit(EXIT_FAILURE); }

// Load system information into socket data structures

socketInfo.sin_family = AF_INET; socketInfo.sin_addr.s_addr = htonl(INADDR_ANY); // Use any address available to the system socketInfo.sin_port = htons(portNumber); // Set port number

// Bind the socket to a local socket address

if( bind(socketHandle, (struct sockaddr *) &socketInfo, sizeof(struct sockaddr_in)) < 0) { close(socketHandle); perror("bind"); exit(EXIT_FAILURE); }

listen(socketHandle, 1);

int socketConnection; for(;;) // infinite loop to handle remote connections. This should be limited. { if( (socketConnection = accept(socketHandle, NULL, NULL)) < 0) { close(socketHandle); if(errno == EINTR) continue; perror("accept"); exit(EXIT_FAILURE); } switch(fork()) { case -1: perror("fork"); close(socketHandle); close(socketConnection);

exit(EXIT_FAILURE); case 0: // Child process - do stuff close(socketHandle); // Do your server stuff like read/write messages to the socket here! exit(0); default: // Parent process, look for another connection close(socketConnection); continue; } }

}

For more on the use of the fork() function see the YoLinux.com fork() tutorial.

Socket Client:

Simple Socket client:

File: client.cpp #include <iostream>#include <string.h>#include <sys/types.h>#include <sys/socket.h>#include <netdb.h>#include <unistd.h>#include <errno.h>#define MAXHOSTNAME 256using namespace std;

main(){ struct sockaddr_in remoteSocketInfo; struct hostent *hPtr; int socketHandle; char *remoteHost="localhost"; int portNumber = 8080;

bzero(&remoteSocketInfo, sizeof(sockaddr_in)); // Clear structure memory

// Get system information

if((hPtr = gethostbyname(remoteHost)) == NULL) { cerr << "System DNS name resolution not configured properly." << endl; cerr << "Error number: " << ECONNREFUSED << endl; exit(EXIT_FAILURE); }

// create socket

if((socketHandle = socket(AF_INET, SOCK_STREAM, 0)) < 0)

{ close(socketHandle); exit(EXIT_FAILURE); }

// Load system information into socket data structures

memcpy((char *)&remoteSocketInfo.sin_addr, hPtr->h_addr, hPtr->h_length); remoteSocketInfo.sin_family = AF_INET; remoteSocketInfo.sin_port = htons((u_short)portNumber); // Set port number

if(connect(socketHandle, (struct sockaddr *)&remoteSocketInfo, sizeof(sockaddr_in)) < 0) { close(socketHandle); exit(EXIT_FAILURE); }

int rc = 0; // Actual number of bytes read by function read() char buf[512];

strcpy(buf,"Message to send"); send(socketHandle, buf, strlen(buf)+1, 0);}

Test Simple Client and Server Socket program:

Note that this runs on a single system using "localhost".

Compile the simple client and the simple server: g++ server.cpp -o server g++ client.cpp -o client

Start the server: ./server This will block at the "accept()" call and await a connection from a client.

Start the client: ./client This will connect to the server and write the message "Message to send". The server will receive the message and write it out.

Name Resolution and Network Information Lookup:Network information data lookup for:

DNS: Name resolution associates IP address to a hostname using DNS (Domain Name System) name servers. Name resolution invokes a series of library routines to query the name servers.

TCP/IP ports and associated services Network Protocols Network name information

Function Description

gethostname(char *name, size_t len) returns hostname of local host

getservbyname(const char *name,const char *proto)

returns a structure of type servent for the given host name and protocol

gethostbyname(const char *name) returns a structure of type hostent for the given host name

getservbyport(int port, const char *proto)

returns a servent structure for the line that matches the port port given in network byte order using protocol proto. If proto is NULL, any protocol will be matched.

getservent(void) returns a structure servent containing the broken out fields from the line in /etc/services

getprotobyname(const char *name) returns a structure protoent containing the broken out fields from the line in /etc/protocols

getprotobynumber(int proto) returns a protoent structure for the line that matches the protocol number

getprotoent(void) returns a structure protoent containing the broken out fields from the line in /etc/protocols

getnetbyname(const char *name) a structure netent containing the broken out fields from the line in /etc/networks

getnetbyaddr(long net, int type) returns a netent structure for the line that matches the network number net of type "type"

getaddrinfo(const char *node, const char *service, const struct addrinfo *hints, struct addrinfo **res)void freeaddrinfo(struct addrinfo *res)

Returns 0 if it succeeds or an error code. Network address and service translationfreeaddrinfo() frees the memory that was allocated by getaddrinfo()

getnetent(void) returns a structure netent containing the broken out fields from the line in /etc/networks

Data Structures returned:

hostent defined in <netdb.h>struct hostent { char *h_name; /* official name of host */ char **h_aliases; /* alias list */ int h_addrtype; /* host address type */ int h_length; /* length of address */ char **h_addr_list; /* list of addresses */}

Lookup of data in /etc/hosts or from DNS resolution.

servent defined in <netdb.h>struct servent { char *s_name; /* official service name */ char **s_aliases; /* alias list */ int s_port; /* port number */ char *s_proto; /* protocol to use */}

Lookup of data in /etc/services

protoent defined in <netdb.h>struct protoent { char *p_name; /* official protocol name */ char **p_aliases; /* alias list */ int p_proto; /* protocol number */}

Lookup of data in /etc/protocols

netent defined in <netdb.h>struct netent { char *n_name; /* official network name */ char **n_aliases; /* alias list */ int n_addrtype; /* net address type */ unsigned long int n_net; /* network number */}

Lookup of data in /etc/networks

Socket Configuration Options:

Socket options:

One can "get" (read) the current socket options or "set" them to new values. The default values are obtained from the OS:

Level Option Type Default Description

IPPROTO_IP TCP_NODELAY int 0 Don't delay send to coalesce packets. If set, disable the Nagle algorithm. When not set, data is buffered until there is a sufficient amount to send out, thereby avoiding the frequent sending of small packets,

which results in poor utilization of the network. Don't use with TCP_CORK. This option is overridden by TCP_CORK

IPPROTO_IP TCP_MAXSEG int 536 Maximum segment size for outgoing TCP packets. TCP will impose its minimum and maximum bounds over the value provided.

IPPROTO_IP TCP_CORK int 0 Control sending of partial frames. If set, don't send out partial frames. Not cross platform.

IPPROTO_IP TCP_KEEPIDLE int 7200 When the SO_KEEPALIVE option is enabled, TCP probes a connection that has been idle for some amount of time. The default value for this idle period is 2 hours. The TCP_KEEPIDLE option can be used to affect this value for a given socket, and specifies the number of seconds of idle time between keepalive probes. Not cross platform. This option takes an int value, with a range of 1 to 32767.

IPPROTO_IP TCP_KEEPINTVL int 75 Specifies the interval between packets that are sent to validate the connection.Not cross platform.

IPPROTO_IP TCP_KEEPCNT int 9 When the SO_KEEPALIVE option is enabled, TCP probes a connection that has been idle for some amount of time. If the remote system does not respond to a keepalive probe, TCP retransmits the probe a certain number of times before a connection is considered to be broken. The TCP_KEEPCNT option can be used to affect this value for a given socket, and specifies the maximum number of keepalive probes to be sent. This option takes an int value, with a range of 1 to 32767. Not cross platform.

IPPROTO_IP TCP_SYNCNT int 5 Number of SYN retransmits that TCP should send before aborting the attempt to connect. It cannot exceed 255.

IPPROTO_IP TCP_LINGER2 int 60 Life time of orphaned FIN-WAIT-2 state. Not

to be confused with option SO_LINGERNot cross platform.

SOL_SOCKET SO_REUSEADDR int(bool)

0 Allow local address reuse. If a problem is encountered when attempting to bind to a port which has been closed but not released (may take up to 2 minutes as defined by TIME_WAIT). Apply the SO_REUSEADDR socket option to release the resource immediately and to get around the TIME_WAIT state.0 = disables, 1 = enables

SOL_SOCKET SO_REUSEPORT int(bool)

0 This option is AF_INET socket-specific. This option allows multiple processes to share a port. All incoming multicast or broadcast UDP datagrams that are destined for the port are delivered to all sockets that are bound to the port. All processes that share the port must specify this option.0 = disables, 1 = enables

SOL_SOCKET SO_ERROR int(bool)

0 When an error occurs on a socket, set error variable

so_error and notify process0 = disables, 1 = enables

SOL_SOCKET SO_BROADCAST int(bool)

0 Permit sending of broadcast datagrams0 = disables, 1 = enables

SOL_SOCKET SO_SNDBUF int(value)

16384 Send buffer size

SOL_SOCKET SO_RCVBUF int(value)

87380 Receive buffer size

SOL_SOCKET SO_KEEPALIVE int(bool)

0 Periodically test if connection is alive0 = disables, 1 = enables

SOL_SOCKET SO_SNDTIMEO timeval(struct)

00

Set timeout period for socket send.Disable by setting timeval.tv_sec = 0 sec, timeval.tv_usec = 0 usec (default)Affects write() writev() send() sendto() and sendmsg()

SOL_SOCKET SO_RCVTIMEO timeval(struct)

00

Set timeout period for socket receive.Disable by setting timeval.tv_sec = 0 sec, timeval.tv_usec = 0 usec (default)Affects read() readv() recv() recvfrom() and recvmsg()

SOL_SOCKET SO_LINGER linger(struct)

00

Specifies how close function will operate for connection

protocols (TCP)l_onoff: 0 = disables, 1 = enablesl_linger: 0 = unsent data discarded, 1 = close() does not return untill all unsent data is transmitted or remote connection is closedStructure defined in sys/socket.h

SOL_SOCKET SO_RCVLOWAT int(value)

1 Specifies number of bytes used as a threshold by select() to consider a socket read ready

SOL_SOCKET SO_SNDLOWAT int(value)

1 Specifies number of bytes used as a threshold by select() to consider a socket write ready

SOL_SOCKET SO_TYPE int(value)

undefined Specifies socket type (e.g., tcp (SOCK_STREAM), udp (SOCK_DGRAM), etc.) For use with getsockopt() only.

IPPROTO_IP macro defines are found in /usr/include/netinet/tcp.h SOL_SOCKET macro defines require /usr/include/sys/socket.h

For a full list of options see the TCP man page

For a full list of IP options see the IP(7) man page

Function Prototypes: int getsockopt(int s, int level, int optname, void *optval, socklen_t *optlen);

int setsockopt(int s, int level, int optname, const void *optval, socklen_t optlen);

getsockopt/setsockopt arguments: int sockfd: Socket file descriptor. Returned by call to "socket". int level: See table above int optname: See table above void *optval: Pointer to value or data structure optlen: Length of "optval" Returns 0: Sucess, -1: Failure and errno may be set.

Code to read socket options:

File: printSocketOptions.c#include <sys/socket.h>#include <netinet/in.h>#include <netinet/tcp.h>#include <errno.h>#include <stdio.h>

int main(){ int socketHandle;

// create socket

if((socketHandle = socket(AF_INET, SOCK_STREAM, IPPROTO_IP)) < 0) { close(socketHandle); perror("socket"); }

int iSocketOption = 0; int iSocketOptionLen = sizeof(int);;

struct linger SocketOptionLinger; int iSocketOptionLingerLen = sizeof(struct linger);;

getsockopt(socketHandle, IPPROTO_TCP, TCP_NODELAY, (char *)&iSocketOption, &iSocketOptionLen); printf("Socket TCP_NODELAY = %d\n", iSocketOption);

getsockopt(socketHandle, IPPROTO_TCP, TCP_MAXSEG, (char *)&iSocketOption, &iSocketOptionLen); printf("Socket TCP_MAXSEG = %d\n", iSocketOption);

getsockopt(socketHandle, IPPROTO_TCP, TCP_CORK, (char *)&iSocketOption, &iSocketOptionLen); printf("Socket TCP_CORK = %d\n", iSocketOption);

getsockopt(socketHandle, IPPROTO_TCP, TCP_KEEPIDLE, (char *)&iSocketOption, &iSocketOptionLen); printf("Socket TCP_KEEPIDLE = %d\n", iSocketOption);

getsockopt(socketHandle, IPPROTO_TCP, TCP_KEEPINTVL, (char *)&iSocketOption, &iSocketOptionLen); printf("Socket TCP_KEEPINTVL = %d\n", iSocketOption);

getsockopt(socketHandle, IPPROTO_TCP, TCP_KEEPCNT, (char *)&iSocketOption, &iSocketOptionLen); printf("Socket TCP_KEEPCNT = %d\n", iSocketOption);

getsockopt(socketHandle, IPPROTO_TCP, TCP_SYNCNT, (char *)&iSocketOption, &iSocketOptionLen); printf("Socket TCP_SYNCNT = %d\n", iSocketOption);

getsockopt(socketHandle, IPPROTO_TCP, TCP_LINGER2, (char *)&iSocketOption, &iSocketOptionLen); printf("Socket TCP_LINGER2 = %d\n", iSocketOption);

getsockopt(socketHandle, SOL_SOCKET, SO_REUSEADDR, (char *)&iSocketOption, &iSocketOptionLen); printf("Socket SO_REUSEADDR = %d\n", iSocketOption);

getsockopt(socketHandle, SOL_SOCKET, SO_ERROR, (char *)&iSocketOption, &iSocketOptionLen); printf("Socket SO_ERROR = %d\n", iSocketOption);

getsockopt(socketHandle, SOL_SOCKET, SO_BROADCAST, (char *)&iSocketOption, &iSocketOptionLen); printf("Socket SO_BROADCAST = %d\n", iSocketOption);

getsockopt(socketHandle, SOL_SOCKET, SO_KEEPALIVE, (char *)&iSocketOption, &iSocketOptionLen); printf("Socket SO_KEEPALIVE = %d\n", iSocketOption);

getsockopt(socketHandle, SOL_SOCKET, SO_SNDBUF, (char *)&iSocketOption, &iSocketOptionLen); printf("Socket SO_SNDBUF = %d\n", iSocketOption);

getsockopt(socketHandle, SOL_SOCKET, SO_RCVBUF, (char *)&iSocketOption, &iSocketOptionLen); printf("Socket SO_RCVBUF = %d\n", iSocketOption);

getsockopt(socketHandle, SOL_SOCKET, SO_LINGER, (char *)&SocketOptionLinger, &iSocketOptionLingerLen); printf("Socket SO_LINGER = %d time = %d\n", SocketOptionLinger.l_onoff, SocketOptionLinger.l_linger);

getsockopt(socketHandle, SOL_SOCKET, SO_RCVLOWAT, (char *)&iSocketOption, &iSocketOptionLen); printf("Socket SO_RCVLOWAT = %d\n", iSocketOption);}

Compile: gcc -o printSocketOptions printSocketOptions.c

getsockopt man page: get a particular socket option for the specified socket.

Set socket options:

Socket "keep-alive":

int iOption = 1; // Turn on keep-alive, 0 = disables, 1 = enables

if (setsockopt(socketHandle, SOL_SOCKET, SO_KEEPALIVE, (const char *) &iOption, sizeof(int)) == SOCKET_ERROR){ cerr << "Set keepalive: Keepalive option failed" << endl;}

Set socket client options:

Socket re-use:

int iOption = 0; // Reuse address option to set, 0 = disables, 1 = enables

if (setsockopt(socketHandle, SOL_SOCKET, SO_REUSEADDR, (const char *) &iOption, sizeof(int)) == SOCKET_ERROR){ cerr << "Set reuse address: Client set reuse address option failed" << endl;}

When a socket connection is closed with a call to close(), shutdown() or exit(), both the client and server will send a FIN (final) packet and will then send an acknowledgment (ACK) that they received the packet. The side which initiates the closure will be in a TIME_WAIT state until the process has been completed. This time out period is generally 2-4 minutes in duration. It is hoped that all packets are received in a timely manner and the entire time out duration is not required. When an application is abnormally terminated, the TIME_WAIT period is entered for the full duration.

Setting the SO_REUSEADDR option explicitly allows a process to bind a port in the TIME_WAIT state. This is to avoid the error "bind: Address Already in Use". One caviat is that the process can not be to the same address and port as the previous connection. If it is, the SO_REUSEADDR option will not help and the duration of the TIME_WAIT will be in effect. For more info see How to avoid the "Address Already in Use" error.

Solution: Enable socket linger:

linger Option;Option.l_onoff = 1;Option.l_linger = 0;

if(setsockopt(socketHandle, SOL_SOCKET, SO_LINGER, (const char *) &Option, sizeof(linger)) == -1){ cerr << "Set SO_LINGER option failed" << endl;}

This allows the socket to die quickly and allow the address to be reused again. Warning: This linger configuration specified may/will result in data loss upon socket termination, thus it would not have the robustness required for a banking transaction but would be ok for a recreational app.

Broadcast:

int iOption = 0; // Broadcast option to set, 0 = disables, 1 = enables

if (setsockopt(socketHandle, SOL_SOCKET, SO_BROADCAST, (const char *) &iOption, sizeof(int)) == SOCKET_ERROR){ cerr << "Set reuse address: Client set reuse address option failed" << endl;}

Struct: remoteSocketInfo.sin_addr.s_addr = htonl(INADDR_BROADCAST);

setsockopt man page: set a particular socket option for the specified socket.

Test Socket Availability:

Function to test if a socket or set of sockets has data and can be read (Test so you don't get blocked on a read) or written.

#include <sys/select.h>#include <sys/time.h>

...

...

bool isReadyToRead(int _socketHandle, const long &_lWaitTimeMicroseconds){ int iSelectReturn = 0; // Number of sockets meeting the criteria given to select() timeval timeToWait; int fd_max = -1; // Max socket descriptor to limit search plus one. fd_set readSetOfSockets; // Bitset representing the socket we want to read // 32-bit mask representing 0-31 descriptors where each // bit reflects the socket descriptor based on its bit position.

timeToWait.tv_sec = 0; timeToWait.tv_usec = _lWaitTimeMicroseconds;

FD_ZERO(&readSetOfSockets); FD_SET(_socketHandle, &readSetOfSockets);

if(_socketHandle > fd_max) { fd_max = _socketHandle; }

iSelectReturn = select(fd_max + 1, &readSetOfSockets, (fd_set*) 0, (fd_set*) 0, &timeToWait);

// iSelectReturn -1: ERROR, 0: no data, >0: Number of descriptors found which pass test given to select() if ( iSelectReturn == 0 ) // Not ready to read. No valid descriptors { return false; } else if ( iSelectReturn < 0 ) // Handle error { cerr << "*** Failed with error " << errno << " ***" << endl;

close(_socketHandle); return false; }

// Got here because iSelectReturn > 0 thus data available on at least one descriptor // Is our socket in the return list of readable sockets if ( FD_ISSET(_socketHandle, &readSetOfSockets) ) { return true; } else { return false; }

return false;}

Arguments to select(): 1. int fd_max: Highest socket descriptor number. When opening a

socket with the call socket(), the function returns a socket descriptor number. The call to select() will loop through all of the socket descriptors from zero up to fd_max to perform the "test".

2. fd_set *readSetOfSockets: This is a pointer to the variable holding the set of bits representing the set of sockets to test for readability. (Read will not block) The default number of bytes detected for the socket to be

considered ready to read is 1. To change this default use (in this example 8 bytes):

int nBytes = 8; setsockopt(socketHandle, SOL_SOCKET, SO_RCVLOWAT,(const char *)

&nBytes, sizeof(int)) 3. fd_set *writeSetOfSockets: This is a pointer to the variable

holding the set of bits representing the set of sockets to test for writeability. (Write will not block)

4. fd_set *exceptfds: This is a pointer to the variable holding the set of bits representing the set of sockets to test for exceptions.

5. struct timeval *timeout: This structure holds the upper bound on the amount of time elapsed before select returns. It may be zero, causing select to return immediately. If timeout is NULL (no timeout), select can block indefinitely.

6. struct timeval {7. long tv_sec; /* seconds */8. long tv_usec; /* microseconds */9. };

Note: Any of the tests (read/write/exceptions) can be set to NULL to ignore that test.

Also see the select man page

Port Numbers:

The use of port numbers below 1024 are limited to the root process. For a list of port numbers with established uses see the file /etc/services.

Posts are 16 bit identifiers. Many are reserved and managed by the Internet Assigned Numbers Authority (IANA). See RFC 1700.

Host and Network Byte Order:

Note that when transferring data between different platforms or with Java, the byte order endianess will have to be considered. The network (the neutral medium) byte order is Big Endian and the byte order to which data is usually marshalled.

Host processor byte order:

Host Processor Endianness

Intel x86 processor family Little endian

Power PC processor family Big endian

SUN SPARC Big endian

Mips Big endian (IRIX)

Mips Little endian (NT)

Note that it is the processor architecture which determines the endianness and NOT the OS. The exception is for processors which support both big and little endian byte ordering, such as the MIPS processor.

Also note: Java processes and stores data in big endian byte ordering on any platform.

Character data is not subject to this problem as each character is one byte in length but integer is.

Integer data can be converted from/to host or network byte order with the following routines:

Function Description

ntohl()Network to host byte order conversion for long integer data (uint32_t)

ntohs()Network to host byte order conversion for short integer data (uint16_t)

htonl()Host to network byte order conversion for long integer data (uint32_t)

htons()Host to network byte order conversion for short integer data (uint16_t)

Requires #include <arpa/inet.h> The routines are aware of the processor they are running on and thus either perform a conversion or not, depending if it is required.

Man pages: ntohl/htonl, ntohs/htons: convert values between host and network byte

order

Note that one uses these calls for portable code on any platform. The port of the library to that platform determines whether a byte swap will occur and not your source code.

Include file/code snipit to determine the processor endianess: File: is_bigendian.h

#ifndef __IS_BIGENDIAN__#define __IS_BIGENDIAN__

const int bsti = 1; // Byte swap test integer

#define is_bigendian() ( (*(char*)&bsti) == 0 )

#endif // __IS_BIGENDIAN__

Code snipit to swap bytes "in place" and convert endian:

#include <netdb.h>#include "is_bigendian.h"

/** In-place swapping of bytes to match endianness of hardware

@param[in/out] *object : memory to swap in-place @param[in] _size : length in bytes*/void swapbytes(void *_object, size_t _size){ unsigned char *start, *end;

if(!is_bigendian()) { for ( start = (unsigned char *)_object, end = start + _size - 1; start < end; ++start, --end ) { unsigned char swap = *start; *start = *end; *end = swap; } }}

Swaping bit field structures:

#include <netdb.h>#include "is_bigendian.h"

void swapbytes(void *_object, size_t _size);

struct ExampleA{#ifdef BIGENDIAN unsigned int a:1; unsigned int b:1; unsigned int c:1; unsigned int d:1; unsigned int e:4; unsigned int f:8; unsigned int g:8; unsigned int h:8;#else unsigned int h:8; unsigned int g:8; unsigned int f:8; unsigned int e:4; unsigned int d:1;

unsigned int c:1; unsigned int b:1; unsigned int a:1;#endif};

...

...

// Bits: // |B31....B25 |B24....B16 |B15....B8 |B7....B0 | Big endian // |B7....B0 |B15....B8 |B24....B16 |B31....B25 | Little endian

// Intel host to network: // Reverse bit field structure and then byte swap. // Just byteswap for int.

// Code body

struct ExampleA exampleA; char tmpStore[4];

...

// assign member variables: exampleA.a = 0; exampleA.b = 0; exampleA.c = 1; exampleA.d = 1; exampleA.e = 3; ...

// Use memcpy() because we can't cast a bitfield memcpy(&tmpStore, &exampleA, sizeof(ExampleA));

swapbytes((void *)&tmpStore, sizeof(ExampleA));

...

A bit reversal routine (use as needed):

unsigned long reverseBitsUint32l(unsigned long x) { unsigned long tmplg = 0; int ii = 0;

for(h = ii = 0; ii < 32; ii++) { h = (tmplg << 1) + (x & 1); x >>= 1; }

return tmplg;}

Socket Tips and Best Practices: Re-connect: If a socket client attempts to connect to a socket server and

fails, one should attempt to re-attach after a given waiting period, a fixed number of times.

...

...

int iTry = 0;int mRetrymax = 10;int mRetrywait = 2;

// If this client can not connect to the server, try again after period "Retrywait".

while ( connect(socketHandle, (struct sockaddr *)&remoteSocketInfo, sizeof(sockaddr_in)) < 0 ){ iTry++; if( iTry > mRetrymax ) { cerr << "Failed to connect to server. Exceeded maximum allowable attempts: " << mRetrymax << endl; break; // Done retrying! } sleep( mRetrywait );}

...

Check return codes: Just because the socket was opened doesn't mean it will stay open. Dropped connections do happen. Check all read/write socket API function return codes:

o return code > 0: Data received/sent o return code == 0: Socket closed o return code == -1: Check system errno and interpret (or call

"perror()")

For even more robust code for a socket client, close, then open a new socket connection for return codes 0 or -1.

Read entire message: (Variable and fixed length messages) When sending messages using a socket, be aware that the other side of connection must know how much to read. This means that you must:

o Send fixed message sizes of a known number of bytes where all messages sent are of the same size or

o Send a message with a header of a known size where the message size is sent in the header. Thus read the fixed size header and

determine the size of the whole message then read the remaining bytes.

Note that with a UDP client/server communications, the socket read will read the whole frame sent. This is not necessarily true for a TCP stream. The function used to read the stream will return the number of bytes read. If the required number of bytes have not been read, repeat the read for the remaining bytes until the whole message has been read.

In this example, our message header is a single integer used to store the message size in the first 32 bits of the message.

Check return codes: Just because the socket was opened doesn't mean it will stay open. Dropped connections do happen. Check all read/write socket API function return codes:

o return code > 0: Data received/sent o return code == 0: Socket closed o return code == -1: Check system errno and interpret (or call

"perror()")

For even more robust code for a socket client, close, then open a new socket connection for return codes 0 or -1.

Read entire message: (Variable and fixed length messages) When sending messages using a socket, be aware that the other side of connection must know how much to read. This means that you must:

o Send fixed message sizes of a known number of bytes where all messages sent are of the same size or

o Send a message with a header of a known size where the message size is sent in the header. Thus read the fixed size header and determine the size of the whole message then read the remaining bytes.

Note that with a UDP client/server communications, the socket read will read the whole frame sent. This is not necessarily true for a TCP stream. The function used to read the stream will return the number of bytes read. If the required number of bytes have not been read, repeat the read for the remaining bytes until the whole message has been read.

In this example, our message header is a single integer used to store the message size in the first 32 bits of the message.

Check return codes: Just because the socket was opened doesn't mean it will stay open. Dropped connections do happen. Check all read/write socket API function return codes:

o return code > 0: Data received/sent o return code == 0: Socket closed o return code == -1: Check system errno and interpret (or call

"perror()")

For even more robust code for a socket client, close, then open a new socket connection for return codes 0 or -1.

Read entire message: (Variable and fixed length messages) When sending messages using a socket, be aware that the other side of connection must know how much to read. This means that you must:

o Send fixed message sizes of a known number of bytes where all messages sent are of the same size or

o Send a message with a header of a known size where the message size is sent in the header. Thus read the fixed size header and determine the size of the whole message then read the remaining bytes.

Note that with a UDP client/server communications, the socket read will read the whole frame sent. This is not necessarily true for a TCP stream. The function used to read the stream will return the number of bytes read. If the required number of bytes have not been read, repeat the read for the remaining bytes until the whole message has been read.

In this example, our message header is a single integer used to store the message size in the first 32 bits of the message.

This example applies to either client or server TCP sockets.

Another method of ensuring a complete read of a fixed size message is to use the MSG_WAITALL flag:

...

...

int flags = MSG_WAITALL;int rc = recv(socketConnection, buffer, length, flags);

...

...

On "SOCK_STREAM" (TCP) sockets, this flag requests that the recv() function block until the full amount of data specified (length) can be returned. The function may return the smaller amount of data if the socket is a message-based socket, if a signal is caught or if the connection is terminated.

UDP message order: TCP will guarentee that the order of the message delivery is maintained. UDP will not guarentee the message delivery order and thus you will have to maintain a counter in the message if the order is important.

Signals: The socket layer can issue signals, which if not handled, can terminate your application. When a socket is terminated at one end, the other may receive a SIGPIPE signal.

Program received signal SIGPIPE, Broken pipe.0x000000395d00de45 in send () from /lib64/libpthread.so.0(gdb) where#0 0x000000395d00de45 in send () from /lib64/libpthread.so.0#1 0x00000000004969c5 in bla bla bla......

Note GDB will report a received signal even if it's being ignored by the application. In this example, we tried to send() over a broken socket link.

Set up a signal handler at the beginning of your program to handle SIGPIPE:

#include <stdlib.h>#include <signal.h>

/// Ignore signal with this handlervoid handleSigpipe(int signum){ cout << "SIGPIPE ignored" << endl; return;}

int main(){ /// Ignore SIGPIPE "Broken pipe" signals when socket connections are broken. signal(SIGPIPE, handleSigpipe);

... ...}

Also see: o C++ Signal class tutorial - YoLinux.com tutorial o signal: Signal handling o sigaction: examine and change a signal action

Socket BSD API man pages:Sockets API:

socket: establish socket interface gethostname: obtain hostname of system gethostbyname: returns a structure of type hostent for the given host

name bind: bind a name to a socket listen: listen for connections on a socket accept: accept a connection on a socket connect: initiate a connection on a socket setsockopt: set a particular socket option for the specified socket. close: close a file descriptor shutdown: shut down part of a full-duplex connection

Interrogate a socket: select: synchronous I/O multiplexing FD_ZERO(), FD_CLR(), FD_SET(), FD_ISSET(): Set socket bit masks poll: check on the state of a socket in a set of sockets. The set can be

tested to see if any socket can be written to, read from or if an error occurred.

getsockopt: retrieve the current value of a particular socket option for the specified socket.

Read/Write: recv/recvfrom/recvmsg: Read socket send/sendto/sendmsg: Write socket

Convert: ntohl/htonl, ntohs/htons: convert values between host and network byte

order inet_pton: Create a network address structure inet_ntop: Parse network address structures

Other supporting system calls: exit: Terminate process perror: Output explanation of an error code protocols: Network Protocols (see /etc/protocols)

Static, Shared Dynamic and Loadable Linux Libraries

Why libraries are used:

This methodology, also known as "shared components" or "archive libraries", groups together multiple compiled object code files into a single file known as a library. Typically C functions/C++ classes and methods which can be shared by more than one application are broken out of the application's source code,

compiled and bundled into a library. The C standard libraries and C++ STL are examples of shared components which can be linked with your code. The benefit is that each and every object file need not be stated when linking because the developer can reference the individual library. This simplifies the multiple use and sharing of software components between applications. It also allows application vendors a way to simply release an API to interface with an application. Components which are large can be created for dynamic use, thus the library remain separate from the executable reducing it's size and thus disk space used. The library components are then called by various applications for use when needed.

Linux Library Types:

There are two Linux C/C++ library types which can be created:

1. Static libraries (.a): Library of object code which is linked with, and becomes part of the application.

2. Dynamically linked shared object libraries (.so): There is only one form of this library but it can be used in two ways.

1. Dynamically linked at run time but statically aware. The libraries must be available during compile/link phase. The shared objects are not included into the executable component but are tied to the execution.

2. Dynamically loaded/unloaded and linked during execution (i.e. browser plug-in) using the dynamic linking loader system functions.

Library naming conventions:

Libraries are typically names with the prefix "lib". This is true for all the C standard libraries. When linking, the command line reference to the library will not contain the library prefix or suffix.

Thus the following link command: gcc src-file.c -lm -lpthread The libraries referenced in this example for inclusion during linking are the math library and the thread library. They are found in /usr/lib/libm.a and /usr/lib/libpthread.a.

Static Libraries: (.a)

How to generate a library:

Compile: cc -Wall -c ctest1.c ctest2.c Compiler options:

o -Wall: include warnings. See man page for warnings specified. Create library "libctest.a": ar -cvq libctest.a ctest1.o ctest2.o List files in library: ar -t libctest.a Linking with the library:

o cc -o executable-name prog.c libctest.a o cc -o executable-name prog.c -L/path/to/library-directory -

lctest Example files:

o ctest1.c

o ctest1.c

void ctest1(int *i){ *i=5;}

o ctest2.c void ctest2(int *i){ *i=100;}

o prog.c

#include <stdio.h>void ctest1(int *);void ctest2(int *);

int main(){ int x; ctest1(&x); printf("Valx=%d\n",x);

return 0;}

Historical note: After creating the library it was once necessary to run the command: ranlib ctest.a. This created a symbol table within the archive. Ranlib is now embedded into the "ar" command.

Note for MS/Windows developers: The Linux/Unix ".a" library is conceptually the same as the Visual C++ static ".lib" libraries.

Dynamically Linked "Shared Object" Libraries: (.so)

How to generate a shared object: (Dynamically linked object library file.) Note that this is a two step process.

1. Create object code 2. Create library 3. Optional: create default version using a symbolic link.

Library creation example: gcc -Wall -fPIC -c *.c gcc -shared -Wl,-soname,libctest.so.1 -o libctest.so.1.0 *.o mv libctest.so.1.0 /opt/lib ln -sf /opt/lib/libctest.so.1.0 /opt/lib/libctest.so ln -sf /opt/lib/libctest.so.1.0 /opt/lib/libctest.so.1

This creates the library libctest.so.1.0 and symbolic links to it.

Compiler options:

-Wall: include warnings. See man page for warnings specified. -fPIC: Compiler directive to output position independent code, a

characteristic required by shared libraries. Also see "-fpic". -shared: Produce a shared object which can then be linked with

other objects to form an executable. -W1: Pass options to linker.

In this example the options to be passed on to the linker are: "-soname libctest.so.1". The name passed with the "-o" option is passed to gcc.

Option -o: Output of operation. In this case the name of the shared object to be output will be "libctest.so.1.0"

Library Links:

The link to /opt/lib/libctest.so allows the naming convention for the compile flag -lctest to work.

The link to /opt/lib/libctest.so.1 allows the run time binding to work. See dependency below.

Compile main program and link with shared object library:

Compiling for runtime linking with a dynamically linked libctest.so.1.0: gcc -Wall -I/path/to/include-files -L/path/to/libraries prog.c -lctest -o progUse: gcc -Wall -L/opt/lib prog.c -lctest -o prog

Where the name of the library is libctest.so. (This is why you must create the symbolic links or you will get the error "/usr/bin/ld: cannot find -lctest".)

The libraries will NOT be included in the executable but will be dynamically linked during runtime execution.

List Dependencies:

The shared library dependencies of the executable can be listed with the command: ldd name-of-executable

Example: ldd prog libctest.so.1 => /opt/lib/libctest.so.1 (0x00002aaaaaaac000) libc.so.6 => /lib64/tls/libc.so.6 (0x0000003aa4e00000) /lib64/ld-linux-x86-64.so.2 (0x0000003aa4c00000)

Run Program:

Set path: export LD_LIBRARY_PATH=/opt/lib:$LD_LIBRARY_PATH Run: prog

Man Pages:

gcc - GNU C compiler ld - The GNU Linker ldd - List dependencies

Links:

LDP: Shared libraries

Library Path:

In order for an executable to find the required libraries to link with during run time, one must configure the system so that the libraries can be found. Methods available: (Do at least one of the following)

1. Add library directories to be included during dynamic linking to the file /etc/ld.so.conf

Sample: /etc/ld.so.conf

/usr/X11R6/lib/usr/lib...../usr/lib/sane

/usr/lib/mysql/opt/lib

Add the library path to this file and then execute the command (as root) ldconfig to configure the linker run-time bindings. You can use the "-f file-name" flag to reference another configuration file if you are developing for different environments. See man page for command ldconfig.

OR

2. Add specified directory to library cache: (as root) ldconfig -n /opt/lib Where /opt/lib is the directory containing your library libctest.so (When developing and just adding your current directory: ldconfig -n . Link with -L.)

This will NOT permanently configure the system to include this directory. The information will be lost upon system reboot.

OR

3. Specify the environment variable LD_LIBRARY_PATH to point to the directory paths containing the shared object library. This will specify to the run time loader that the library paths will be used during execution to resolve dependencies. (Linux/Solaris: LD_LIBRARY_PATH, SGI: LD_LIBRARYN32_PATH, AIX: LIBPATH, Mac OS X: DYLD_LIBRARY_PATH, HP-UX: SHLIB_PATH)

Example (bash shell): export LD_LIBRARY_PATH=/opt/lib:$LD_LIBRARY_PATH or add to your ~/.bashrc file:

...if [ -d /opt/lib ];then LD_LIBRARY_PATH=/opt/lib:$LD_LIBRARY_PATHfi

...

export LD_LIBRARY_PATH

This instructs the run time loader to look in the path described by the

environment variable LD_LIBRARY_PATH, to resolve shared libraries. This will include the path /opt/lib.

Library paths used should conform to the "Linux Standard Base" directory structure.

Library Info:

The command "nm" lists symbols contained in the object file or shared library.

Use the command nm -D libctest.so.1.0 (or nm --dynamic libctest.so.1.0)

0000000000100988 A __bss_start000000000000068c T ctest100000000000006a0 T ctest2 w __cxa_finalize00000000001007b0 A _DYNAMIC0000000000100988 A _edata0000000000100990 A _end00000000000006f8 T _fini0000000000100958 A _GLOBAL_OFFSET_TABLE_ w __gmon_start__00000000000005b0 T _init w _Jv_RegisterClasses

Man page for nm

Symbol Type

Description

AThe symbol's value is absolute, and will not be changed by further linking.

B Un-initialized data section

D Initialized data section

T Normal code section

UUndefined symbol used but not defined. Dependency on another library.

WDoubly defined symbol. If found, allow definition in another library to resolve dependency.

Also see: objdump man page

Library Versions:

Library versions should be specified for shared objects if the function interfaces are expected to change (C++ public/protected class definitions), more or fewer functions are included in the library, the function prototype changes (return data type (int, const int, ...) or argument list changes) or data type changes (object definitions: class data members, inheritance, virtual functions, ...).

The library version can be specified when the shared object library is created. If the library is expected to be updated, then a library version should be specified. This is especially important for shared object libraries which are dynamically linked. This also avoids the Microsoft "DLL hell" problem of conflicting libraries where a system upgrade which changes a standard library breaks an older application expecting an older version of the the shared object function.

Versioning occurs with the GNU C/C++ libraries as well. This often make binaries compiled with one version of the GNU tools incompatible with binaries compiled with other versions unless those versions also reside on the system. Multiple versions of the same library can reside on the same system due to versioning. The version of the library is included in the symbol name so the linker knows which version to link with.

One can look at the symbol version used: nm csub1.o

00000000 T ctest1

No version is specified in object code by default. ld and object file layout

There is one GNU C/C++ compiler flag that explicitly deals with symbol versioning. Specify the version script to use at compile time with the flag: --version-script=your-version-script-file Note: This is only useful when creating shared libraries. It is assumed that the programmer knows which libraries to link with when static linking. Runtime linking allows opportunity for library incompatibility.

GNU/Linux, see examples of version scripts here: sysdeps/unix/sysv/linux/Versions

Some symbols may also get version strings from assembler code which appears in glibc headers files. Look at include/libc-symbols.h.

Example: nm /lib/libc.so.6 | more

00000000 A GCC_3.000000000 A GLIBC_2.000000000 A GLIBC_2.100000000 A GLIBC_2.1.100000000 A GLIBC_2.1.200000000 A GLIBC_2.1.3

00000000 A GLIBC_2.200000000 A GLIBC_2.2.100000000 A GLIBC_2.2.200000000 A GLIBC_2.2.300000000 A GLIBC_2.2.4.....

Note the use of a version script.

Library referencing a versioned library: nm /lib/libutil-2.2.5.so

..

... U strcpy@@GLIBC_2.0 U strncmp@@GLIBC_2.0 U strncpy@@GLIBC_2.0.....

Links:

Symbol versioning GNU.org: ld version scripts

Dynamic loading and un-loading of shared libraries using libdl:

These libraries are dynamically loaded / unloaded and linked during execution. Usefull for creating a "plug-in" architecture.

Prototype include file for the library: ctest.h

#ifndef CTEST_H#define CTEST_H

#ifdef __cplusplusextern "C" {#endif

void ctest1(int *);void ctest2(int *);

#ifdef __cplusplus}#endif

#endif

Use the notation extern "C" so the libraries can be used with C and C++. This statement prevents the C++ from name mangling and thus creating "unresolved symbols" when linking.

Load and unload the library libctest.so (created above), dynamically:

#include <stdio.h>#include <dlfcn.h>#include "ctest.h"

int main(int argc, char **argv) { void *lib_handle; double (*fn)(int *); int x; char *error;

lib_handle = dlopen("/opt/lib/libctest.so", RTLD_LAZY); if (!lib_handle) { fprintf(stderr, "%s\n", dlerror()); exit(1); }

fn = dlsym(lib_handle, "ctest1"); if ((error = dlerror()) != NULL) { fprintf(stderr, "%s\n", error); exit(1); }

(*fn)(&x); printf("Valx=%d\n",x);

dlclose(lib_handle); return 0;}

gcc -rdynamic -o progdl progdl.c -ldl

Explanation:

dlopen("/opt/lib/libctest.so", RTLD_LAZY); Open shared library named "libctest.so". The second argument indicates the binding. See include file dlfcn.h. Returns NULL if it fails. Options:

o RTLD_LAZY: If specified, Linux is not concerned about unresolved symbols until they are referenced.

o RTLD_NOW: All unresolved symbols resolved when dlopen() is called.

o RTLD_GLOBAL: Make symbol libraries visible. dlsym(lib_handle, "ctest1");

Returns address to the function which has been loaded with the shared library.. Returns NULL if it fails. Note: When using C++ functions, first use nm to find the "mangled" symbol name or use the extern "C" construct to avoid name mangling. i.e. extern "C" void function-name();

Object code location: Object code archive libraries can be located with either the executable or the loadable library. Object code routines used by both should not be duplicated in each. This is especially true for code which use static variables such as singleton classes. A static variable is global and thus can only be represented once. Including it twice will provide unexpected results. The programmer can specify that specific object code be linked with the executable by using linker commands which are passed on by the compiler.

Use the "-Wl" gcc/g++ compiler flag to pass command line arguments on to the GNU "ld" linker.

Example makefile statement: g++ -rdynamic -o appexe $(OBJ) $(LINKFLAGS) -Wl,--whole-archive -L{AA_libs} -laa -Wl,--no-whole-archive $(LIBS)

--whole-archive: This linker directive specifies that the libraries listed following this directive (in this case AA_libs) shall be included in the resulting output even though there may not be any calls requiring its presence. This option is used to specify libraries which the loadable libraries will require at run time.

-no-whole-archive: This needs to be specified whether you list additional object files or not. The gcc/g++ compiler will add its own list of archive libraries and you would not want all the object code in the archive library linked in if not needed. It toggles the behavior back to normal for the rest of the archive libraries.

Man pages:

dlopen() - gain access to an executable object file dclose() - close a dlopen object dlsym() - obtain the address of a symbol from a dlopen object dlvsym() - Programming interface to dynamic linking loader. dlerror() - get diagnostic information

Links:

Shared Libraries-Dynamic Loading and Unloading

GNOME Glib dynamic loading of modules - cross platform API for dynamically loading "plug-ins".

C++ class objects and dynamic loading:

C++ and name mangling:

When running the above "C" examples with the "C++" compiler one will quickly find that "C++" function names get mangled and thus will not work unless the function definitions are protected with extern "C"{}.

Note that the following are not equivalent: extern "C"{ int functionx();}

extern "C" int functionx();

The following are equivalent: extern "C"{ extern int functionx();}

extern "C" int functionx();

Dynamic loading of C++ classes:

The dynamic library loading routines enable the programmer to load "C" functions. In C++ we would like to load class member functions. In fact the entire class may be in the library and we may want to load and have access to the entire object and all of its member functions. Do this by passing a "C" class factory function which instantiates the class.

The class ".h" file: class Abc {

...

...

};

// Class factory "C" functions

typedef Abc* create_t;typedef void destroy_t(Abc*);

The class ".cpp" file: Abc::Abc(){

...}

extern "C"{ // These two "C" functions manage the creation and destruction of the class Abc

Abc* create() { return new Abc; }

void destroy(Abc* p) { delete p; // Can use a base class or derived class pointer here }}

This file is the source to the library. The "C" functions to instantiate (create) and destroy a class defined in the dynamically loaded library where "Abc" is the C++ class.

Main executable which calls the loadable libraries: // load the symbols create_t* create_abc = (create_t*) dlsym(lib_handle, "create");

...

...

destroy_t* destroy_abc = (destroy_t*) dlsym(lib_handle, "destroy");

...

...

Pitfalls: The new/delete of the C++ class should both be provided by the

executable or the library but not split. This is so that there is no surprise if one overloads new/delete in one or the other.

Links: LinuxJournal.com: Dynamic Class Loading for C++ on Linux dlopen howto

Comparison to the Microsoft DLL:

The Microsoft Windows equivalent to the Linux / Unix shared object (".so") is the ".dll". The Microsoft Windows DLL file usually has the extension ".dll", but may also use the extension ".ocx". On the old 16 bit windows, the dynamically linked libraries were also named with the ".exe" suffix. "Executing" the DLL will load it into memory.

The Visual C++ .NET IDE wizard will create a DLL framework through the GUI, and generates a ".def" file. This "module definition file" lists the functions to be exported. When exporting C++ functions, the C++ mangled names are used. Using the Visual C++ compiler to generate a ".map" file will allow you to discover the C++ mangled name to use in the ".def" file. The "SECTIONS" label in the ".def" file will define the portions which are "shared". Unfortunately the generation of DLLs are tightly coupled to the Microsoft IDE, so much so that I would not recomend trying to create one without it.

The Microsoft Windows C++ equivalent functions to libdl are the following functions:

::LoadLibrary() - dlopen() ::GetProcAddress() - dlsym() ::FreeLibrary() - dlclose()

[Potential Pitfall]: Microsoft Visual C++ .NET compilers do not allow the linking controll that the GNU linker "ld" allows (i.e. --whole-archive, -no-whole-archive). All symbols need to be resolved by the VC++ compiler for both the loadable library and the application executable individually and thus it can cause duplication of libraries when the library is loaded. This is especially bad when using static variables (i.e. used in singleton patterns) as you will get two memory locations for the static variable, one used by the loadable library and the other used by the program executable. This breaks the whole static variable concept and the singleton pattern. Thus you can not use a static variable which is referenced by by both the loadable library and the application executable as they will be unique and different. To use a unique static variable, you must pass a pointer to that static variable to the other module so that each module (main executable and DLL library) can use the same instatiation. On MS/Windows you can use shared memory or a memory mapped file so that the main executable and DLL library can share a pointer to an address they both will use.

Cross platform (Linux and MS/Windows) C++ code snippet: Include file declaration: (.h or .hpp)

class Abc{public: static Abc* Instance(); // Function declaration. Could also be used as a public class member function.

private: static Abc *mInstance; // Singleton. Use this declaration in C++ class member variable declaration. ...}

C/C++ Function source: (.cpp) /// Singleton instantiationAbc* Abc::mInstance = 0; // Use this declaration for C++ class member

variable // (Defined outside of class definition in ".cpp" file)

// Return unique pointer to instance of Abc or create it if it does not exist.// (Unique to both exe and dll)

static Abc* Abc::Instance() // Singleton{#ifdef WIN32 // If pointer to instance of Abc exists (true) then return instance pointer else look for // instance pointer in memory mapped pointer. If the instance pointer does not exist in // memory mapped pointer, return a newly created pointer to an instance of Abc.

return mInstance ? mInstance : (mInstance = (Abc*) MemoryMappedPointers::getPointer("Abc")) ? mInstance : (mInstance = (Abc*) MemoryMappedPointers::createEntry("Abc",(void*)new Abc));#else // If pointer to instance of Abc exists (true) then return instance pointer // else return a newly created pointer to an instance of Abc.

return mInstance ? mInstance : (mInstance = new Abc);#endif}

Windows linker will pull two instances of object, one in exe and one in loadable module. Specify one for both to use by using memory mapped pointer so both exe and loadable library point to same variable or object. Note that the GNU linker does not have this problem. For more on singletons see the YoLinux.com C++ singleton software design pattern tutorial.

Cross platform programming of loadable libraries:#ifndef USE_PRECOMPILED_HEADERS#ifdef WIN32#include <direct.h>#include <windows.h>#else#include <sys/types.h>#include <dlfcn.h>#endif#include <iostream>#endif

using namespace std;

#ifdef WIN32 HINSTANCE lib_handle;#else void *lib_handle;#endif

// Where retType is the pointer to a return type of the function // This return type can be int, float, double, etc or a struct or class.

typedef retType* func_t;

// load the library -------------------------------------------------#ifdef WIN32 string nameOfLibToLoad("C:\opt\lib\libctest.dll"); lib_handle = LoadLibrary(TEXT(nameOfLibToLoad.c_str())); if (!lib_handle) { cerr << "Cannot load library: " << TEXT(nameOfDllToLoad.c_str()) << endl; }#else string nameOfLibToLoad("/opt/lib/libctest.so"); lib_handle = dlopen(nameOfLibToLoad.c_str(), RTLD_LAZY); if (!lib_handle) { cerr << "Cannot load library: " << dlerror() << endl; }#endif

...

...

...

// load the symbols -------------------------------------------------#ifdef WIN32 func_t* fn_handle = (func_t*) GetProcAddress(lib_handle, "superfunctionx"); if (!fn_handle) { cerr << "Cannot load symbol superfunctionx: " << GetLastError() << endl; }#else // reset errors dlerror();

// load the symbols (handle to function "superfunctionx") func_t* fn_handle= (func_t*) dlsym(lib_handle, "superfunctionx"); const char* dlsym_error = dlerror(); if (dlsym_error) { cerr << "Cannot load symbol superfunctionx: " << dlsym_error << endl; }#endif

...

...

...

// unload the library -----------------------------------------------

#ifdef WIN32 FreeLibrary(lib_handle);#else dlclose(lib_handle);#endif

Tools:

Man pages:

ar - create, modify, and extract from archives ranlib - generate index to archive nm - list symbols from object files ld - Linker ldconfig - configure dynamic linker run-time bindings

ldconfig -p : Print the lists of directories and candidate libraries stored in the current cache. i.e. /sbin/ldconfig -p |grep libGL

ldd - print shared library dependencies gcc/g++ - GNU project C and C++ compiler man page to: ld.so - a.out dynamic linker/loader