Unix Lisp Based Shared Libraries

Unix Allegro Common Lisp developers can now create Lisp based shared libraries that can be used by other developers writing C, C++, or Java applications.

This functionality is available on all Unix platforms.

Sections

How it Works
An Example: A Shared Library that computes factorials
    fact.c details
    fact.cl details
    ftest.c details
Compilation and Delivery
Additional Tips
OS Specific Notes
    Sun Solaris
    Dec Alpha Unix
    SGI IRIX
    Hewlett-Packard HP-UX
    IBM AIX
    Linux

How it Works

The Lisp developer creates a "wrapper" shared library (C is used in the example below) to provide the public functional interface developers will use. The shared library also contains private functions used to transfer control back and forth between C wrappers and Lisp code. A companion Lisp image contains the custom code needed to provide the functionality behind the public interface as well as utility functions needed by the C/Lisp interfacing.

Unix OS thread functionality is used to start the Lisp image on a separate OS thread. Because Unix ACL 5.0 versions are not integrated with OS thread functionality, all Lisp processing must occur in the Lisp thread. (With Windows ACL 5.0, once the Lisp starts, Lisp processing may occur on any thread, including the original main thread, or other threads created outside of Lisp.) Two semaphores are used to synchronize processing between the Lisp thread and the original main thread.

An Example: A Shared Library that computes factorials

Since Lisp easily handles large valued integer calculations, a Lisp based factorial() function is a good, simple example. To avoid C integer overflow, the answer is placed in a result string, rather than returned as a number.

The fact.c file contains the shared library C wrapper source. One thing you'll immediately notice are the many OS specific #if C macros. Unfortunately, thread functionality is not yet constant across all Unix OS's; this accounts for the majority of the macros. Architecture differences account for the other macros.

The fact.cl file contains the code that computes the factorial.

The ftest.c file is an example C program that uses the factorial calculating shared library.

fact.c details

The load_lisp() private function runs in a new thread created by the original main thread. It loads the Lisp image and starts it running. It calls the ACL library function lisp_init(), which is described in the ACL Release Notes User-defined main() documentation. Note that the shlibs.h file is found in the ACL distribution misc directory. Use the compiler -I option to add that directory to the include file search list when compiling fact.c. The ACL library function tcm() is called to prevent the lisp from writing to stdout and stderr during initialization and subsequent processing. During your development phase, you may wish to comment this call out, in order to see 'format or 'print debugging output.

The wait_on_semaphore() and release_semaphore() private functions are used to synchronize processing between the Lisp thread and the original main thread. This example uses pipes as semaphores. Since file descriptors are a limited resource, and some Unix OS's provide specialized semaphore functionality, you may wish to change this code for your own work. See the OS specific sections, below, for more information.

The initialize_factorial() public function creates the Lisp thread. Its argument is a pointer to the main program environment that is available as the third main() argument. Besides initializing the synchronization semaphores, it uses the ACL library function find_file_using_pathstring() to identify the directory containing the fact.dxl Lisp image. It's a good design decision to use the same environmental variable that the OS uses to find shared libraries for the third argument. For example, on Solaris, use LD_LIBRARY_PATH. The wait_on_semaphore() function is called to wait for the Lisp thread to respond that all initialization activities are completed. Note that on AIX, the thread creation code is found in ftest.c, and some extra fact.c public functions are needed. See the AIX OS specific notes for more information.

The release_end_semaphore()  function is provided for the Lisp image to call when initialization steps are complete, releasing the semaphore initialize_factorial() is waiting on, allowing it to return to its caller.

The set_factorial_callback() function is provided for the Lisp image to call during initialization steps to set the callback address for the Lisp 'factorial callback

The factorial() public function wraps the C/Lisp operations that calculate the factorial result. It sets some global variables that the Lisp thread will access, releases the semaphore that the wait_for_work() function is waiting on in the Lisp thread, and then waits on another semaphore for the Lisp thread to respond that the factorial calculation is complete. It then returns, using a global value set by the Lisp thread.

The wait_for_work() function is called by the Lisp image after initialization steps are complete and it has released the semaphore used to synchronize the initialization phase. It loops forever, responding when the original main thread releases a semaphore that indicates that factorial() has been called. It invokes the 'factorial callback and then releases a semaphore that tells the main thread that Lisp processing is complete. It then goes back to waiting for more work.

The copy_factorial_result() function is called by Lisp during   factorial processing to pass the result string into C data.

The terminate_factorial() public function wakes up wait_for_work() in such a way that it breaks out of its endless loop, causing the Lisp thread to unwind. On Irix and Linux, where separate threads have separate process id's. it is imperative that the Lisp thread unwind; otherwise a hang will occur.

fact.cl details

The 'factorial function calculates the factorial result.

The wrapper C shared library is loaded and foreign function definitions are made as necessary.

The 'factorial-callback callback calls 'factorial, converts the result to a string, and passes the result string to C data using the copy_factorial_result() function.

The 'initialize-factorial function passes the callback address back to C, releases the initialization semaphore, and goes into a "wait for work" mode, all using C functions defined in fact.c. When the wait_for_work() call returns, indicating that terminate_factorial() has been called, 'exit is called, to unwind the Lisp thread.

The excl::*restart-init-function* variable is set to 'initialize-factorial, so that the initialization steps commence when the Lisp image is started.

ftest.c details

The ftest.c file is fairly straightforward. With the exception of AIX, it calls the C wrapper shared library public functions initialize_factorial() and factorial(). It calls factorial() twice to demonstrate that the Lisp based shared library behaves the same as any other shared library would. On the way out,   terminate_factorial() is called. Looking at this code, there is no way to tell that the factorial shared library is Lisp based.

See the AIX OS specific notes to understand why AIX requires a different setup. As described in the notes, the extra AIX code could be moved to an include file, making the AIX version also look like usual shared library programming.

Compilation and Delivery

See the OS Specific Notes section, below, for OS specific C shared library wrapper generation commands.

Look at the ACL 5.0 Release Notes section about application delivery. 'excl::generate-application is a useful tool for generating the fact.dxl image for delivery and gathering together all needed files for delivery preparation.

Remember when writing your shared library documentation to tell users to add the delivery directory to the OS dynamic library load environmental variable.

Additional Tips

This example only uses one Lisp callback. If you wish to provide many callbacks, you must pass all the callback addresses from Lisp to C, and you must be able to determine which callback is appropriate when wait_for_work() wakes up. If you have many callbacks, it may make sense to load the addresses into a C structure defined in the lisp and pass the structure's address to C as a foreign function call argument. Similarly, rather than set up many global argument variables, it may make sense to define a union in C for wait_for_work() to work with. There can be a union member element available for saving the callback address. You can have elements that can be used to store callback arguments. Put the union in a structure that also contains an 'id' element. Then provide public C wrappers for each callback. In each wrapper, set the 'id' structure element to a unique number and then load the callback arguments into the appropriate union member. Then, when wait_for_work() wakes up, it can use the 'id' element to determine which callback to invoke and retrieve the callback arguments out of the appropriate union member.

If you don't call tcm() in your C shared library wrapper, then you will be able to see 'print and 'format that you may put in your Lisp code for debugging purposes. As long as the original main thread is not seeking to read from standard input, you can respond to breaks as you would normally do when starting lisp in a shell.

OS Specific Notes

The following OS specific notes sections assume you are building your shared library in the ACL product installation examples/unix-shared-library directory..

On most Unix OS's, the dynamic load environmental variable must include the directories containing the acl503 and fact shared libraries to successfully run the ftest example executable.

Sun Solaris

Tested on SunOS 5.5.1.
Dynamic load environmental variable: LD_LIBRARY_PATH
Shared library file extension: .so

Example shared library generation commands:

cc -c -I.../../misc fact.c
ld -G -olibfact.so fact.o -lpthread -L../.. -lacl503

Example Lisp image creation commands:

:cl fact
(dumplisp :name "fact.dxl")

Example executable creation command:

cc -o ftest ftest.c -L. -lfact

Notes

  1. POSIX semaphores are available as an alternative to select() calls. See the sem_init() man page.
  2. Don't load shared libraries linked with SunOS thread libraries multiple times into the same ACL session - it will cause a
    crash. See the example code in fact.cl for a way to protect against this.

Dec Alpha Unix

Tested on OSF1 V4.0.
Dynamic load environmental variable: LD_LIBRARY_PATH
Shared library file extension: .so

Example shared library generation command:

cc -shared -o libfact.so -taso -xtaso -xtaso_short  -I../../misc fact.c -lpthread -L../.. -lacl503

Example Lisp image creation commands:

:cl fact
(dumplisp :name "fact.dxl")

Example executable creation command:

cc -taso -o ftest ftest.c -L. -lfact

Notes

  1. POSIX semaphores are available as an alternative to select() calls. See the sem_init() man page.
  2. Don't load shared libraries linked with Dec Unix thread libraries multiple times into the same ACL session - it will cause a
    crash. See the example code in fact.cl for a way to protect against this.

SGI IRIX

Tested on IRIX64 6.2.
Dynamic load environmental variable: LD_LIBRARY_PATH
Shared library file extension: .so

Example shared library generation command:

cc -shared -n32 -o libfact.so -I../../misc fact.c -L../.. -lacl503

Example Lisp image creation commands:

:cl fact
(dumplisp :name "fact.dxl")

Example executable creation command:

cc -n32 -o ftest ftest.c -L. -lfact

Notes

  1. The example uses the IRIX sproc() shared process functionality.
  2. Ignore warnings about replacing signal() function.
  3. Since IRIX shared processes behave in many ways like separate processes, you should provide a Lisp thread termination function like terminate_factorial() to prevent hung programs or wasteful unwound processes.

Hewlett-Packard HP-UX

Tested on HP-UX B.10.20.
Dynamic load environmental variable: SHLIB_PATH
Shared library file extension: .sl

Example shared library generation commands:

cc -c +z -Ae -I../../misc fact.c
ld -b +s -o libfact.sl fact.o -ldce -L../.. -lacl503

Example Lisp image creation commands:

:cl fact
(dumplisp :name "fact.dxl")

Example executable creation command:

CC -o ftest ftest.c -L. -lfact

Notes

  1. Requires that optional DCE package be available on computer.
  2. You MUST use the CC compiler when generating an executable linked with a library that was
    linked with the DCE library.
  3. Make sure that  SHLIB_PATH contains /usr/lib.
  4. Ignore compiler warnings regarding a DCE include file.
  5. On our HP machine, running a DCE based application in csh can result in a logout when the application
    exits. You may have to run your application in sh or ksh to avoid this problem.

IBM AIX

Tested on AIX 4.2.
Dynamic load environmental variable: LIBPATH
Shared library file extension: .so

Example shared library generation commands:

cc -c -DAIX -I../../misc fact.c
ld  -brtl -bnodelcsect -D0 -H4096 -T512 -bM:SRE -bE:fact.exp -o libfact.so fact.o -lpthreads -L../.. -lacl503 -lc

Example Lisp image creation commands:

:cl fact
(dumplisp :name "fact.dxl")

Example executable creation command:

cc_r -brtl -bnodelcsect -H4096 -T512 -DAIX -o ftest ftest.c -L. -lfact

Notes

  1. Before creating the shared library, create a fact.exp file, containing:

    #!
    initialize_factorial
    release_end_semaphore
    wait_on_end_semaphore
    set_factorial_callback
    factorial
    wait_for_work
    copy_factorial_result
    lisp_init_func
    terminate_factorial
  2. You MUST use the cc_r compiler to compile and generate an executable calling thread functions. Notice how
    the AIX version of ftest.c contains thread creation code that would usually be found in the shared library
    source code. If you wish to shield your users from such awkwardness, you could put code like this in an
    include file, and use C macros to ensure that it is compiled in only one file. Also, notice the extra code
    in the AIX version of fact.c that is needed to support removing the thread creation code from the shared
    library source.
  3. Ignore ld warnings about __start.

Linux

Tested on Red Hat 5.0.
Dynamic load environmental variable: LD_LIBRARY_PATH
Shared library file extension: .so

Example shared library generation command:

cc -shared  -o libfact.so -I../../misc fact.c -lpthread -L../.. -lacl503

Example Lisp image creation commands:

:cl fact
(dumplisp :name "fact.dxl")

Example executable creation command:

cc  -o ftest ftest.c -L. -lfact

Notes

  1. Since Linux threads behave in many ways like separate processes, you should provide a Lisp thread termination function like terminate_factorial() to prevent hung programs or wasteful unwound processes.