.if \n(.g .do char \[a:] \[:a] .\" \(a: is a lower case a with diaeresis (a umlaut) .\" \(-> is an arrow pointing to the right .\" \(mu is a multiplication sign (cross) .\" \*- is an em dash . .\" .fp 1 PR .\" .fp 2 PI .\" .fp 3 PB .nr VS 20 .fp 5 C .fp 6 CO .ie \n(.U .ds ^4 ^4 .el .ds ^4 \u\s-1\|4\s0\d . \" Second level section .de P .NH 2 .. . \" Scheme code start .de Ss .KS .ta 8.5c .nr sF \\n(.f .ft 5 .ps -2 .\".vs -2 .vs 13 .nf .in 1c .if !\n(.U .sp .3c .. . \" Scheme code end .de Se .in .fi .vs .ps .ft \\n(sF .KE .if !\n(.U .sp .5 .. . \" Newline in Scheme code .de Sl .sp .52 .. .nr lS 0 1 . \" Listing start .de Ls .br .KF .sp .5 .LP \l'\\n(.lu_' .. . \" Listing caption: .Lc caption .de Lc .sp .2 .ce 999 \f3\s-1Listing \\n+(lS:\fP \c \\$1\s0 .if !\\$2 \s-1\&\\$2\s0 .ce 0 .. . \" Listing end: .Le reference .de Le .tm s/@L(\\$1)/\\n(lS/g .LP \l'\\n(.lu_' .sp .KE .. . \" Notes start (at end of listing) .de Ns .sp .ps -2 .vs -2 .in 1c .ll -1c .. . \" Notes end .de Ne .ll .in .vs .ps .sp -.3 .. .ds E Elk .TL \*E: The Extension Language Kit .AU Oliver Laumann* and Carsten Bormann\v'-.2m'\(dg\v'.2m' .AI * Technische Universit\(a:t Berlin, Germany .br \(dg Universit\(a:t Bremen, Germany .AB In the past, users of an application generally were at the mercy of its authors when it came to adapting it to their individual needs and tastes. Fitting an application with an \f2extension language\fP (or \f2embedded language\fP) enables users to customize and enhance it without having to modify its source code. Recently, variants of Lisp have become increasingly popular for this purpose, to the point where the abundance of different dialects has grown into a problem. Of the two standardized dialects of Lisp, only \f2Scheme\fP is suitably modest, yet sufficiently general, to serve as an extension language. .PP \f2\*E\fP, the \f2Extension Language Kit\fP, is a Scheme implementation that is intended to be used as a general, reusable extension language subsystem for integration into existing and future applications. Applications can define their own Scheme data types and primitives, providing for a tightly-knit integration of the C/C++ parts of the application with Scheme code. Library interfaces, for example to the UNIX operating system and to various X Window System libraries, show the effectiveness of this approach. Several features of \*E such as dynamic loading of object files and freezing of fully customized applications into executables (implemented for those UNIX environments where it was feasible) increase its usability as the backbone of a complex application. \*E has been used in this way for seven years within a locally-developed ODA-based multimedia document editor; it has been used in numerous other projects after it could be made freely available five years ago. .AE .NH Introduction .PP The designers and implementors of a large or complex application can rarely anticipate all requirements future users will have on the application. Typically, users wish to be able to customize the user interfaces of applications according to their personal tastes or requirements, or they want to extend the functionality of an application (either by combining existing functions into new ones or by adding entirely new capabilities). This is especially true for applications used routinely, such as text editors, and for applications with a high degree of user interaction or with complex graphical user interfaces. .PP Certainly any application can be customized by modifying its source code and recompiling it. But this approach is often not feasible, as the source code of the application or the tools needed to recompile it may not be available. Even if it were feasible, it would be a time-consuming process; it would be hard to keep up with new releases of the application; and the coexistence of multiple, similar versions of the same application would become a general maintenance headache. .PP The alternative to this approach is not to ``hard-wire'' the entire functionality and all external aspects of an application in the source code at all, but to provide means to customize the application's behavior later by its users. .P Early Customization and Extension Languages .PP Many applications support at least simple methods for customization, such as command line options or configuration files. More powerful tools for customization are \f2macro languages\fP, \f2command languages\fP, or \f2scripting languages\fP that are typically found in text editors and word processors. Prominent examples of such customization and extension languages are the macro language of the now legendary TECO editor and, in UNIX, the macro language of the \f2troff\fP text formatter [Ossanna 1979] and the configuration language of the \f2sendmail\fP program. .PP Although many of these classic extension languages are quite powerful (some of them are full-fledged programming languages), they have a reputation of being ``cryptic'' and hard to understand and use by untrained users. The prevailing opinion seems to be that only experts can actually benefit from these types of extension languages (for example, people who have mastered the \f2sendmail\fP configuration language in all details are commonly appointed the status of a ``guru''). In fact, it can be observed that only very few users of the \f2troff\fP text formatter (whose macro language is reputed to be particularly cryptic) are using macro packages written by themselves; many users give up after some time and fall back on vendor-supplied macro packages or packages written by a ``troff guru.'' .PP Experience also indicates that simplified or specialized extension languages often have more features added and grow until they resemble a full programming language. Such ``organically grown'' extension languages are likely to be contorted designs as they will consist of several levels of extensions glued on to their initial, more limited design. .P High-Level Extension Languages .PP Recently application designers have begun to abandon specialized and cryptic macro-style extension languages in favor of extension languages that resemble usual high-level programming languages, mainly languages with Algol/Pascal-style or Lisp-style syntax and semantics. Prominent examples of such high-level extension languages are TPU developed by DEC, the \f2Ness\fP language of the Andrew Toolkit [Hansen 1990], AutoDesk's CAD extension language (a dialect of Lisp), and \f2Emacs-Lisp\fP, the extension language of Richard Stallman's popular GNU Emacs editor [Stallman 1981, Lewis et al. 1990]. .PP Emacs was the first wide-spread application to employ an already existing and widely used high-level programming language as its extension and customization language. Emacs-Lisp is a dynamically scoped dialect of Lisp with additional operations for text-editing. The approach taken by Emacs has been tremendously successful; users of Emacs have contributed a wealth of extensions written in Emacs-Lisp. .PP Note that Emacs-Lisp is not a \f2scripting language\fP. It is tightly interwoven with the application for which it provides extensibility. It also is somewhat inaccessible to the casual user, who is unlikely to have previous experience with Lisp-like languages. This can be contrasted with languages such as Tcl [Ousterhout 1990] and REXX [Cowlishaw 1985], whose underlying models are no less complex, but which are similar enough to well-known languages such as BASIC to present less of an obstacle to casual users. On the other hand, non-trivial extensions benefit from the structuring functionality inherent in general purpose programming languages such as Lisp. .P \*E as a General, Reusable Extension Language .PP Using Lisp or Lisp-style languages as extension languages seems to enjoy growing popularity; several applications besides Emacs now use dialects of Lisp as their extension language. This development has one disadvantage: the number of incompatible (but similar) extension languages is continually growing. Users have to learn a new language for each new application, and application writers keep implementing new extension language interpreters instead of reusing existing ones. .PP These problems can be solved by a general, reusable extension language implementation that application writers can include into their applications, an \f2extension language kit\fP. The main objective of the \f2\*E\fP project was to develop such an extension language kit and to make it freely available to encourage use by application writers. .NH Overview of the Extension Language Kit .P The Evolution of \*E .PP We were prompted to develop \*E when a search for a suitable extension language implementation for ISOTEXT [Bormann et al. 1988, Bormann 1991] was fruitless. ISOTEXT, a document processing system with a graphical user interface, is almost entirely written in C++; its user interface is based on the X window system [Scheifler et al. 1986, Scheifler et al. 1992] and the OSF/Motif widget set. Customizability and extensibility through a full extension language were basic requirements on the design of ISOTEXT. .PP As we consider language design to be the domain of a ``selected few'' and did not want to act as amateurs in this field, we decided to use an existing programming language as the basis for the extension language of ISOTEXT. This decision was also influenced by our desire to develop a general, reusable extension language implementation that is not hard-wired into one specific application. For a number of reasons an interpreted language seemed preferable: extensions can be added to (or modified in) a running application without re-linking it; bugs in extensions can be caught in the interpreter and do not crash the application; interpreted languages usually offer better debugging facilities; and implementing an interpreter generally is easier than implementing a compiler. .PP From the beginning we favored Lisp or a dialect of Lisp as the basis for a general extension language. Most dialects of the Lisp family are ``small'', easy to implement, general-purpose languages with simple syntax and powerful semantics, and the suitability of Lisp as an extension language had already been demonstrated by several applications, among them GNU Emacs. Early in the project we considered to use Emacs-Lisp, but it appeared infeasible to isolate the Lisp interpreter from the rest of Emacs. In addition, at the time we investigated Emacs-Lisp it was lacking several desirable language features, such as support for floating point and arbitrary precision numbers (\f2bignums\fP). We also considered using MIT Scheme [MIT 1984], but due to the enormous size of its implementation it would have dominated the size of the application. .P Scheme as an Extension Language .PP As other implementations of Lisp or Lisp-like languages available did not meet our requirements, we finally decided to write an interpreter for the Lisp dialect \f2Scheme\fP [Clinger et al. 1991, Dybvig 1987, Springer et al. 1989, Abelson et al. 1985]. This Scheme interpreter is the main component of the \*E package. Scheme is a simplified, ``cleaned-up'' dialect of Lisp with first-class procedures and static scoping rules. The Scheme language is based on only a few language features and semantic concepts; it consists of a small core of syntactic forms, a set of extended forms derived from them, and a number of standard procedures (\f2primitive\fP procedures) that operate on a comprehensive set of types of objects (among them numbers, lists, vectors, symbols, characters, and strings). In 1990 Scheme became an IEEE standard [IEEE\|Std\|1178-1990] (the standard document, although only 50 pages long, includes the formal semantics of the language). .PP The standardization effort has increased the acceptance of Scheme; for instance, the Extension Language Working Group of the CAD Framework Initiative has recently selected Scheme as the extension language for future CAD applications [CFI 1991a, CFI 1991b]. Among the established programming languages we consider Scheme the ideal candidate for a general extension language \*- it is standardized; its semantics are well-defined; it has a simple syntax and is easy to implement; and it is sufficiently small to not dwarf the application it makes extensible. .P Extending the Extension Language .PP The implementation of an extension language must itself be extensible. Extension language code that manipulates objects or state of the application requires adding application-specific primitive procedures to the base extension language. To allow \*E programs to be expressive in the context of a given application, application writers are encouraged (and expected) to extend standard Scheme by a rich set of application-specific data types and Scheme primitives to operate on objects of these types. In fact, easy extensibility of the language has been the primary design consideration in the development of \*E (as opposed to performance or number of language features). Adding new types and primitives to \*E is an inexpensive operation; it is not uncommon for an application to define hundreds of application-specific Scheme primitives. .\" .\" implementation must fit well to `host language' (schreibt cabo...) .PP All primitive procedures of \*E are implemented as C or C++ functions. This is true for both built-in primitives (such as \f2car\fP and \f2cdr\fP) and primitives defined by extensions. From the Scheme programmers' point of view, primitives and types from the base set of the language are indistinguishable from application-specific primitives and types. Extensions ``register'' new primitives with the interpreter by supplying the name of the primitive along with a pointer to the function implementing the primitive and information about the arguments and calling style. New types are defined in a similar way. Registration of new primitives and types usually takes place on startup of the interpreter or when a compiled extension is loaded into the running interpreter. .PP Another way to use the extension mechanisms of \*E is to provide interfaces to libraries, such as the C library or the libraries of the X window system (e.\|g.\& \f2Xlib\fP). \*E has no facility to directly import ``foreign'' functions (although one such facility has been contributed as an extension to \*E). Therefore, a small amount of code acting as ``glue'' between \*E and the library has to be written to make the contents of a library available to Scheme programmers. The main purpose of this interface code is to check the arguments supplied to the library functions, to convert Scheme objects into C types, and to convert the results of library functions back into Scheme objects. Such \f2library extensions\fP often act as an additional layer between the application to be extended and the libraries used by the application; they allow the application writers to abstract from the details of the libraries. Although it is useful to distinguish between \f2library\fP extensions and extensions interfacing to \f2applications\fP, there is no technical difference \*- in both cases a collection of types and functions is made available to the Scheme world. .PP Since many of today's applications need to interact with the X Window System, library extensions are included with \*E that interface to the X11 ``Xlib'' (similar in its functionality to ``CLX'' [CLX 1991], but implemented on top of Xlib), to the X11 toolkit intrinsics (``Xt''), and to the Athena and OSF/Motif widget sets. .PP In addition, the \*E UNIX extension provides Scheme access to most UNIX system calls and operating system interface C library functions\**. .FS The UNIX extension defines procedures for low-level, file-descriptor-based I/O; creation of pipes; file/record locking; file and directory system calls; process creation and control; signal handling; error handling; and obtaining information about date, time, users, limits, process resources, etc. .FE The extension supports a wide range of different UNIX platforms without restricting its functionality to the lowest common denominator or the POSIX 1003.1 functions. To facilitate writing portable Scheme programs, the extension attempts to hide differences between the types of supported UNIX flavors. .\"(Two examples are appended: one forks off a process .\"and communicates with it through pipes; the other one measure the maximum .\"capacity of a pipe using non-blocking I/O.) .NH Using \*E in Applications .\" .P .\" Bringing Everything Together .PP In contrast to other extension language implementations (e.\|g.\& Tcl), \*E does not provide its functionality in the form of a library that is statically linked into an application to be extended. Instead, the object modules comprising the application and all required library extensions are dynamically linked with and loaded into the running Scheme interpreter. To accomplish this, the \f2load\fP primitive of \*E has been extended to load not only files containing Scheme code, but also object files \*- compiled extensions written in C or C++. Dynamic loading enables applications to load less frequently used modules into the running program only on demand; such an application is initially smaller than the equivalent statically linked application (where all modules must be combined into one large executable file). .PP Dynamic loading of object files is often used together with the \f2dump\fP primitive that creates an executable file from the running interpreter, similar to \f2unexec\fP of GNU Emacs or \f2dump\%lisp\fP in some Lisp systems. The \f2dump\fP primitive of \*E differs from existing, similar mechanisms in that the newly created executable, when called, starts at the point where \f2dump\fP was called in the original invocation (as opposed to the program's \f2main\fP entry point). Here the return value of \f2dump\fP is ``true'', while in the original invocation it returns ``false'' \*- not unlike the UNIX \f2fork\fP system call. .P Dynamic Loading and Dump in Cooperation .PP To generate a new instance of an application one would typically invoke the Scheme interpreter, load all object modules and all Scheme code required initially, perform all initializations that can survive a \f2dump\fP, and finally dump an image of the running interpreter containing all the loaded code into a new executable on disk. The use of \f2dump\fP avoids time-consuming activities such as loading of object files and other initializations on each startup. The dumped executable, when started, resumes after the call to \f2dump\fP; at this point one would perform the remaining, environment-dependent initializations and finally invoke the application's ``main program'' (e.\|g.\& enter the X toolkit's event processing main loop). Listing @L(dump) shows a (slightly simplified) Scheme program that generates and starts a new instance of an application. .Ls .Ss ;;; Load initially required object files and Scheme files of ;;; application and dump image into executable file. ;;; Dumped file enters application's main loop on startup. .Sl (load 'main.o) ; initial object modules (load 'edit.o) (load 'x11.o) ; (a library extension) \&... (load 'ui.scm) ; initial Scheme files (load 'custom.scm) (load 'x11.scm) \&... (initialize-application) .Sl (if (dump 'a.out) (begin ; dumped a.out starts execution here (initialize-depending-on-environment) (main-loop-of-application) (exit))) .Sl ;; Original invocation gets here when dump is finished. We're done. .Se .Lc "Scheme code to generate and start an application" .Ns \f2Note:\fP Filenames can be given as symbols (besides the usual string literals). A more meaningful name than a.out would probably be chosen in practice. .Ne .Le dump .PP On systems that do not support dynamic linking and loading of object files (such as older versions of UNIX System V) or where \f2dump\fP cannot be implemented, the interpreter kernel and the application and library extensions are linked statically and combined into one executable. .PP In any event, in an application using \*E, the control initially rests in the Scheme interpreter. The interpreter acts as the ``main program'' of the application; it is the interpreter's \f2main()\fP function which is invoked on startup of the program. Therefore the first code to execute in an application is Scheme code; this Scheme code provides the shell functionality of the application (hence it is called \f2shell code\fP). The shell code may perform a few simple tasks, for instance, load a user-provided initialization file containing customization code for the application and then enter the application's main loop, or it may be as complex as in ISOTEXT, where the entire X-based user interface is written in Scheme. .P Making Oneself Known to the Extension Language .PP The application, as it is linked with the extension language interpreter, has full access to all external functions and variables of the interpreter kernel. The interpreter, on the other hand, does not have any knowledge of the contents of dynamically linked and loaded object modules; all it sees of an object file being loaded is the file's symbol table. To obtain ``hooks'' into a newly loaded extension, the interpreter searches the symbol table of each object file being loaded for functions whose names start with the prefix ``elk_init_'' (\f2extension initialization functions\fP) and invokes these functions as they are encountered. Likewise, to support extensions written in C++, any C++ static constructors found in the symbol table are called. When linked statically with its extensions, the interpreter must scan its own symbol table on startup to find and invoke the initialization functions. (Similar support is available for calling extension finalization functions and C++ static destructors on termination.) .PP Besides initializing private data of the modules being loaded, these initialization functions register with the interpreter the Scheme primitives and Scheme data types implemented by the extensions. To enable extensions to register new primitive procedures and types, the interpreter kernel exports two functions: \f2Define_Primitive()\fP to register a new Scheme primitive and \f2Define_Type()\fP to register a new Scheme data type. Both functions take pointers to C functions as arguments that implement the new primitive or the basic access functions of the type (such as the print function and the equality predicates). .PP A simple example for a library extension is presented in Appendix A. .NH Notes on the Implementation .PP Designing \*E, not as another Scheme implementation, but as an extension language kit, provided a design space different from that traditionally available for Lisp implementations. The necessary deviations from the treaded paths of UNIX programming uncovered limitations in portability, aggravated by badly tested corners of standard UNIX facilities. This section discusses the more interesting examples of such issues. .P Implementing Continuations .PP Finding a way to efficiently implement Scheme's \f2continuations\fP called for considerable efforts during the design phase of \*E. Continuations are a powerful language feature; they support the definition of arbitrary control structures such as non-local loop and procedure exits, \f2break\fP and \f2return\fP as in C, exception handling facilities, explicit backtracking, co-routines, or multitasking based on \f2engines\fP [Dybvig 1987]. .PP The primitive procedure .Ss \s+1(call-with-current-continuation \f2receiver\fP)\s0 .Se packages up the current execution state of the program into an object (the \f2continuation\fP or \f2escape procedure\fP) and passes this object as an argument to \f2receiver\fP (which is a procedure of one argument). Continuations are first-class objects in Scheme; they are represented as procedures of one argument (not to be confused with the \f2receiver\fP procedure). Each time a continuation procedure is called with a value, it causes this value to be returned as the result of the \f2call-with-current-continuation\fP expression which created this continuation. If the procedure \f2receiver\fP terminates normally (i.\|e.\& does not invoke the continuation given to it), the value returned by \f2call-with-current-continuation\fP is the return value of \f2receiver\fP. .PP As long as the use of a continuation is confined to the runtime of the \f2receiver\fP procedure, \f2call-with-current-continuation\fP is similar in its functionality to \f2catch/throw\fP in most Lisp dialects or \f2setjmp/longjmp\fP in C. However, continuations, like all procedures in Scheme, have indefinite extent (unlimited lifetime); they can be stored in variables and called an arbitrary number of times, even after the \f2receiver\fP and the enclosing \f2call-with-current-continuation\fP have already terminated. Listing @L(call-cc) shows a program fragment where continuations are used to get back an arbitrary number of times into the middle of an expression whose computation has already been completed. While not particularly useful, this example demonstrates that continuations can be used to build control structures that cannot be implemented by means of less general language features like catch/throw or setjmp/longjmp. .Ls .Ss (define my-function (lambda (n m) (+ n (mark m))) ; return n+m .Sl (define get-back "uninitialized") .Sl (define mark ; identity function, but also (lambda (value) ; assign current continuation (call-with-current-continuation ; to a global variable (lambda (continuation) (set! get-back continuation) ; (assign it) value)))) .Sl .Sl (my-function 10 20) ; invoke my-function \f2prints 30\fP (get-back 5) ; resume with new value \f2prints 15\fP (get-back 0) ; ...once more \f2prints 10\fP .Se .Lc "Using continuations with unlimited extent" .Le call-cc .PP The different approaches applicable to implementing continuations are intimately tied to the strategies used for interpreting the language itself. Scheme interpreters generally employ a lexical analyzer and parser \*- the \f2reader\fP \*- to read and parse the Scheme source code and produce an intermediate representation of the program. During this phase, symbols are collected in a global hash table (in Lisp jargon, the symbols are \f2interned\fP), and a tree structure representing the program's S-expressions is built up on the heap of the interpreter. The majority of interpreters compile this intermediate representation into an abstract machine language (such as \f2byte code\fP). The evaluator is then implemented as an abstract machine which interprets the low-level language; this machine \*- usually a simple stack machine \*- may even be implemented in hardware. .PP In an abstract machine implementation, the straightforward approach to implement \f2call-with-current-continuation\fP is to package up the contents of the abstract machine's registers (program counter, stack pointer, etc.) and runtime stack. Since continuations have indefinite extent, it would not suffice to just capture its registers (as the C library function \f2setjmp\fP does for the real machine). To be able to continue the evaluation of procedures that have already returned and whose frames are therefore no longer on the stack, a continuation must also embody the contents of the abstract machine's stack at the time it is created. When a continuation is applied, the machine resumes the ``frozen'' computation by restoring the saved registers and stack contents of the abstract machine. .PP Just saving the abstract machine's state would not work in \*E, because at the time a continuation is created, arbitrary library functions may be active in addition to Scheme primitives. For instance, consider the \*E interface to the ``Xt'' toolkit intrinsics of the X window system. Here, a typical scenario is that some Scheme procedure invokes the primitive that enters the toolkit's event dispatching main loop (\f2XtAppMainLoop()\fP). When an event arrives (for example, a mouse button press event), the toolkit's main loop invokes a callback function, which in turn calls a user-supplied Scheme procedure to be executed when a mouse button is pressed. This Scheme procedure might in turn invoke yet another function from the ``Xt'' library, and so on. A similar example would be a \f2qsort\fP or \f2ftw\fP extension to \*E, where the user-supplied function called by the \f2qsort()\fP or \f2ftw()\fP C library function would invoke a procedure written in Scheme. .PP The interpreter's thread of execution at any time obviously involves both Scheme primitives and library functions (such as \f2XtAppMainLoop()\fP and \f2qsort()\fP in the examples above) in an arbitrary combination. Therefore, a continuation must embody not only the execution state of the active Scheme procedures, but also that of the currently active library functions (such as local variables used by the library functions). In the approach used by \*E, a continuation is created by capturing the machine's registers \*- like \f2setjmp\fP in C does \*- and the C runtime stack. When a continuation is applied later, the registers and the saved stack contents are copied back. Actually, we did not follow the usual ``abstract machine'' technique in \*E at all; instead, the Scheme evaluator directly interprets the intermediate representation produced by the reader. In a sense, it is the ``real'' machine (the hardware on which \*E is executed) that plays the role the abstract machine plays in implementations with byte-code compilation. .PP Although the abstract machine technique usually yields faster execution of Scheme code, the performance of \*E resembles that of existing interpreters employing this technique, and the implementation of \*E is simpler than that of comparable interpreters using byte-code compilation. While the technique to implement continuations in \*E is not strictly portable \*- it is based on certain assumptions on the machine's stack layout and the C compiler and runtime environment \*- .\"implementations of the small machine-dependent part now exist for it works on most major machine architectures (with two exceptions, which are supported using \f2asm\fP statements). .P The Implementation of ``dump'' .PP Continuations provide a natural basis for implementing the execution-state preserving semantics of the \f2dump\fP primitive. When called, \f2dump\fP invokes \f2call-with-current-continuation\fP. The real work is done in the \f2receiver\fP procedure; it stores the newly created continuation into a global variable, sets a global \f2was-dumped\fP flag to indicate that a dump has taken place, creates an executable file from the image of the running process, and finally returns ``false''. The return value of the \f2dump\fP primitive is the return value of this call to \f2call-with-current-continuation\fP, i.\|e.\& ``false'' if a dump has just been performed. .PP When the interpreter \*- either the original program or a dumped executable \*- is started, it examines the \f2was-dumped\fP flag as its very first action. If the flag is set, the running interpreter was started from a dumped executable. In this case the interpreter immediately invokes, with an argument of ``true'', the continuation that was saved away by a call to \f2dump\fP; this causes that call to \f2dump\fP to finish and return ``true'' to its caller. If, on the other hand, the \f2was-dumped\fP flag is not set (i.\|e.\& the running process was not started from a dumped image), the interpreter initializes and starts up as usual. .PP Before writing an image of the running process to disk, \f2dump\fP has to close all open Scheme file ports, as open file descriptors would not survive a \f2dump\fP \*- they would no longer be valid in the dumped executable. Generally, this is true for all objects pointing to information maintained by the UNIX kernel, such as the current directory, the current signal dispositions, resource limits, or interval timers. Users and implementors of \*E extensions must be aware of this particular restriction. For instance, users of the X11 extensions have to make sure that, if \f2dump\fP is to be used, connections to X-displays are only established in the dumped invocation. .PP To be able to create an executable from the running process, \f2dump\fP has to open and read the a.out file from which the running process was started (actually, if the system linker has been called to dynamically load object files, the output of the most recent invocation of the linker is used instead of the original a.out). The symbol table of the new executable is copied from the a.out file of the running program; in addition, the a.out header has to be read to obtain the length of the text segment and the start of the data segment of the running process. To do so, \f2dump\fP has to determine the filename of the a.out file from which the process was started based on the information in \f2argv[0]\fP and in the PATH environment variable. This approach is obviously based on several prerequisites: \f2dump\fP must be able to access its a.out file (\f2argv[0]\fP must carry meaningful information; the file must be readable) and the running program's a.out file must not have been stripped. It would have been advantageous for the implementation of \f2dump\fP if the entire a.out file were automatically mapped into memory on startup, like it is done, for instance, in NeXT-OS/Mach. .PP \f2dump\fP combines the data segment and the ``bss'' segment of the running process into the data segment of the new executable. If \*E had a separate heap for storing constant objects (future versions may have one), \f2dump\fP could place this read-only part of the memory into the new executable's text segment to make it sharable. When the interpreter's heap is written to disk, \f2dump\fP seeks over the unused portions of the heap, so that fake blocks (holes) can be used for these parts of the file. This results in a considerable conservation of disk space in the final executable, as at least half of the interpreter's heap is unused at any time due to the garbage collection algorithm of \*E. .PP Since the a.out formats used in the numerous versions of UNIX differ vastly, \*E has to include separate implementations of \f2dump\fP for the currently supported a.out formats. Version 2.2 of \*E handles the BSD-style a.out format used in BSD and ``derived'' UNIX versions (such as SunOS 4.1), the COFF a.out format (used in older releases of UNIX System V and in A/UX), Convex SOFF, Extended COFF of MIPS-based computers (DEC, SGI), and the ELF a.out format of System V Release 4 and related UNIX versions (Solaris 2.x, OSF/1). .P Dynamic Loading of Object Files .PP When loading an object file during runtime, addresses within this object file must be relocated to their new location in the program's address space. To allow extensions to directly reference objects of the interpreter kernel, such as the heap and the built-in primitives, unresolved references into the \f2base program\fP must be resolved during dynamic loading. Finally, the object file needs to be able to export its entry points (such as \*E's extension initialization functions) to the base program. .PP More than one object file may have to be loaded into one invocation of \*E. To manage non-trivial, hierarchically structured sets of extensions, where a number of high-level extensions require one or more lower-level extensions to be loaded, it is essential that object files loaded later can make use of the symbols defined by previously loaded object files. As this style of dynamic loading allows building complex systems from small components incrementally, we will use the term \f2incremental loading\fP. .PP With the advent of 4.0\|BSD in 1980 [Joy 1980], support for incremental loading was added to the system linker and has since been supported by most major UNIX variants: when the \-A option and the name of the base executable are supplied to the linker, linking is performed in a way that the object file produced by the linker can be read into the already running executable. The symbol table of the resulting object file is a combination of the symbols defined by the base program and the newly defined symbols added by the linking process, from the object file or from libraries used in linking. Only this newly linked code and data is entered into the resulting object file. The incremental style of dynamic loading is achieved by saving the resulting output file each time the linker is invoked and using this file as the base program for the next incremental loading step, such that both old and new symbols can be referenced. .PP Incremental loading is generally supported by the linkers of UNIX versions that use the BSD-style a.out format and by those of several UNIX systems based on more modern a.out formats (e.\|g.\& Ultrix). It is not supported by any existing release of UNIX System V. Some newer UNIX versions that have shared libraries and dynamic linking (such as System V Release 4 or SunOS) offer a library interface to the dynamic linker. In some systems this kind of interface is intended to replace the incremental loading functionality of the system linker. These dynamic linker interfaces usually come in the form of a library that exports functions such as \f2dlopen()\fP to map a shared object module or shared library into the address space of the caller (the base program) and \f2dlsym()\fP to obtain the address of a function or data item in the newly attached object module. .PP In some implementations, object files attached through \f2dlopen()\fP may directly reference symbols in the base program; in other implementations they may not. In any case, object files cannot directly reference symbols defined by objects that have been placed into the program by previous calls to \f2dlopen()\fP (only, if at all, indirectly by calling \f2dlsym()\fP). Thus, these dynamic linker interfaces are clearly inferior to incremental loading, as they lack the important capability to load a set of object files \f2incrementally\fP. Many vendors who have replaced ``/bin/ld \-A'' by a \f2dlopen\fP-style library in their UNIX systems, or who intend to do so, do not seem to be aware of the fact that this change will break applications that rely on incremental loading. .PP For \*E, the consequence of being restricted to dynamic linker interfaces of that kind is that, except for the simplest applications, one must pre-link all possible combinations of extensions that are not completely independent of each other. In general, given a set of \f2n\fP extensions each of which can be based on one out of \f2m\fP other extensions, this means having to prepare and keep around \f2n\|\(mu\|m\fP pre-linked object files; not to mention the contortions one has to go through when the hierarchy of extensions has a depth greater than two (not an unlikely scenario in practice). If the number of extensions and relations between them is larger than trivial, or if the extensions are large or require large libraries, keeping around all pre-linked combinations of object modules will cause a maintenance problem and may waste a considerable amount of disk space. .PP Another, although minor, problem with these dynamic linker interfaces is that they usually offer only a simple-minded function (such as \f2dlsym()\fP) to look up the address of a specific symbol of a newly accessed object module (typically some kind of module initialization function); but they do not provide a way to scan all newly defined symbols. This functionality is insufficient to implement extension initialization in \*E, where a dynamically loadable extension often is composed from a number of small modules, each defining its own initialization function. Requiring a single, common initialization function name for the entire object file implies that (often configuration-dependent) ``glue code'' must be added to call all the individual initialization functions, including the C++ static constructors. .PP Version 2.2 of \*E supports dynamic loading in environments with ``ld\|\|\-A'' (such as BSD, SunOS 4, Ultrix, and certain versions of SGI Irix and HP-UX), in HP-UX 9 (based on \f2shl_load\fP), and in MACH/NeXT-OS (\f2rld_load\fP). By generating shared objects on the fly, System V Release 4 and SunOS 5 (Solaris 2) are also supported, although in a limited and not yet satisfactory way. .P Non-Standard Language Features .PP As the current version of the Scheme standard (deliberately) does not specify several important language issues, such as error handling or syntactic extensions, we have added a number of non-standard language features to the Scheme interpreter of \*E to fill some of the holes. .PP A proposal for a macro extension has only recently been added as an addendum to the \f2Revised\*(^4 Report on the Algorithmic Language Scheme\fP [Clinger et al. 1991] and is still being discussed controversially within the Scheme community. To avoid having to wait for a final version of a macro system to evolve and be included in the Scheme standard, we implemented a simple-minded macro mechanism in \*E that resembles the macro facilities offered by various existing Scheme and Lisp systems. .PP One area where the Scheme standard does not specify any language features yet is error and exception handling; the standard merely states which error situations a conforming implementation is required to detect and report. Since it is essential for a non-trivial application to be able to gracefully handle error situations (such as failures in interactions with the operating system) and other exceptional conditions, we have added a simple error and exception handling facility to \*E. .PP When an error is detected by the interpreter, a user-supplied error handling procedure is invoked with arguments identifying the type and source of the error. The standard interactive environment (top-level) of \*E provides a default error handler that prints an error message and then resumes the main read-eval-print loop by means of a \f2reset\fP primitive. Most primitives of \*E and the extensions use this error handling facility to signal an error, as opposed to indicating failure by a distinctive return value (which would be prone to being ignored). To by-pass the standard error handler and ``catch'' failure of a particular primitive, programs may enclose the call to the primitive by \f2call-with-current-continuation\fP and dynamically bind the error handler to the continuation (as shown in listing @L(errset)). .Ls .Ss (define (open-input-file-or-not name) (call-with-current-continuation (lambda (return) ; \f6return\fP becomes escape procedure (fluid-let ((error-handler ; rebind \f6error-handler\fP (lambda args (return #f)))) (open-input-file name))))) .Se .Lc "A version of open-input-file that returns the newly opened port \ on success, #f on error" .Le errset .PP \*E provides a similar facility to handle an \f2interrupt\fP exception: a user-supplied interrupt handler is invoked when a SIGINT signal is sent to the interpreter (usually by typing the interrupt character on the keyboard). Support for other exceptions, such as timer interrupts, may be provided in future versions. .PP Another non-standard primitive that facilitates handling of errors is \f2dynamic-wind\fP, a generalization of the \f2unwind-protect\fP form offered by many Lisp dialects. \f2dynamic-wind\fP is used to implement the \f2fluid-let\fP special form (to create \f2fluid\fP or dynamic variable bindings). Both \f2dynamic-wind\fP and \f2fluid-let\fP are also provided by several other Scheme dialects [MIT 1984, Dybvig 1987]. .PP The current version of the Scheme standard does not provide any language features that would make it possible to implement a useful Scheme debugger (apart from a debugger based on source code instrumentation). To compensate for this shortcoming, we have added a few primitives that aid the implementation of a simple interactive debugger, among them an \f2eval\fP primitive (although, in theory, \f2eval\fP could be implemented by writing an expression into a temporary file and then loading this file). In addition, \*E, like a few other Scheme dialects, provides lexical environments as first class (but immutable) objects. Other non-standard primitives that aid writing debuggers are \f2procedure-lambda\fP to obtain the lambda expression that evaluated to a given procedure, and a primitive that returns the list of currently active procedures together with their actual arguments and the lexical environments in which the procedure calls took place (a \f2backtrace\fP). .\" .\" provide, require; autoloading .P Garbage Collection .PP The garbage collector of \*E is based on the \f2stop-and-copy\fP algorithm (see e.\|g. [Abelson et al. 1985]). The heap area is divided into two \f2semispaces\fP, only one of which is active during normal operation. In a garbage collection, all objects that are still reachable are moved into the unused semispace; the previously used semispace then remains unused until the next garbage collection. An incremental, generational garbage collector for \*E, inspired by Yip's garbage collector [Yip 1991], has recently been implemented as an alternative to the stop-and-copy garbage collector\**. .FS With a generational garbage collector, objects surviving garbage collections will not be touched again until there is only a certain amount of memory left on the heap, triggering a full garbage collection. Particularly in applications with large amounts of Scheme code or other constant data, partial GCs run much faster than full GCs. With incremental garbage collection, starting a garbage collection does not suspend the application until the GC is done; instead, the collector returns control to the application almost immediately (after having marked pages of interest unreadable with the \f2mprotect\fP system call) and regains control with a SIGSEGV signal. .FE .PP Extensions to \*E can register \f2before-GC\fP and \f2after-GC\fP functions with the interpreter; these functions are invoked by the garbage collector immediately before and after each garbage collection run. Within \f2after-GC\fP functions, extensions can determine whether a particular Scheme object has become garbage, i.\|e. no references to the object exist any longer. In this case, an extension may perform some kind of clean-up action; for example, if the now unreferenced object contains a handle to an open file, close this file. .PP The \*E distribution contains a library based on this mechanism that enables extensions to register a \f2termination function\fP for objects of a particular type. The termination function associated with an object is then invoked by the garbage collector automatically when this object has been detected to be unused. The Xlib extension of \*E uses this library to perform suitable finalization operations on objects created by the extensions, for example, close windows, unload fonts, and free colormap objects that have become unreferenced. This mechanism is slightly complicated by the fact that objects may have to be terminated in a predefined order; for instance, when an X11 display becomes garbage, all objects associated with this display must be terminated before the display itself is finally closed. .P Library Extensions .PP The problems we encountered when designing and implementing \*E's interfaces to the C libraries of X11 are likely to apply to a wide range of similar APIs. The X11 libraries, especially Xlib, are quite complex; the core Xlib alone exports more than 600 functions and macros, with numerous different mechanisms for passing arguments and for manipulating objects, some of which could be considered rather verbose and error-prone. This complexity is, at least partly, caused by the semantic restrictiveness of the C programming language. Thus, when designing the Scheme language interface, we had the opportunity to eliminate some of the ``warts.'' .PP If integration of a library with an extension language (or interactive language in general) is not anticipated at the time the programmer's interface of the library is designed, writing a properly functioning extension language interface to this library can become rather challenging or even impossible. This problem is exemplified by the ``Xt'' toolkit intrinsics library of X11, in particular by earlier versions of this library. The following example illustrates a typical difficulty caused by the ``static'' nature of the programmer's interface to ``Xt'': .PP Each class of graphical objects (\f2widgets\fP in ``Xt'' terminology) exports a list of attributes (\f2resources\fP) that are associated with objects of this class. A function is provided by ``Xt'' to obtain the list of resources of a widget class together with the name and C type (integer, string, pixmap, color, etc.) of each resource. On this basis, operations like setting the value of a widget's resource from within Scheme can be implemented in a straightforward way. The ``Xt'' extension just has to check if the user-supplied Scheme value can be converted into a C object of the resource's type, perform this conversion, and call the Xt-function to set the resource, or complain to the user if the value is not suitable for this resource. However, in early versions of Xt, some classes of widgets had a subset of resources (the \f2constraint resources\fP) whose names and types could not be obtained by an ``Xt'' application. While this omission was usually not perceived as a problem for C programmers (who would know each widget's resources \f2a priori\fP from reading the documentation), it had a dramatic effect on \*E's ``Xt'' extension, as now the knowledge about these resources had to be hard-wired into the extension. As a result, the extension's source code had to be modified for each new widget set to be made usable from within Scheme code. .PP This particular problem has been remedied in recent releases of X11, though several similar problems remain; even in the UNIX C library. While design flaws of library interfaces often go unnoticed or are considered minor when writing C or C++ programs (e.\|g.\& the fact that implementations of the \f2qsort()\fP functions are non-reentrant), they become crucial when these libraries are made accessible to an extension language. As the importance of extension languages is growing, it is essential that future library interfaces are designed with the particular requirements of extensions languages in mind. .PP .\" automatic generation of interfaces / foreign functions .NH Practical Experiences with \*E .P \*E and ISOTEXT .PP In developing the document processing system ISOTEXT, \*E proved to be a major asset [Bormann 1991]. Scheme was used as the implementation language for all user interface aspects of ISOTEXT. Apart from providing extensibility to users of ISOTEXT, using \*E as the base for ISOTEXT made it possible to write the shell code in a high level language with all its amenities, e.\|g.\& automatic storage reclamation. As no recompilation and relinking is necessary, it is a quick operation to apply and test changes to the user interface. .PP \*E provides for a strong ``firewall'' in the ISOTEXT system: bugs in the Scheme code give rise to errors at the Scheme level, which can easily be debugged using the (primitive, but functional) built-in debugger of \*E, while conditions such as core dumps always are the result of bugs in the ISOTEXT kernel implementation. .PP All this assistance for the development of ISOTEXT could be obtained without sacrificing the performance of the ISOTEXT kernel system, which is still written in efficient C++. .PP \*E also allowed us to isolate the ISOTEXT kernel from the choice of an X toolkit: the ISOTEXT kernel is unaware of the toolkit being used (``Xt'' with OSF/Motif). The Scheme code builds a user interface using the Motif library interface and provides X windows to the ISOTEXT kernel. Input is processed by the Scheme code which calls editor primitives provided by the ISOTEXT kernel and schedules redisplay operations. Replacing Xt and OSF/Motif by e.\|g.\& \f2Xview\fP would require no changes in the ISOTEXT kernel. .\".PP .\"As extensions and the \*E kernel are effectively linked together, the .\"current interface between the two allows extensions to call every .\"global function in the \*E kernel. .\"This makes it difficult to rewrite in Scheme primitives that originally .\"were written in C. .PP The work on ISOTEXT clearly identified one single main problem in writing non-trivial extensions: as any request for new heap space can trigger a garbage collection, extensions must register local or temporary Scheme objects with the garbage collector to protect them from being discarded during a GC run caused by any nested procedure call. While this scheme has the advantage that maximum utilization of the available heap space is guaranteed, it imposes a strict discipline on the extension programmer. Failure to properly protect temporary Scheme objects usually results in delayed crashes of the application that are hard to trace back to the actual source of the problem. For instance, when developing the X11 extensions to \*E, most of the time spent for debugging was due to GC-related bugs. .\" Similarly, the following bug in the interpreter kernel went unnoticed for years: .\" .Ss .\" \s+1newframe = Add_Binding(newframe, Car(b), Eval(val));\s0 .\" .Se .\" Depending on the C compiler used, \f2newframe\fP is pushed on the argument .\" stack before \f2Eval\fP, which may trigger a GC, is called. .\" The GC generally moves the object to which \f2newframe\fP points, .\" updating the variable \f2newframe\fP, but not the copy .\" on the argument stack, which is now a dangling reference. .\" When the GNU C compiler uncovered this problem, the line was changed to .\" the proper: .\" .Ss .\" \s+1temp = Eval(val);\s0 .\" \s+1newframe = Add_Binding(newframe, Car(b), temp);\s0 .\" .Se .\" .\"(cabo: recruiting problems) .P \*E and TELES.VISION .PP Another example for using Elk and its X interface as the basis for a user interface subsystem is the TELES.VISION desktop video conferencing system [TELES 1991]. First, a somewhat generalized User Interface Management System was built in about 1500 lines of Scheme, which was then instantiated to build a number of revisions of the TELES.VISION user interface. The user interface communicates with the rest of the conferencing system via a remote procedure call C library, using Scheme continuations as a basis for a simple form of multithreading. According to the TELES.VISION implementors [Bastian 1993], Elk was a ``perfect fit'' for this application, with the single exception that its initial garbage collector placed too heavy a burden on the memory starved initial environment (where 8 MB of memory had to be shared between an operating system, various realtime device drivers, drivers for video codec hardware, and an MS-Windows emulation subsystem). This has since been remedied by adding memory. Using Elk also helped when TELES.VISION was ported to OS/2 \*- in particular, its continuations ported easily. Also, Elk was used in the TELES.VISION project to build a rapid prototype of the central conference management subsystem (again using continuations to provide multithreading) within less than two weeks. .P Other Projects .PP While \*E has been used in the ISOTEXT project since 1987, legal issues prevented making it publicly available until the fall of 1989. Since, \*E has gained acceptance, in fact sufficient momentum to encourage others to contribute software. Elk has been used successfully as an extension language for a hypertext database, a distributed version management system, various CAD programs, testing and simulation systems for digital circuits as well as environmental models. It also has found use simply as a Scheme programming environment, in particular for its X and Motif interface. .PP The X extensions have proven useful in particular for writers of applications with graphical user interfaces based on X; \*E enables them to write their user interfaces or parts thereof in Scheme to achieve a high degree of customizability. .PP \*E also has found use as a free-standing Scheme implementation. In combination with the X extensions it is well-suited for teaching X to beginners, as a tool for interactively exploring X, and as a platform for rapid prototyping of X-based applications. .PP Outside of the UNIX world, we are aware of user-done ports to DOS (both 16 bit and 32 bit using DJGPP), OS/2, and MacOS. .PP Users cited the following features as significant for their choice of Elk: dynamic object code loading, dumping of ready-to-run executables, Elk's performance, its legally unencumbered availability, and finally its simplicity and adaptability (and, as users say, its consistent, clean and well-structured code). .PP Users are not happy with various artificial limitations still in the system (such as the static heap size which with the stop-and-copy garbage collector needs to be fixed at invocation time), with Elk's performance, and with the fact that Elk ``likes to be in control'' (i.e.\&, supplies the main program). In addition, prospective users tend to ponder acceptance problems with their fellow workers and customers (who might not be well versed in Lisp/Scheme) before committing to Elk. Finally, for many extension language applications, Elk is ``too big'', and users have asked for versions without the more expensive Elk features such as arbitrary size number support or continuations. On the other hand, users have asked for additional features such as an inter-process communication interface, or a better debugger. Also, a port to MS-Windows has been actively sought. .NH Conclusions .PP Since the \*E project began, both the research community and significant industry projects have generated increasing numbers of ``embeddable language'' implementations. While many such languages inherit the syntactic flavor of BASIC, those projects that focus on the ability to build non-trivial extensions recently seem to almost exclusively turn to the Scheme language. .PP Scheme has proven to be an effective language for extension language purposes. In the beginning of the ISOTEXT project, there were concerns that an implementation of the full Scheme language would be both too large and too slow. These reservations proved to be unfounded: the binary code size of \*E is still significantly below that of a medium size application such as \f2vi\fP. While the performance of \*E may be uninspiring (no compiler is available), this turned out not to be a critical issue, as any bottlenecks can easily be replaced by a primitive recoded in C or C++. .PP There also were concerns that Scheme was going to be hard to learn for UNIX users familiar with, say, the Bourne Shell and C. This seems to be more of a problem with initial acceptance than with a steep learning curve: after having overcome the initial barrier (which generally had to do mainly with perceiving the syntax as queer), users reported the same rapid increase in productivity they already knew from shell programming. It certainly has not been necessary to recruit Lisp programmers to be able to extend applications with \*E. .PP .ds Tx "T\v'.2m'E\v'-.2m'X Finally, \*E was an exercise in writing portable software without being restricted to what is considered portable today. Apart from the well-known problem that true portability between current relevant platforms cannot be attained by just picking one of the proclaimed ``standards'', and the unwieldy situation that there are too many standards for (auto-)configuration of software, a significant part of the effort in generating \*E was consumed by devising support for each new platform for dynamic loading, generation of executables from running programs, and switching between threads of control (continuations). Note that many non-trivial applications of today (apart from Lisp programming environments, GNU emacs and \*(Tx come to mind) need one or more of these features; also note that most relevant current platforms can be made to support these features quite well \*- just in wildly different ways. .NH Availability .PP \*E is available in legally unencumbered status. The current version as of June 1994 is 2.2. The most recent version of \*E is available via anonymous FTP from ftp.x.org (/contrib) and ftp.fu-berlin.de (/pub/unix/languages/scheme). .NH Acknowledgments .PP An early version of \*E was written while one of us was employed at TELES GmbH, Berlin. We are grateful to Prof.\& Dr.\& Sigram Schindler of TELES and TU Berlin for providing the work environment for ISOTEXT and \*E and for the permission to publish this software. .PP The present version is a result of our research work at Technische Universit\(a:t Berlin, with the benefit of the work of many contributors. In particular, we wish to thank Marco Scheibe who wrote the generational, incremental garbage collector. .NH References .IP "[Abelson et al. 1985] Harold Abelson and Gerald J. Sussman with Julie Sussman, \f2Structure and Interpretation of Computer Programs\fP, MIT Press, Cambridge, Mass., 1985. .\" .IP "[Bastian 1993]" Personal communication with Jan Bastian, TELES. .\" .IP "[Bormann et al. 1988] Ute Bormann, Carsten Bormann, C. Bathe, SDE \*- A WYSIWYG Editing and Formatting System for ODA and SGML Documents, ESPRIT '88, \f2Proceedings of the 5th Annual ESPRIT Conference, Brussels\fP, November 14-17, 1988. .\" .IP "[Bormann 1991]" Carsten Bormann, Open Document Processing and the ISOTEXT System, Doctoral Dissertation, TU-Berlin, 1991. .\" .IP "[CFI 1991a]" CAD Framework Initiative, CFI Extension Language Sub-Committee, \f2CFI Extension Language Selection Document\fP, CFI Document Number 87, CAD Framework Initiative Inc., Austin, Texas, 1991. .\" .IP "[CFI 1991b]" CAD Framework Initiative, Extension Language Working Group: Architecture Technical Sub-Committee, \f2Extension Language: Core Language Selection\fP, Draft Proposal Version 0.7, CFI Document Number ARCH-91-G-1, CAD Framework Initiative Inc., Austin, Texas, 1991. .\" .IP "[Clinger et al. 1991]" William Clinger and Jonathan Rees (Editors), \f2Revised\*(^4 Report on the Algorithmic Language Scheme\fP, November 2, 1991. Available as ftp://cs.indiana.edu/pub/scheme-repository/doc/r4rs.ps.Z. .\" .IP "[CLX 1991]" CLX \*- Common LISP X Interface, 1991. (Part of the X11 Release 5 distribution available from the MIT software distribution center.) .IP "[Cowlishaw 1985]" M. F. Cowlishaw, \f2The REXX Language \*- A Practical Approach to Programming\fP Prentice Hall, Englewood Cliffs, NJ, 1985. .\" .IP "[Dybvig 1987]" R. Kent Dybvig, \f2The Scheme Programming Language\fP, Prentice Hall, Englewood Cliffs, NJ, 1987. .\" .IP "[Hansen 1990]" Wilfred J. Hansen, Enhancing documents with embedded programs: How Ness extends insets in the Andrew ToolKit, \f2Proceedings of IEEE Computer Society 1990 International Conference on Computer Languages\fP, March 12-15, 1990, New Orleans. .\" .IP "[IEEE\|Std\|1178-1990]" \f2IEEE Standard for the Scheme Programming Language\fP, New York, May 28, 1991 (approved December 10, 1990). .\" .IP "[Joy 1980]" Bill Joy, Changes in the VAX system in the Fourth Berkeley Distribution, Computer Systems Research Group, University of California, Berkeley, November 1980. .\" .IP "[Lewis et al. 1990]" Bil Lewis, Dan LaLiberte, the GNU Manual Group, GNU Emacs Lisp Reference Manual, Edition 1.03, Free Software Foundation, Cambridge, Mass., December 1990. .\" .IP "[MIT 1984]" MIT Scheme Manual, Seventh Edition, Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, Mass., September 1984. .\" .IP "[Ossanna 1979]" J. F. Ossanna, Nroff/Troff User's Manual, UNIX Programmer's Manual, Seventh Edition, vol. 2, Bell Telephone Laboratories, Murray Hill, NJ, January 1979. .\" .IP "[Ousterhout 1990]" John K. Ousterhout, Tcl: An Embeddable Command Language, \f2Proceedings of the USENIX 1990 Winter Conference\fP, January 1990, pp. 133-146. .\" .IP "[Scheifler et al. 1986]" Robert W. Scheifler and Jim Gettys, The X Window System, \f2ACM Transactions on Graphics\fP, vol. 5, no. 2, pp. 79-109, 1986. .\" .IP "[Scheifler et al. 1992]" Robert Scheifler and James Gettys, \f2X Window System\fP, Third Edition, Digital Press, 1992. .\" .IP "[Springer et al. 1989]" George Springer and Daniel O. Friedman, \f2Scheme and the Art of Programming\fP, MIT Press, Cambridge, Mass., 1989. .\" .IP "[Stallman 1981]" Richard M. Stallman, EMACS \*- The Extensible, Customizable, Self-documenting Display Editor Production System, \f2SIGPLAN Notices\fP, vol. 16, no. 6, pp. 147-156, Association for Computing Machinery, New York, 1981. .\" .IP "[TELES 1991] Das TELES.VISION System \*- Philosophie und Technologie, TELES GmbH, Berlin, 1991 (in German). .\" .IP "[Yip 1991]" G. May Yip, Incremental, Generational Mostly-Copying Garbage Collection in Uncooperative Environments, WRL Research Report 91/8, DEC Western Research Laboratory, Palo Alto, California, 1991. .ie \n(.U \{\ .NH S A Appendix: Extending \*E \*- An Example .\} .el \{\ .bp .SH .nr H1 1 .af H1 A Appendix A: Extending \*E \*- An Example .\} .P The ``ndbm'' Library Extension .PP The extensibility mechanisms of \*E can be demonstrated best by examining a simple library extension. Consider the \f2ndbm\fP library that is available on most versions of UNIX. This library implements functions to maintain a simple database file of key/contents pairs. .PP As shown in Listing @L(ndbm), both the keys and the data to be stored are described by the type \f2datum\fP; it consists of the data (a string of bytes) and the length of the data. \f2dbm_open()\fP opens a database file and returns a handle to that file to be used in subsequent operations on that database (a pointer to an opaque data type, similar to the \f2fopen\fP and \f2readdir\fP interfaces); it returns a null pointer if the file could not be opened. A database is closed by a call to \f2dbm_close()\fP. The data stored under a given key is accessed by the function \f2dbm_fetch()\fP; it returns an object of type \f2datum\fP (with a null \f2dptr\fP if the key could not be found). \f2dbm_store\fP is used to insert an entry into a database and to modify an existing entry; it returns zero on success and a non-zero value on error. .Ls .Ss #include .Sl .Sl typedef struct { char *dptr; int dsize; } datum; .Sl .Sl DBM *dbm_open(char *file, int flags, int mode); .Sl void dbm_close(DBM *db); .Sl datum dbm_fetch(DBM *db, datum key); .Sl int dbm_store(DBM *db, datum key, datum data, int flags); .Se .Lc "The UNIX \f2ndbm\fP library" .Ns \f2Note:\fP For simplicity, several functions have been omitted. The \f2flags\fP and \f2mode\fP arguments of \f2dbm_open\fP are that of the \f2open\fP system call. The \f2flags\fP argument of \f2dbm_store\fP can be DBM_INSERT to insert a new entry into the database or DBM_REPLACE to change an existing entry. .Ne .Le ndbm .PP The straightforward way to write an \f2ndbm\fP extension to \*E is to provide a new Scheme data type \f2dbm-file\fP together with the obligatory type predicate \f2dbm-file?\fP and the Scheme primitive procedures \f2dbm-open\fP, \f2dbm-close\fP, \f2dbm-fetch\fP and \f2dbm-store\fP that operate on objects of type \f2dbm-file\fP. .PP \f2dbm-open\fP receives the filename (a string or a symbol); the second argument is one of the symbols \f2reader\fP (open the file read-only), \f2writer\fP (read and write access), and \f2create\fP (read and write access, create new file if it does not exist). The optional filemode argument is an integer. \f2dbm-open\fP returns an object of type \f2dbm-file\fP or #f (false) if the file could not be opened. \f2dbm-close\fP closes the database file associated with its argument of type \f2dbm-file\fP. As this function is called for its side-effect only, and for lack of a better result, it returns a non-printing object. .PP \f2dbm-fetch\fP expects a \f2dbm-file\fP and a string argument (the key to be searched) and returns a string (the data stored under the key) or #f if the key does not exist. Note that in \*E strings may contain arbitrary 8-bit characters, including the null byte. \f2dbm-store\fP is called with a \f2dbm-file\fP, two strings (key and data) and one of the symbols \f2insert\fP and \f2replace\fP. Its integer return value is the return value of \f2dbm_store()\fP. .PP These procedures and the new \f2dbm-file\fP type can be used by application programmers to manipulate database files in those parts of their applications that are written in Scheme. Listing @L(ndbm-example) shows a small example. .Ls .Ss (define expand-mail-alias (lambda (alias) (let ((d (dbm-open "/etc/aliases" 'reader))) (if (not d) (error 'expand-mail-alias "cannot open database")) (unwind-protect (dbm-fetch d alias) (dbm-close d))))) .Sl (define address-of-staff (expand-mail-alias "staff")) .Se .Lc "Using the ndbm extension" .Ns \f2Note:\fP The \f2unwind-protect\fP and the \f2error\fP form are not present in standard Scheme. .Ne .Le ndbm-example .P The Anatomy of a Scheme Type .PP Listing @L(ndbm-skeleton) shows the part of the extension that deals with the new data type \f2dbm-file\fP and the extension initialization function. The variable \f2T_Dbm\fP will hold the unique identifier of the newly defined type. The structure \f2S_Dbm\fP defines the C representation of the type; one such C structure is declared for each composite Scheme type. Its main component is the handle of the database file that is contained in each object of type \f2dbm-file\fP. .Ls .Ss #include #include .Sl int T_Dbm; .Sl struct S_Dbm { DBM *dbm; char alive; /* 0 or 1 */ }; .Sl #define DBMF(obj) ((struct S_Dbm *)POINTER(obj)) .Sl int Dbm_Equal(a, b) Object a, b; { return DBMF(a)->alive && DBMF(b)->alive && DBMF(a)->dbm == DBMF(b)->dbm; } .Sl void Dbm_Print(d, port) Object d, port; { Printf(port, "#[dbm-file %lu]", DBMF(d)->dbm); } .Sl Object P_Is_Dbm(x) Object x; { return TYPE(x) == T_Dbm ? True : False; } .Sl void elk_init_dbm() { Define_Primitive(P_Is_Dbm, "dbm-file?", 1, 1, EVAL); Define_Primitive(P_Dbm_Open, "dbm-open", 2, 3, VARARGS); Define_Primitive(P_Dbm_Close, "dbm-close", 1, 1, EVAL); Define_Primitive(P_Dbm_Store, "dbm-store", 4, 4, EVAL); Define_Primitive(P_Dbm_Fetch, "dbm-fetch", 2, 2, EVAL); .Sl T_Dbm = Define_Type("dbm-file", sizeof(struct S_Dbm), Dbm_Equal, Dbm_Equal, Dbm_Print, NOFUNC); } .Se .Lc "Skeleton of the ndbm extension" .Ns \f2Note:\fP For simplicity some details have been omitted in this listing, and the calling interface of some functions has been simplified; the program would not compile in this form. A working \f2gdbm\fP (GNU dbm) extension is included in the \*E distribution. .Ne .Le ndbm-skeleton .PP Scheme objects can usually live longer than their underlying C objects. In case of the \f2dbm-file\fP type, a Scheme object of that type can obviously still be referenced after its database handle has been closed by a call to \f2dbm-close\fP. As \*E extensions must not crash the application, we must prevent such stale objects from being used in further calls to \f2dbm-fetch\fP, \f2dbm-store\fP, and \f2dbm-close\fP. One way to achieve this is to record in each Scheme object whether the underlying C object is still alive or has been terminated. The boolean component \f2alive\fP in the \f2dbm-file\fP type serves this purpose. It is initialized with true and is set to false in \f2dbm-close\fP. Further operations on objects with \f2alive\fP being false are rejected. .PP The interpreter stores all Scheme objects in variables of type \f2Object\fP. An \f2Object\fP is typically a 32-bit value; it is composed of a \f2tag\fP part and a \f2pointer\fP part. The \f2tag\fP part indicates the type of the object, and the remaining bits hold the actual memory address of the object (they point into the interpreter's heap). The macros \f2TYPE\fP and \f2POINTER\fP are provided to extract the fields of an \f2Object\fP. Each type definition must define a macro to extract the object's memory address from an \f2Object\fP (by means of \f2POINTER\fP) and then cast it into a pointer to the underlying C structure (see \f2#define DBMF\fP in Listing @L(ndbm-skeleton)). .PP \f2Dbm_Equal()\fP implements both the \f2eqv?\fP and the \f2equal?\fP predicates for \f2dbm-file\fP objects; it returns true if both objects being compared are alive and contain identical \f2DBM\fP handles. .PP \f2Dbm_Print()\fP is called by the interpreter each time an object of type \f2dbm-file\fP is to be printed; it is invoked with the object and the Scheme port to which the output is to be sent. .PP \f2P_Is_Dbm()\fP implements the primitive procedure \f2dbm-file?\fP (the type predicate). As with all primitives, it receives arguments of type \f2Object\fP and returns an \f2Object\fP, and it has a name beginning with ``P_''. .PP The definition of the initialization function \f2elk_init_dbm()\fP is straightforward; it invokes \f2Define_Primitive()\fP once for each primitive procedure and finally \f2Define_Type()\fP to make the new type known to the interpreter. .PP The arguments that can be supplied to \f2Define_Primitive()\fP are a pointer to the function implementing the primitive procedure, the Scheme name of the primitive, the minimum and maximum number of arguments, and a symbol indicating the \f2calling discipline\fP of the primitive. For most of the functions in this example, the calling discipline is \f2EVAL\fP, indicating a normal procedure with a fixed number of arguments, such as \f2car\fP. Elk also supports procedures with variable argument list, such as \f2list\fP (\f2VARARGS\fP); and \f2NOEVAL\fP for \f2special forms\fP (variable number of unevaluated arguments). .PP \f2Define_Type()\fP is invoked with the Scheme name of the type, the size of the type's representation in C or C++ (given as a constant or as a function), two functions implementing the \f2eqv?\fP and \f2equal?\fP predicates for objects of this type, a function that is called by the interpreter to print an object of the new type (the type's \f2print function\fP), and a function providing information about the type to the garbage collector. The return value of \f2Define_Type()\fP is a ``handle'' to the newly defined type (a small, unique integer); its main uses are to check the type of arguments supplied to primitive procedures and to instantiate objects of this type. .P Primitive Procedures \*- The Details .PP Listing @L(dbm-open) gives the definitions of the primitives \f2dbm-open\fP and \f2dbm-close\fP. .Ls .Ss static SYMDESCR Flag_Syms[] = { { "reader", O_RDONLY }, { "writer", O_RDWR }, { "create", O_RDWR|O_CREAT }, { 0, 0 } }; .Sl Object P_Dbm_Open(argc, argv) int argc; Object *argv; { char *p; DBM *dp; Object d; .Sl Make_C_String(argv[0], p); dp = dbm_open(p, Symbols_To_Bits(argv[1], 0, Flag_Syms), argc == 3 ? Get_Integer(argv[2]) : 0666); if (dp == 0) return False; d = Alloc_Object(sizeof(struct S_Dbm), T_Dbm, 0); DBMF(d)->dbm = dp; DBMF(d)->alive = 1; return d; } .Sl void Check_Dbm(d) Object d; { Check_Type(d, T_Dbm); if (!DBMF(d)->alive) Primitive_Error("invalid dbm-file: ~s", d); } .Sl Object P_Dbm_Close(d) Object d; { Check_Dbm(d); DBMF(d)->alive = 0; dbm_close(DBMF(d)->dbm); return Void; } .Se .Lc "ndbm extension \*- implementation of \f2dbm-open\fP and \f2dbm-close\fP .Le dbm-open .PP \f2dbm-open\fP, as it has an optional argument, is a function with \f2VARARGS\fP calling discipline (not to be confused with the C language feature of the same name), as indicated by the last argument to the \f2Define_Primitive\fP call. Primitives of this type receive an array of \f2Objects\fP and a count. .PP The initial call to the macro \f2Make_C_String\fP checks if the first argument to \f2dbm-open\fP is a string (or a symbol) and converts it to a C string. To obtain the second argument to \f2dbm_open()\fP, the symbol passed to the Scheme primitive (\f2reader\fP, \f2writer\fP, etc.) has to be mapped to a corresponding flags combination (\f2O_RDONLY\fP, \f2O_RDWR\fP, etc.). This is accomplished by the \*E function \f2Symbols_To_Bits()\fP; it is invoked with a Scheme symbol, a flag indicating whether a single symbol or a list of symbols (a mask) is to be converted, and a table of pairs of symbol names and C integers. The third argument to \f2dbm_open\fP is the filemode; \f2Get_Integer()\fP converts a Scheme number to a C integer. \f2dbm-open\fP finally allocates a new Scheme object of type \f2T_Dbm\fP on the heap, initializes the components of the object, and returns it. .PP The auxiliary function \f2Check_Dbm()\fP is used by the remaining primitives to check whether a given object is of type \f2dbm-file\fP and if so, whether it is stale. In this case an error is signaled; \f2Primitive_Error()\fP enters the error handler of \*E. .PP \f2P_Dbm_Close()\fP just marks the object as stale by setting \f2alive\fP to false and closes the database file. .PP Listing @L(dbm-store) shows the implementation of \f2dbm-store\fP and \f2dbm-fetch\fP. \f2Make_Integer()\fP is the counterpart to \f2Get_Integer()\fP; it converts a C integer into a Scheme number. Likewise, \f2Make_String()\fP converts a C string into a Scheme string. .bp .Ls .Ss static SYMDESCR Store_Syms[] = { { "insert", DBM_INSERT }, { "replace", DBM_REPLACE }, { 0, 0 } }; .Sl Object P_Dbm_Store(d, key, content, flag) Object d, key, content, flag; { datum k, c; int result; .Sl Check_Dbm(d); Check_Type(key, T_String); Check_Type(content, T_String); k.dptr = STRING(key)->data; k.dsize = STRING(key)->size; c.dptr = STRING(content)->data; c.dsize = STRING(content)->size; result = dbm_store(DBMF(d)->dbm, k, c, Symbols_To_Bits(flag, 0, Store_Syms)); return Make_Integer(result); } .Sl Object P_Dbm_Fetch(d, key) Object d, key; { datum k, c; .Sl Check_Dbm(d); Check_Type(key, T_String); k.dptr = STRING(key)->data; k.dsize = STRING(key)->size; c = dbm_fetch(DBMF(d)->dbm, k); return c.dptr ? Make_String(c.dptr, c.dsize) : False; } .Se .Lc "ndbm extension \*- implementation of \f2dbm-store\fP and \f2dbm-fetch\fP .Le dbm-store .\" ---------- .\" Erwaehnen: Little Languages .\" ---------- .\" Read the CFI papers again