+FFIGEN Manifesto and Overview
+(HTML) (ps.gz, 26 KB)
++
+FFIGEN User's Manual
+(HTML) (ps.gz, 30 KB)
++
+FFIGEN Back-end for Chez Scheme Version 5
+(ps.gz, 52 KB)
+
++ +FFIGEN (Foreign Function Interface GENerator) is a program suite that +facilitates the writing of translators from C header files to foreign +function interfaces for particular language implementations. + +
+
+
+FFIGEN Manifesto and Overview
+(HTML) (ps.gz, 26 KB)
+
+
+FFIGEN User's Manual
+(HTML) (ps.gz, 30 KB)
+
+
+FFIGEN Back-end for Chez Scheme Version 5
+(ps.gz, 52 KB)
+
+
+There are three motivating observations behind FFIGEN. The first is +that C header files are hard to parse because of the preprocessor, +general syntactic grunge, and the problem of getting the data layouts +right. The second is that foreign function interfaces differ widely and +that translations to different FFIs can't be the same, yet should share +work as much as possible. The third is that not all translations are +suitable for all purposes; there may be multiple valid translations - +each of which serves a different need - for any given language's FFI. + +
+ +For these reasons, a translator from C header syntax to an FFI should +have two parts: one target-independent front-end that translates from +the header file into a rational intermediate form and which can be used +with all translators, and a target-dependent back-end that translates +from the intermediate form to an FFI for the target system, using a +translation policy to guide the translation. This design nicely +facilitates writing back-ends for multiple languages, multiple FFIs per +language, and multiple policies per FFI. + +
+ +FFIGEN is a system that implements the split-translation philosophy. + +
+ +
+ +The current version of FFIGEN is available as a set of modifications to +lcc version 3.4b; you also need to get the lcc sources. +The FFIGEN +distribution includes documentation on how to write back-ends and a +documented example back-end for the FFI of Chez Scheme version 5. + +
+ This is a preliminary release of FFIGEN. It works, but +is neither complete nor polished. + +
+
+Click here to download the
+full FFIGEN distribution. (148 KB)
+
+This archive has not been updated with the fixes in the bug fix file (below).
+
+
+Click here to download bug fixes up to February 13, 1996. (29 KB)
+Fixes to chez.sch to handle structs/unions that are declared but not
+defined; function pointers; and unsigned shorts (a typo). Also a minor fix
+to policy.sch to remove gratuitous non-standard-ness (use of reverse! rather
+than reverse). Also included generated standard libraries for Chez Scheme
+back-end (unknowingly left out of distribution). Unpack in lcc main
+directory.
+
+
+Click here to download an example of a Chez
+Scheme policy file, left out of distribution.
+
+
+Click here to download the lcc
+3.4b distribution.
+(965 KB)
+
+
+ +
+ +Related systems: + +
+
+ +The FFIGEN to-do list. + +
+lth@acm.org
+24 May 2000
+
+
diff --git a/www/manifesto.html b/www/manifesto.html
new file mode 100644
index 0000000..1bfb05c
--- /dev/null
+++ b/www/manifesto.html
@@ -0,0 +1,241 @@
+
+
+
+
++ +FFIGEN (Foreign Function Interface GENerator) is a program suite which +facilitates the writing of translators from C header files to foreign +function interfaces for particular language implementations.
+ +On a more general level, FFIGEN is a statement about how such +translators should be structured for maximum usability, namely as a +single translator from C to a rational intermediate language and as +multiple translators from the intermediate language to separate FFI +translations. In the present document I motivate this two-level +structure by arguing that the many policy questions inherent in choosing +a mapping from one language to another cannot be accomodated in a single +translator, and that the two-level structure promotes significant code +reuse. Companion documents present the program suite itself.
+
Many language implementations have mechanisms which provide support for +call-outs to other, typically more primitive, languages. In particular, +implementations of very-high-level languages like Scheme, Common Lisp, +Standard ML, and Haskell support call-outs to system-level languages, +typically C. Other examples include the support for call-outs to C and +assembly language in C++, the EXTRINSIC directive in HPF, and the +<*EXTERNAL*> pragma in DEC SRC Modula-3. Mechanisms to call-out +to other languages are typically called foreign function +interfaces (FFIs). The purpose of an FFI is often to gain access to +functionality which is not (efficiently) expressible in the language +itself; other times the FFI is used to allow the program to interface to +existing libraries.
+ +FFIs are only rarely part of the language definition; the only examples +I can think of are the support for C and assembly in C++ and the +EXTRINSIC directive in HPF. More typically, each language +implementation has its own idiosyncratic and often ad-hoc mechanism for +supporting foreign data types, functions, and variables. The mechanisms +are not standardized probably because they depend to a large extent on +the calling conventions of the procedure being called, the operating +system on which the program is running, the architecture of the machine, +the data types of the language being called, the version of the +compilers for the host and foreign languages, and so on. (In the +following I will refer to a point in the space made from the product of +the preceding attributes as a target.) Since the system +dependencies are considerable, it is unlikely that a fully general and +portable FFI can be defined for a language, and in addition, an +interface that works with all targets is likely to be neither functional +nor convenient. The chances for any portable, standardized language to +adopt a non-trivial FFI therefore seem slight. This is not to say that +an adequate job can't be done in many cases--for example, Franz Allegro +Common Lisp sports a sophisticated FFI which supports C and Fortran +seemingly very well--only that no standard and general +solution is likely to emerge.
+ +Based on these observations, an approach to inter-language calling would +be to accept the fact that FFIs are implementation-dependent and instead +concentrate our effort on a higher level of abstraction: that of the +library interface. Even if the FFI is target-dependent, most of the +time the interface to a library is not (which is the beauty of an +interface in the first place). If, for each library, there existed a +reasonable definition of its interface, then a program could take that +definition and generate FFI code for the library for a given target. +This is the approach advocated by the creators of the ILU system (see +section 3).
+ +However, manufacturers of libraries are not distributing +reasonable definitions of the interfaces to their libraries. All you +usually get is a C or C++ header file. A header file is not a +reasonable definition of the interface because of the baggage it +carries: nested include files, preprocessor macros, conditional +compilation, syntactic peculiarities, implementation language target +dependencies, and so on. In the best of all worlds, the manufacturer +would distribute the interfaces in an interface definition language like +the Object Management Group's IDL or ILU's ISL, and maybe one day that +will be common. In the mean time, we must fend for ourselves.
+ +What we must do is to provide a translator which takes as its input not +a reasonable definition but instead a C or C++ header file or set of +header files, and produces as its output the FFI code for the library +for a given target. However, such a program is likely to be complicated +and there will be one version for each target. Maintaining all these +translators will be an unpleasant task. We could of course have one +translator, to IDL or ISL, and translators from the interface language +to the FFI, and as we will see, this is a variation on the mechanism +implemented by FFIGEN.
+ +An additional important problem is that there is not one but several +translations for every target. A given interface can be translated to +any of several FFIs depending on the desired policy for the +translation. For example, consider a function + +
+ char *fgets(char*, int, FILE*). ++ +What does char* translate to? Consider the FFI provided by Chez +Scheme version 5. It has a string type which in a parameter +position causes the address of the first character of the string +argument to be passed to the function, but which in the return position +causes the characters to be copied from the storage pointed to by the +return value (if not NULL) into a fresh Scheme string. So if we +translate char* as string, we end up with (since +FILE* is translated as an unsigned int) + +
+ (define fgets + (foreign-function "fgets" + (string integer-32 unsigned-32) + string)) ++ +which is expensive because the string is (needlessly) copied on return. +On the other hand, we can treat a char* as "just a pointer" and +translate as: + +
+ (define fgets + (foreign-function "fgets" + (unsigned-32 integer-32 unsigned-32) + unsigned-32)) ++ +but this does not let us access the characters in the buffer using +Scheme's string functions, since the buffer is not a string. In the +end, it appears that no fixed translation for char* is possible; +even if a fixed translation (and then: which one of them?) is adequate +in most situations, there will be special cases. (Arguably, it +would have been better for fgets() to return a truth value or the +number of characters read.) + +
The bottom line is, there is a lot of policy that goes into a +translation into a specific FFI. Hence we have a slogan (the core of +the Manifesto):
+ ++A good foreign function interface is 25% code and 75% policy. ++ +
It should be a goal, then, to separate the ardous task of parsing and +type-checking C headers and translating them into a rational +intermediate form, from the task of translating the intermediate form +into a FFI specification for a given target and translation policy.
+ +I have written a program, which I call ffigen, which takes +as its input a C header file and produces as its output a rational +translation of the interface defined by the header file. A rational +translation is one in which unnecessary or redundant syntax has been +removed, preprocessor macros have been expanded, and preprocessor +conditionals have been resolved so that definitions have been included +or excluded corrspondingly. The exact format of the intermediate code +is described in a companion document, the FFIGEN +User's Manual. ffigen functions as the front-end +of a system which translates C headers into foreign function +interfaces.
+ +Each target system will have one or more specific back-ends which +take the intermediate form and produce translations for particular +targets and translation policies. Substantial parts of the back-end +code is largely target-independent and can therefore be shared by +multiple back-ends.
+ +I have written one back-end to serve as a sample; it produces FFI code +for Chez Scheme version 5. It is documented in a companion document, +FFIGEN Back-end for Chez Scheme Version 5.
+ + +Kenneth B. Russell of MIT has implemented a system called Header2Scheme +which translates C++ to the FFI of the SCM Scheme system. FFIGEN and +Header2Scheme are fairly different at this point. My goal with FFIGEN +was to cover all of ANSI C including the preprocessor in a reasonable +way; this is doable because ANSI C is a small, fixed, and fairly simple +language. C++, on the other hand, is a very large, changing, and +complex language, and Header2Scheme therefore handles only part of it at +this time (as of version 1.2, it does not handle preprocessor macros, +typedefs, and enums). In addition, my emphasis was on not fixing policy +at all, which gives great freedom (and more work) to back-end writers, +whereas Russell has mostly fixed the policy. On the other hand, +Header2Scheme allows some policy decisions to be expressed in auxiliary +files given to the translator, and I have yet to experiment with these +mechanisms in FFIGEN. Header2Scheme is available from URL +
+http://www-white.media.mit.edu/~kbrussel/Header2Scheme ++ + +
A message (<1996Jan17.121933.25825@chemabs.uucp>) posted +to the Usenet group comp.lang.scheme (among others) alleged that +Apple has a translator for their Dylan implementation which will take a +C header file and generate Dylan FFI glue for it. I know nothing else +about this system (but would appreciate hearing about it from anyone who +knows).
+ +The ILU (Inter-Language Unification) system from Xerox PARC provides +cross-language calling functionality for modules which have interfaces +specified in ISL, the ILU interface definition language. ILU will take +the interfaces and produce stubs (glue, as it were) for the languages so +that they can call each other. The ISL file specifies the interface +somewhat abstractly in terms of data types which are meaningful in ISL +but which have various mappings in the target languages; again, one +mapping is assumed to fit all.
+ +FFIGEN is based on the lcc ANSI C compiler. See the FFIGEN User's Manual for full acknowlegements +and a copyright notice.
+ +This work has been supported by ARPA under U.S. Army grant +No. DABT63-94-C-0029, "Programming Environments, Compiler Technology +and Runtime Systems for Object Oriented Parallel Processing".
+ +lcc -ffigen); this can
+ be useful since the back end can generate C files which
+ #includes the source header file.
++ #define SoSphSetOverride(_this, state) \ + SoNodeSetOverride((SoNode *)_this, state) ++ and it would be nice to provide some support for such cases in the form + of already-tokenized output. +
+Press here to go to the FFIGEN home page. + +
FFIGEN is a program system which facilitates the writing of +translators from C header files to foreign function interfaces for +particular programming language implementations. This document +describes its structure and use. The discussion is aimed at translator +writers; everyone else should confine themselves to section 3. A +companion document, FFIGEN Manifesto and +Overview, motivates the work, and other companion documents describe +specific translator implementations. In particular, the document +FFIGEN Back-end for Chez Scheme Version 5 describes one +translator in detail.
+ +FFIGEN is based on the lcc C compiler, which is copyrighted +software. See Section 10 for a full copyright notice.
+ +To generate a translation of a header file you run the ffigen +command to generate an intermediate form of the C header files you want +to translate, and then run the back-end on the resulting files to +generate the foreign function interface for the library.
+ +Your task, should you choose to accept it, is to implement the +target-specific parts of the back-end for your particular target (which +is to say, combination of host language implementation, operating +system, architecture, foreign language implementation, and translation +policy). You should be able to use the FFIGEN front-end and the +target-independent parts of the back-end pretty much as they are.
+ +How to implement the target-specific parts of the back-end is +discussed in Section 6. Use of the front end is described in Section 2. +The intermediate format is described in Section 4, and the +target-independent parts of the back-end and their interface to the +target-dependent part are described in Section 5. Finally, Section 7 +covers some issues which need to be tackled in the future.
+ +The command ffigen is run on a set of header files with +preprocessor option and include file options. Arguments are processed +in order. For each header file (type .h) and all the files it +includes, a single preprocessor file (type .ffi) is +produced.
+ +The options are: +
The intermediate format consists of s-expressions following this grammar: + +
+ <file> -> <record> ... + <record> -> (function <filename> <name> <type> <attrs>) + | (var <filename> <name> <type> <attrs>) + | (type <filename> <name> <type>) + | (struct <filename> <name> ((<name> <type>) ...)) + | (union <filename> <name> ((<name> <type>) ...)) + | (enum <filename> <name> ((<name> <value>) ...)) + | (enum-ident <filename> <name> <value>) + | (macro <filename> <name+args> <body>) + <type> -> (<primitive> <attrs>) + | (struct-ref <tag>) + | (union-ref <tag>) + | (enum-ref <tag>) + | (function (<type> ...) <type>) + | (pointer <type>) + | (array <value> <type>) + <attrs> -> (<attr> ...) + <attr> -> static | extern | const | volatile + <primitive> -> char | signed-char | unsigned-char | short + | unsigned-short | int | unsigned | long + | unsigned-long | float | double | void + <value> -> <integer> + <filename> -> <string> + <name> -> <string> + <body> -> <string> + <name+args> -> <string> + <tag> -> <string> ++ +Notes relating to the grammar: + +
The target-independent back-end is a Scheme program called +process which reads the intermediate form into memory and +performs some initial processing. It exports some global variables and +a number of procedures which are used to access the structures in the +database of intermediate records, and imports two target-dependent +functions from the target-dependent back-end. This section describes +the interfaces.
+ +The global variables which hold the database are: + +
+ (define functions '()) ; list of function records + (define vars '()) ; list of var records + (define types '()) ; list of type records + (define structs '()) ; list of struct records + (define unions '()) ; list of union records + (define macros '()) ; list of macro records + (define enums '()) ; list of enum records + (define enum-idents '()) ; list of enum-ident records ++ +Each of these contains a list of all the records of the type indicated +by their names. Note that records may look different internally than +in the defined intermediate form, so accessor functions (see below) should +always be used. + +
In addition, there are two globals which are set but not used by +the target-independent back-end: + +
+ (define source-file #f) ; name of the input file itself + (define filenames '()) ; names of all files in the input ++ + +
The main entry point to the back end is the procedure process, +which takes a single file name as an argument. Process +initializes globals, reads the file, and processes the records. + +
+ (define (process filename) ...) ++ +
Record processing consists of some general analysis and target-specific +code generation. First, the target-specific procedure +select-functions is called; it must set or reset the +"referenced" bit in each record depending on whether the function is +interesting to the back-end or not. After computing reachability of +structured types and setting the referenced bits of those types which +are reachable, a translation is generated by a call to the back-end +function generate-translation, which takes no arguments. + +
+ (define (select-functions) ...) + (define (generate-translation) ...) ++ +
A number of data structure accessors and mutators are also available. +These are generic procedures which work on all of the record types. + +
+ (define (file r) ...) ; file name of record + (define (name r) ...) ; name in records which have one + (define (type r) ...) ; type in records which have one + (define (attrs r) ...) ; attrs in records which have one + (define (fields r) ...) ; fields in struct/union record + (define (value r) ...) ; value of enum-ident record + (define (tag r) ...) ; tag in struct/union/union/-ref record + + (define (referenced? r) ...) ; is record referenced? + (define (referenced! r) ...) ; set referenced bit + (define (unreferenced! r) ...) ; reset referenced bit ++ +Arguably the tag accessor should go away and name +should simply be used in its place. As it is, name is not +defined on struct-ref, union-ref, and +enum-ref records. + +
The procedure record-tag returns the tag of the record currently +being held. It can also be applied to types. + +
+ (define (record-tag r) ...) ; get record tag ++ +
All records can have back-end specific values attached to them; usually +these are cached names for operations on structured values, so for now +the procedures which manipulate the back-end specific data are called +cache-name to remember a value and cached-names to return +the list of remembered values: + +
+ (define (cache-name r v) ...) ; remember value in record + (define (cached-names r) ...) ; retrieve remembered values ++ +We should probably replace this with a more general property-list-like +mechanism. + +
In addition, two procedures extract parts of function types: + +
+ (define (arglist r) ...) ; function argument types + (define (rett r) ...) ; function return type ++ +
Some utilities to deal with file names are also provided: + +
+ (define (strip-extension fn) ...) + (define (strip-path fn) ...) + (define (get-path fn) ...) ++ +
A string macro expander makes it easier to generate C code, for the back +ends that need it. The macro expander is called instantiate and +is called with a string template and a vector of arguments (which are +also strings). The template contains patterns of the form @n +where n is a single digit; when such a pattern is seen it is +replaced with the corresponding value from the argument vector. + +
+ (define (instantiate template arguments) ...) ++ +
Two procedures, struct-names and union-names, take a +structure (or union) and returns a list of all the typedef names which +reference the structure directly. + +
+ (define (struct-names struct) ...) + (define (union-names union) ...) ++ +
An association function which searches one of the record lists for a +given record by the name field is also available: + +
+ (define (lookup key items) ...) ++ +
The procedure user-defined-tag? determines whether a tag was +defined by the user or generated by the system: + +
+ (define (user-defined-tag? x) ...) ++ +
The procedure warn takes some arbitrary arguments and generates +a warning message on standard output: + +
+ (define (warn msg . rest) ...) ++ +
Some standard predicates take a type and test its kind: +primitive-type? is true if the argument is of a primitive type as +outlined in the grammar above; basic-type? is true if the +argument is a primitive type or a pointer type; array-type? is +true if the argument is an array type, and finally, +structured-type? is true if the argument is a struct-ref +or union-ref type: + +
+ (define (primitive-type? t) ...) + (define (basic-type? t) ...) + (define (array-type? t) ...) + (define (structured-type? t) ...) ++ +
To write the target-dependent back-end, you must decide on the policy +for the translation and then implement the translation. The policy +covers such issues as: which constructs in C are or are not handled; the +translation for each handled construct; how non-handled constructs are +dealt with (ignored, detected with warnings, detected with errors); how +to deal with exceptional cases (consider the fgets example from +the Manifesto).
+ +For a concrete example, see the companion document FFIGEN Backend +for Chez Scheme Version 5, which addresses many of the choices to be +made and their possible solutions.
+ +A number of features will be supported in the future:
+ +A number of features will most likely be supported, but need +to be investigated:
+ +In addition, there are some issues to investigate in a larger +perspective:
+ +My goal is to support as many target languages as is reasonable, but I +can't write all the translators myself (I lack the time and, in many +cases, the knowledge). Targets that I will take care of include STk, +and, if no-one beats me to it, Scsh, both Scheme systems. Someone has +already volunteered to write the ILU back-end. Others are interested +in back-ends for Modula-3 and Mercury.
+ +Volunteers for any translator back-end are welcome to e-mail me and +volunteer their help. I will coach, coordinate, and help out as much as +possible.
+ +FFIGEN is based on the freely available lcc ANSI C compiler, +implemented by Christopher Fraser (of AT&T Bell Labs) and David Hanson +(of Princeton University).
+ +I would like to thank Fraser and Hanson for producing such an excellent +system; lcc has been a joy to work with, and their book, A +Retargetable C Compiler: Design and Implementation, made the +implementation of the FFIGEN front end in the matter of roughly a single +work day possible. Would it be that all software was this clean!
+ +The development of FFIGEN was supported by ARPA +under U.S. Army grant No. DABT63-94-C-0029, +``Programming Environments, Compiler Technology and Runtime Systems +for Object Oriented Parallel Processing''.
+ +++ +The authors of this software are Christopher W. Fraser and +David R. Hanson.
+ +Copyright (c) 1991,1992,1993,1994,1995 by AT&T, Christopher W. Fraser, +and David R. Hanson. All Rights Reserved.
+ +Permission to use, copy, modify, and distribute this software for any +purpose, subject to the provisions described below, without fee is +hereby granted, provided that this entire notice is included in all +copies of any software that is or includes a copy or modification of +this software and in all copies of the supporting documentation for +such software.
+ +THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR IMPLIED +WARRANTY. IN PARTICULAR, NEITHER THE AUTHORS NOR AT&T MAKE ANY +REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE MERCHANTABILITY +OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR PURPOSE.
+ +lcc is not public-domain software, shareware, and it is not protected +by a `copyleft' agreement, like the code from the Free Software +Foundation.
+ +lcc is available free for your personal research and instructional use +under the `fair use' provisions of the copyright law. You may, +however, redistribute the lcc in whole or in part provided you +acknowledge its source and include this COPYRIGHT file.
+ +You may not sell lcc or any product derived from it in which it is a +significant part of the value of the product. Using the lcc front end +to build a C syntax checker is an example of this kind of product.
+ +You may use parts of lcc in products as long as you charge for only +those components that are entirely your own and you acknowledge the use +of lcc clearly in all product documentation and distribution media. You +must state clearly that your product uses or is based on parts of lcc +and that lcc is available free of charge. You must also request that +bug reports on your product be reported to you. Using the lcc front +end to build a C compiler for the Motorola 88000 chip and charging for +and distributing only the 88000 code generator is an example of this +kind of product.
+ +Using parts of lcc in other products is more problematic. For example, +using parts of lcc in a C++ compiler could save substantial time and +effort and therefore contribute significantly to the profitability of +the product. This kind of use, or any use where others stand to make a +profit from what is primarily our work, is subject to negotiation.
+ +Chris Fraser / cwf@research.att.com
+
+David Hanson / drh@cs.princeton.edu
+Fri Jun 17 11:57:07 EDT 1994