scsh-install-lib/doc/proposal.txt

In Emacs, read this file in -*- Outline -*- mode.

* Introduction

The aim of the following proposal is to define a standard for the
packaging, distribution, installation, use and removal of libraries
for scsh. Such packaged libraries are called "scsh packages" or simply
"packages" below.

This proposal attempts to cover both libraries containing only Scheme
code and libraries containing additional C code. It does not try to
cover applications written in scsh, which are currently considered to
be outside of its scope.

** Package identification and naming

Packages are identified by a globally-unique name. This name should
start with an ASCII letter (a-z or A-Z) and should consist only of
ASCII letters, digits or underscore characters (_). Package names are
case-sensitive, but there should not be two packages with names which
differ only by their capitalisation.

Rationale:

  This restriction on package names ensures that they can be used to
  name directories on current operating systems.

Several versions of a given package can exist. A version is identified
by a sequence of non-negative integers. Versions are ordered
lexicographically.

A version has a printed representation which is obtained by separating
(the printed representation of) its components by dots. For example,
the printed representation of a version composed of the integer 1
followed by the integer 2 is the string "1.2". Below, versions are
usually represented using their printed representation for simplicity,
but it is important to keep in mind that versions are sequences of
integers, not strings.

A specific version of a package is therefore identified by a name and
a version. The full name of a version of a package is obtained by
concatenating:

  - the name of the package,

  - a hyphen (-),

  - the printed representation of the version.

In what follows, the term "package" is often used to designate a
specific version of a package, but this should be clear from the
context.

* Distribution of packages

Packages are distributed in TAR archives, which can optionally be
compressed by GZIP or BZIP2.

The name of the archive is composed by appending:

  - the full name of the package,

  - the string ".tar" indicating that it's a TAR archive,

  - either the string ".gz" if the archive is compressed using GZIP,
    or the string ".bz2" if the archive is compressed using BZIP2, or
    nothing if the archive is not compressed.

** Archive contents

The archive is organised so that it contains one top-level directory
whose name is the full name of the package. This directory is called
the package unpacking directory. All the files belonging to the
package are stored below it.

The unpacking directory contains at least the following files:

  install-pkg
    a script performing the installation of the package,

  README
    a textual file containing a short description of the package,

  COPYING
    a textual file containing the license of the package.

* Downloading and installation of packages

A package can be installed on a target machine by downloading its
archive, expanding it and finally running the installation script
located in the unpacking directory.

** Layouts

The installation script installs files according to some given layout.
A layout maps abstract locations to concrete directories on the target
machine. For example, a layout could map the abstract location "doc"
(where documentation is stored) to the directory
"/usr/local/share/doc/my_package".

Currently, the following abstract locations are defined:

  base

    The "base" location of a package, where the package loading
    script "load.scm" resides.

  active

    Location containing a symbolic link, with the same name as the
    package (without the version), pointing to the "base" location of
    the package. This link is used to designate the "active" version
    of a package, that is the one to load when a package is requested
    by giving only its name, without explicit version.

  scheme

    Location containing all Scheme code. If the package comes with
    some examples showing its usage, they are put in a sub-directory
    called "examples" of this location.

  lib

    Location containing platform-dependent files, like shared
    libraries. This location contains one sub-directory per platform
    for which packages have been installed, and nothing else.

  doc

    Location containing all the package documentation. This location
    contains one or more sub-directories which store the documentation
    in various formats. The contents of these sub-directories is
    standardised as follows, to make it easy for users to find the
    document they need:

      html/

        Directory containing the HTML documentation of the package, if
        any; this directory should at least contain one file called
        "index.html" serving as an entry point to the documentation.

      pdf/

        Directory containing the PDF documentation of the package, if
        any; this directory should contain at least one file called
        "<package>.pdf" where "<package>" is the name of the package.

      ps/

        Directory containing the PostScript documentation of the
        package, if any; this directory should contain at least one
        file called "<package>.ps" where "<package>" is the name of
        the package.

  misc-shared

    Location containing miscellaneous data which does not belong to
    any directory above, and which is platform-independant.

The directories to which a layout maps these abstract locations are
not absolute directories, but rather relative ones. Therefore, a
layout alone is not enough to know where files will end up on the
target machine: a prefix is also required. This prefix is specified on
the command-line during installation, using the "--prefix" option, as
explained below.

Example :

Let's imagine that a user is installing version 1.2 of a package
called "foo". This package contains a file called "COPYING" which has
to be installed in sub-directory "license" of the "doc" location. If
the user chooses to use the default layout, which maps "doc" to
directory "<package_full_name>/doc" (see below), and specifies
"/usr/local/etc/scsh/modules" as a prefix, then the "COPYING" file
will end up in:

  /usr/local/etc/scsh/modules/foo-1.2/doc/license/COPYING
  \_________________________/ \_________/ \_____________/
               1                   2             3

Part 1 is the prefix, part 2 is the layout's mapping for the "doc"
location, and part 3 is the file name relative to the location.

*** Predefined layouts

Every installation script comes with a set of predefined layouts
which serve different aims. They are described below.

**** The "scsh" layout

The "scsh" layout is the default layout. It maps all locations to
sub-directories of a single directory, called the package installation
directory, which contains all the files of the package being installed
and nothing else. Its name is simply the full name of the package in
question, and it resides in the "prefix" directory.

The "scsh" layout maps locations as follows:

  base        -> <prefix>/<package_full_name>
  active      -> <prefix>
  scheme      -> <prefix>/<package_full_name>/scheme
  lib         -> <prefix>/<package_full_name>/lib/<platform>
  doc         -> <prefix>/<package_full_name>/doc
  misc-shared -> <prefix>/<package_full_name>

This layout is well suited for installations performed without the
assistance of an additional package manager, because it makes many
common operations easy. For example, finding to which package a file
belongs is trivial, as is the removal of an installed package.

**** The "fhs" layout

The "fhs" layout maps locations according to the File Hierarchy
Standard (FHS, see ...), as follows:

  base        -> <prefix>/share/scsh/modules/<package_full_name>
  active      -> <prefix>/share/scsh/modules
  scheme      -> <prefix>/share/scsh/modules/<package_full_name>/scheme
  lib         -> <prefix>/lib/scsh/modules/<package_full_name>
  doc         -> <prefix>/share/doc/<package_full_name>
  misc-shared -> <prefix>/share/scsh/modules/<package_full_name>

The main advantage of this layout is that it adheres to the FHS
standard, and is therefore compatible with several packaging policies,
like Debian's, Fink's and others. Its main drawback is that files
belonging to a given package are scattered, and therefore hard to find
when removing or upgrading a package. Its use should therefore be
considered only if third-party tools are available to track files
belonging to a package.

** File permissions

TODO

** Installation procedure

Packages are installed using the "install-pkg" script located in the
package archive. This script must be given the name of the prefix
using the "--prefix" option. It also accepts the following options:

  --layout <name>

    Specifies the layout to use.

  --dry-run

    Print what actions would be performed to install the package, but
    do not perform them.

  --inactive

    Do not activate package after installing it.

** Creating images

TODO (my current idea is to add support to install-lib to easily
      create an image containing the package being installed, and
      maybe some structures opened. Then, at install time, users could
      say that they want an image to be created, and the install
      script would do that).

* Using packages

To use a package, its "loading script" must be loaded in Scheme 48's
exec package. The loading script for a package is a file written in
the Scheme 48 exec language, whose name is "load.scm" and which
resides in the "base" location.

To load this file, one typically uses scsh's "-lel" option along with
a properly defined SCSH_LIB_DIRS environment variable.

Scsh has a list of directories, called the library directories, in
which it looks for files to load when the options -ll or -lel are
used. This list can be given a default value during scsh's
configuration, and this value can be overridden by setting the
environment variable SCSH_LIB_DIRS before running scsh.

In order for scsh to find the package loading scripts, one must make
sure that scsh's library search path contains the names of all
"active" locations which containing packages.

The names of these directories should not end with a slash (/), as
this forces scsh to search them recursively. This could *drastically*
slow down scsh when looking for packages.

Example:

Let's imagine a machine on which the system administrator installs
scsh packages according to the "fhs" layout in prefix directory
"/usr/local". The "active" location for these packages corresponds to
the directory "/usr/local/share/scsh/modules", according to the layout
specification above.

Let's also imagine that there is a user called "john" on this machine,
who installs additional scsh packages for himself in his home
directory, using "/home/john/scsh-packages" as a prefix. To ease their
management, he uses the "scsh" layout. The "active" location for these
packages corresponds to the directory "/home/john/scsh-packages",
according to the layout specification above.

In order to be able to use scsh packages installed both by the
administrator and by himself, user "john" needs to put both active
directories in his SCSH_LIB_DIRS environment variable. The value of
this variable will therefore be:

  "/usr/local/share/scsh/modules" "/home/john/scsh-packages"

Now, in order to use packages "foo" and "bar" in one of his script,
user "john" just needs to load their loading script using the -lel
option when invoking scsh, as follows:

  -lel foo/load.scm -lel bar/load.scm

* Writing packages

Once the Scheme and/or C code for a package has been written, the last
step in turning it into a standard package as defined by this proposal
is to write the installation script.

This script could be written fully by the package author, but in order
to simplify this task a small scsh installation framework is provided.
This framework is composed of several files which are meant to be
included in the package archive. These files are:

  install-pkg
    a trivial sh script which launches scsh on the main function of
    the installation library, passing it all the arguments given by
    the user,

  install-lib.scm
    the code for the installation library, whose public interface is
    documented below,

  install-lib-module.scm
    Scheme 48 interface and structure definitions for the installation
    library,

  configure.scm
    a Scheme library providing a function to query the name of the
    platform, needed by some layouts.

As explained above, when the install-pkg script is invoked, it launches
scsh on the main function of the installation library, which does the
following:

  - parse the command line arguments (e.g the --prefix option),

  - load the package definition file, a (Scheme) file called
    "pkg-def.scm", which is supplied by the package author and which
    contains the installation procedure for the package,

  - install the package which was defined in the previous step.

It is actually possible to define several packages in "pkg-def.scm",
and all will be installed. It should not be often useful, though.

The main job of the package author is therefore to write the package
definition file, "pkg-def.scm".

This file is mostly composed of a package definition statement, which
specifies the name, version and installation code for the package. The
package definition statement is expressed using the following syntax
exported from the installation library:

(define-package <name> <version> <extensions> <body> ...)    (syntax)

  Define a package to be installed. NAME is the package name (a
  string), VERSION its version (a list of integers), EXTENSIONS is an
  association list of extensions, and BODY is the list of statements
  to be evaluated in order to install the package.

The installation statements typically use functions of the
installation library in order to install files in their target
location. The following functions are currently exported:

(install-file <file> <location> [<target-dir>] [<perms>])

  Install the given FILE in the sub-directory TARGET-DIR (which must
  be a relative directory) of the given LOCATION.

  If the directory in which the file is about to be installed does not
  exist, it is created along with all its parents, as needed. If FILE
  is a string, then the installed file will have the same name as the
  original one. If FILE is a pair, then its first element specifies
  the name of the source file, and its second element the name it will
  have once installed. The second element must be a simple file name,
  without any directory part.

  The copied file and all directories created by this command have
  their permissions set to PERMS, an integer which defaults to #o755
  (i.e. read, write and execute for the owner, read and execute for
  the rest).

(install-files <file-list> <location> [<target-dir>] [<perms>])

  Like install-file but for several files, which are specified as a
  list. Each element in the list can be either a simple string or a
  pair, as explained above.

(install-directory <dir> <location> [<target-dir>] [<perms>])

  Install the given DIRectory and all its contents, including
  sub-directories, in sub-directory TARGET-DIR of LOCATION. This is
  similar to what INSTALL-FILE does, but for complete hierarchies.

  Notice that DIR will be installed as a sub-directory of TARGET-DIR.

(install-directories <dir-list> <location> [<target-dir>] [<perms>])

  Install several directories in one go.

(install-directory-contents <dir> <location> [<target-dir>] [<perms>])

  Install the *contents* of the given DIRectory in sub-directory
  TARGET of LOCATION.

An additional function exists to query the mapping of a location:

(get-directory <location> <install?>)

  Get the absolute name of the directory to which the current layout
  maps the abstract LOCATION. If INSTALL? is true, the directory is
  the one valid during installation; If it is false, the directory is
  the one valid after installation, that is when the package is later
  used.

  The distinction between installation-time and usage-time directories
  is necessary to support staged installation, as performed by package
  managers like Debian's APT.

Example:

A typical package definition file for a simple package called
"my_package" whose version is 1.2 could look like this:

(define-package "my_package" (1 2)
  (install-file "load.scm" 'base)
  (install-directory-contents "scheme" 'scheme)
  (install-file ("LICENSE" . "COPYING") 'doc)
  (install-directory-contents "doc" 'doc))

With such a definition, invoking the installation script with
"/usr/local/" as prefix and "fhs" as layout would have the following
effects:

1. The base directory

     /usr/local/share/scsh/modules/my_package-1.2

   would be created and file "load.scm" would be copied to it.

3. All the contents of the directory called "scheme" would be copied
   to directory

     /usr/local/share/scsh/modules/my_package-1.2/scheme

   which would be created before, if needed.

4. File "LICENSE" would be copied to directory

     /usr/local/share/doc/my_package-1.2/

   with name "COPYING".

5. All the contents of the directory called "doc" would be copied to
   directory

     /usr/local/share/doc/my_package-1.2/

6. The package would be activated by creating a symbolic link with
   name

     /usr/local/share/scsh/modules/my_package

   pointing to

     ./my_package-1.2

** Packages containing C code (for shared libraries)

Packages containing C code are more challenging to write, since all
the problems related to C's portability and incompatibilities between
the APIs of the various platforms have to be accounted for.
Fortunately, the GNU Autoconf system simplifies the management of
these problems, and authors of scsh packages containing C code are
strongly encouraged to use it.

Integrating Autoconf into the installation procedure should not be a
major problem thanks to scsh's ability to run separate programs.

* Packaging packages

Most important Unix systems today have one (or several) package
management systems which ease the installation of packages on a
system. In order to avoid confusion between these packages and the
scsh packages discussed above, they will be called "system packages"
in what follows.

It makes perfect sense to provide system packages for scsh packages.
System packages should as much as possible try to use the standard
installation script described above to install scsh packages. This
script currently provides some support for staged installations,
which are required by several packaging systems.

This support is provided through an additional option,
--install-prefix, which specifies the prefix in which the files have
to be copied by the installation script. The files will then have to
be moved from this location to their final prefix directory, i.e. the
one given through the --prefix option, by the system packaging tools.

(The --install-prefix option plays the same role as the DEST_DIR
variable which is typically given to "make install", for makefiles
which support staging directories).

* Glossary

TODO define the following terms

Version

Target machine

Package

(Package) unpacking directory

Layout

(Abstract) location

Package loading script

* Version

$Id: proposal.txt,v 1.3 2003/12/21 20:43:39 michel-schinz Exp $