Long obsolete.

This commit is contained in:
sperber 2003-04-22 12:45:30 +00:00
parent 27b2ae5a4f
commit 0bbb13664e
2 changed files with 0 additions and 567 deletions

View File

@ -1,85 +0,0 @@
<HTML>
<HEAD>
<TITLE>The Scheme Underground Network Package</TITLE>
</HEAD>
<BODY>
<H1>The Scheme Underground Network Package</H1>
I have written a set of libraries for doing Net hacking from Scheme/scsh.
It includes:
<DL>
<DT> An smtp client library.
<DD> Forge mail from the comfort of your own Scheme process.
<DT> rfc822 header library
<DD> Read email-style headers. Useful in several contexts (smtp, http, etc.)
<DT> Simple structured HTML output library
<DD> Balanced delimiters, etc.
<DT> The SU Web server
<DD> This is a complete implementation of an HTTP 1.0 server in Scheme.
The server contains other standalone packages that may separately be of
use:
<UL>
<LI> URI and URL parsers and unparsers.
<LI> A library to help writing CGI scripts in Scheme.
<LI> Server extensions for interfacing to CGI scripts.
<LI> Server extensions for uploading Scheme code.
</UL>
The server has three main design goals:
<DL>
<DT> Extensibility
<DD> The server is in fact nothing but extensions, using a mechanism
called "path handlers" to define URL-specific services. It has a toolkit
of services that can be used as-is, extended or built upon.
User extensions have exactly the same status as the base services.
<P>
The extension mechanism allows for easy implementation of new services
without the overhead of the CGI interface. Since the server is written
on top of the Scheme shell, the full set of Unix system calls and
program tools is available to the implementor.
<DT> Mobile code
<DD> The server allows Scheme code to be uploaded for direct execution
inside the server. The server has complete control over the code,
and can safely execute it in restricted environments that do not
provide access to potentially dangerous primitives (such as the
"delete file" procedure.)
<DT> Clarity
<DD> I wrote this server to help myself understand the Web. It is voluminously
commented, and I hope it will prove to be an aid in understanding the
low-level details of the Web protocols.
</DL>
<P>
The S.U. server has the ability to upload code from Web clients and
execute that code on behalf of the client in a protected environment.
<P>
Some <A HREF="su-httpd.html">simple documentation</A> on the server
is available.
</DL>
<H2>Obtaining the system</H2>
The network code is available by
<A HREF="ftp://ftp-swiss.ai.mit.edu/pub/scsh/contrib/net/net.tar.gz">ftp</A>.
To run the server, you need our 0.4 release of
<A HREF="http://www-swiss.ai.mit.edu/scsh/scsh.html">scsh</A>
which has just been released.
Beyond actually running the server,
the separate parser libraries and other utilites may be of use as separate
modules.
<ADDRESS><A HREF="http://www.ai.mit.edu/people/shivers/">Olin Shivers</A>
/ <A HREF="plan-file">shivers@ai.mit.edu</A></ADDRESS>
</BODY>
</HTML>

View File

@ -1,482 +0,0 @@
<!-- check for *..* emphasis, etc., i.e., e.g. -->
<HTML>
<HEAD>
<TITLE>The Scheme Underground Web system</TITLE>
</HEAD>
<BODY>
<H1>The Scheme Underground Web System</H1>
<ADDRESS><A HREF="http://www.ai.mit.edu/people/shivers/">Olin Shivers</A>
/ <A HREF="plan-file">shivers@ai.mit.edu</A>
</ADDRESS>
July 1995
<BLOCKQUOTE>
Note: Netscape typesets description lists in a manner that makes the
procedure descriptions below blur together, even in the absence of the
HTML COMPACT attribute. You may just wish to print out a simple
<A HREF="su-httpd.txt">ASCII version</A> of this note, instead.
</BLOCKQUOTE>
<!---------------------------------------------------------------------------->
<H2>Introduction</H2>
The
<A HREF="http://www.ai.mit.edu/projects/su/su.html">Scheme underground</A>
Web system is a package of
<A HREF="http://www-swiss.ai.mit.edu/scheme-home.html">Scheme</A>
code that provides
utilities for interacting with the
<A HREF="http://www.w3.org/">World-Wide Web</A>.
This includes:
<UL>
<LI> A Web server.
<LI> URI and URL parsers and un-parsers.
<LI> RFC822-style header parsers.
<LI> Code for performing structured html output
<LI> Code to assist in writing CGI Scheme programs
that can be used by any CGI-compliant HTTP server
(such as NCSA's httpd, or the S.U. Web server).
</UL>
<P>
The code can be obtained via
<A HREF="ftp://ftp-swiss.ai.mit.edu/pub/scsh/contrib/net/net.tar.gz">
anonymous ftp</A>
and is implemented in
<A HREF="http://www-swiss.ai.mit.edu/~jar/s48.html">Scheme 48</A>,
using the system calls and support procedures of
<A HREF="http://www-swiss.ai.mit.edu/scsh/scsh.html">scsh</A>,
the Scheme Shell.
The code was written to be clear and modifiable --
it is voluminously commented and all non-R4RS dependencies are
described at the beginning of each source file.
<P>
I do not have the time to write detailed documentation for these packages.
However, they are very thoroughly commented, and I strongly recommend
reading the source files; they were written to be read, and the source
code comments should provide a clear description of the system.
The remainder of this note gives an overview of the server's basic
architecture and interfaces.
<H2>The Scheme Underground Web Server</H2>
The server was designed with three principle goals in mind:
<DL>
<DT> Extensibility
<DD> The server is designed to make it easy to extend the basic
functionality. In fact, the server is nothing but extensions. There is
no distinction between the set of basic services provided by the server
implementation and user extensions -- they are both implemented in
Scheme, and have equal status. The design is "turtles all the way down."
<DT> Mobile code
<DD> Because the server is written in Scheme 48, it is simple to use the
Scheme 48 module system to upload programs to the server for safe
execution within a protected, server-chosen environment. The server
comes with a simple example upload service to demonstrate this
capability.
<DT> Clarity of implementation
<DD> Because the server is written in a high-level language, it should make
for a clearer exposition of the HTTP protocol and the associated URL
and URI notations than one written in a low-level language such as C.
This also should help to make the server easy to modify and adapt to
different uses.
</DL>
<!---------------------------------------------------------------------------->
<H3>Basic server structure</H3>
The Web server is started by calling the <CODE>httpd</CODE> procedure,
which takes one required and two optional arguments:
<PRE>
(httpd <VAR>path-handler</VAR> [<VAR>port</VAR> <VAR>working-directory</VAR>])
</PRE>
The server accepts connections from the given port, which defaults to 80.
The server runs with the working directory set to the given value,
which defaults to
<PRE>
/usr/local/etc/httpd
</PRE>
<P>
The server's basic loop is to wait on the port for a connection from an HTTP
client. When it receives a connection, it reads in and parses the request into
a special request data structure. Then the server forks a child process, who
binds the current I/O ports to the connection socket, and then hands off to
the top-level path handler (the first argument to <CODE>httpd</CODE>).
The path-handler procedure is responsible for actually serving the request --
it can be any arbitrary computation.
Its output goes directly back to the HTTP client that sent the request.
<P>
Before calling the path handler to service the request, the HTTP server
installs an error handler that fields any uncaught error, sends an
error reply to the client, and aborts the request transaction. Hence
any error caused by a path-handler will be handled in a reasonable and
robust fashion.
<P>
The basic server loop, and the associated request data structure are the fixed
architecture of the S.U. Web server; its flexibility lies in the notion of
path handlers.
<!---------------------------------------------------------------------------->
<H3>Path handlers</H3>
A path handler is a procedure taking two arguments:
<PRE>
(path-handler <VAR>path</VAR> <VAR>req</VAR>)
</PRE>
The <VAR>req</VAR> argument is a request record giving all the details of the
client's request; it has the following structure:
<PRE>
(define-record request
method ; A string such as "GET", "PUT", etc.
uri ; The escaped URI string as read from request line.
url ; An http URL record (see url.scm).
version ; A (major . minor) integer pair.
headers ; An rfc822 header alist (see rfc822.scm).
socket) ; The socket connected to the client.
</PRE>
The <VAR>path</VAR> argument is the URL's path,
parsed and split at slashes into a string list.
For example, if the Web client dereferences URL
<PRE>
http://clark.lcs.mit.edu:8001/h/shivers/code/web.tar.gz
</PRE>
then the server would pass the following path to the top-level handler:
<PRE>
("h" "shivers" "code" "web.tar.gz")
</PRE>
<P>
The path argument's pre-parsed representation as a string list makes it easy
for the path handler to implement recursive operations dispatch on URL paths.
<P>
Path handlers can do anything they like to respond to HTTP requests; they have
the full range of Scheme to implement the desired functionality. When
handling HTTP requests that have an associated entity body (such as POST), the
body should be read from the current input port. Path handlers should in all
cases write their reply to the current output port. Path handlers should
<EM>not</EM> perform I/O on the request record's socket.
Path handlers are frequently called recursively, and doing I/O directly to the
socket might bypass a filtering or other processing step interposed on the
current I/O ports by some superior path handler.
<!---------------------------------------------------------------------------->
<H3>Basic path handlers</H3>
Although the user can write any path-handler he likes, the S.U. server comes
with a useful toolbox of basic path handlers that can be used and built upon:
<DL>
<DT>
<CODE>(alist-path-dispatcher <VAR>ph-alist</VAR> <VAR>default-ph</VAR>) -> <VAR>path-handler</VAR>
</CODE>
<DD>
This procedure takes a string->path-handler alist, and a default
path handler, and returns a handler that dispatches on its path argument.
When the new path handler is applied to a path
<CODE>("foo" "bar" "baz")</CODE>,
it uses the first element of the path -- <CODE>"foo"</CODE> -- to
index into the alist.
If it finds an associated path handler in the alist, it
hands the request off to that handler, passing it the tail of the
path, <CODE>("bar" "baz")</CODE>.
On the other hand, if the path is empty, or the alist search does
not yield a hit, we hand off to the default path handler,
passing it the entire original path, <CODE>("foo" "bar" "baz")</CODE>.
<P>
This procedure is how you say: "If the first element of the URL's path
is `foo', do X; if it's `bar', do Y; otherwise, do Z." If one takes
an object-oriented view of the process, an alist path-handler does
method lookup on the requested operation, dispatching off to the
appropriate method defined for the URL.
<P>
The slash-delimited URI path structure implies an associated
tree of names. The path-handler system and the alist dispatcher
allow you to procedurally define the server's response to any arbitrary
subtree of the path space.
<P>
Example: <br>
A typical top-level path handler is
<PRE>
(define ph
(alist-path-dispatcher
`(("h" . ,(home-dir-handler "public_html"))
("cgi-bin" . ,(cgi-handler "/usr/local/etc/httpd/cgi-bin"))
("seval" . ,seval-handler))
(rooted-file-handler "/usr/local/etc/httpd/htdocs")))
</PRE>
This means:
<UL>
<LI> If the path looks like <CODE>("h" "shivers" "code" "web.tar.gz")</CODE>,
pass the path <CODE>("shivers" "code" "web.tar.gz")</CODE> to a
home-directory path handler.
<LI> If the path looks like <CODE>("cgi-bin" "calendar")</CODE>,
pass <CODE>("calendar")</CODE> off to the CGI path handler.
<LI> If the path looks like <CODE>("seval" ...)</CODE>,
the tail of the path is passed off to the code-uploading seval
path handler.
<LI> Otherwise, the whole path is passed to a rooted file handler, who
will convert it into a filename, rooted at
<CODE>/usr/local/etc/httpd/htdocs</CODE>, and serve that file.
</UL>
<DT> <CODE>(home-dir-handler <VAR>subdir</VAR>) ->
<VAR>path-handler</CODE></VAR>
<DD>
This procedure builds a path handler that does basic file serving
out of home directories. If the resulting path handler is passed
a path of <CODE>(<VAR>user</VAR> . <VAR>file-path</VAR>)</CODE>,
then it serves the file
<PRE>
<VAR>user's-home-directory</VAR>/<VAR>subdir</VAR>/<VAR>file-path</VAR>
</PRE>
The path handler only handles GET requests; the filename is not
allowed to contain <CODE>..</CODE> elements.
<DT>
<CODE>(tilde-home-dir-handler <VAR>subdir</VAR> <VAR>default-path-handler</VAR>)
-> <VAR>path-handler</VAR>
</CODE>
<DD>
This path handler examines the car of the path. If it is a string
beginning with a tilde, <em>e.g.</em>, "<CODE>~ziggy</CODE>",
then the string is taken
to mean a home directory, and the request is served similarly to a
<CODE>home-dir-handler</CODE> path handler.
Otherwise, the request is passed off
in its entirety to the default path handler.
<P>
This procedure is useful for implementing servers that provide the
semantics of the NCSA httpd server.
<DT>
<CODE>(cgi-handler <VAR>cgi-directory</VAR>) -> <VAR>path-handler</VAR>
</CODE>
<DD>
This procedure returns a path-handler that passes the request off to some
program using the CGI interface. The script name is taken from the
car of the path; it is checked for occurrences of <CODE>..</CODE>'s.
If the path is
<PRE>
("my-prog" "foo" "bar")
</PRE>
then the program executed is
<PRE>
<VAR>cgi-directory</VAR>/my-prog
</PRE>
<P>
When the CGI path handler builds the process environment for the
CGI script, several elements
(<em>e.g.</em>, <CODE>$PATH</CODE> and <CODE>$SERVER_SOFTWARE</CODE>)
are request-invariant, and can be computed at server start-up time.
This can be done by calling
<PRE>
(initialise-request-invariant-cgi-env)
</PRE>
when the server starts up. This is <EM>not</EM> necessary,
but will make CGI requests a little faster.
<DT>
<CODE>(rooted-file-handler <VAR>root-dir</VAR>) -> <VAR>path-handler</VAR>
</CODE>
<DD>
Returns a path handler that serves files from a particular root
in the file system. Only the GET operation is provided. The path
argument passed to the handler is converted into a filename,
and appended to <VAR>root-dir</VAR>.
The file name is checked for <CODE>..</CODE> components,
and the transaction is aborted if it does. Otherwise, the file is
served to the client.
<DT>
<CODE>(null-path-handler <VAR>path</VAR> <VAR>req</VAR>)</CODE>
<DD>
This path handler is useful as a default handler. It handles no requests,
always returning a "404 Not found" reply to the client.
</DL>
<!---------------------------------------------------------------------------->
<H3>HTTP errors</H3>
Authors of path-handlers need to be able to handle errors in a reasonably
simple fashion. The S.U. Web server provides a set of error conditions that
correspond to the error replies in the HTTP protocol. These errors can be
raised with the <CODE>http-error</CODE> procedure.
When the server runs a path handler,
it runs it in the context of an error handler that catches these errors,
sends an error reply to the client, and closes the transaction.
<DL>
<DT>
<CODE>(http-error <VAR>reply-code</VAR> <VAR>req</VAR> [<VAR>extra</VAR> ...])</CODE>
<DD>
This raises an http error condition. The reply code is one of the
numeric HTTP error reply codes, which are bound to the variables
<CODE>http-reply/ok</CODE>, <CODE>http-reply/not-found</CODE>,
<CODE>http-reply/bad-request</CODE>, and so
forth. The <VAR>req</VAR> argument is the request record that caused
the error.
Any following <VAR>extra</VAR> args are passed along for
informational purposes.
Different HTTP errors take different types of extra arguments.
For example, the "301 moved permanently" and "302 moved temporarily"
replies use the first two <VAR>extra</VAR> values as the
<CODE>URI:</CODE> and <CODE>Location:</CODE>
fields in the reply header, respectively. See the clauses of the
<CODE>send-http-error-reply</CODE> procedure for details.
<DT>
<CODE>(send-http-error-reply <VAR>reply-code</VAR> <VAR>request</VAR>
[<VAR>extra</VAR> ...])
</CODE>
<DD>
This procedure writes an error reply out to the current output
port. If an error occurs during this process, it is caught, and
the procedure silently returns. The http server's standard error
handler passes all http errors raised during path-handler execution
to this procedure to generate the error reply before aborting the
request transaction.
</DL>
<!---------------------------------------------------------------------------->
<H3>Simple directory generation</H3>
Most path-handlers that serve files to clients eventually call an internal
procedure named <CODE>file-serve</CODE>,
which implements a simple directory-generation service using the
following rules:
<UL>
<LI> If the filename has the <EM>form</EM> of a directory
(<EM>i.e.</EM>, it ends with a slash),
then <CODE>file-serve</CODE> actually looks for a
file named "<CODE>index.html</CODE>" in that directory.
<LI> If the filename names a directory, but is not in directory form
(<EM>i.e.</EM>, it doesn't end in a slash,
as in "<CODE>/usr/include</CODE>" or "<CODE>/usr/raj</CODE>"),
then <CODE>file-serve</CODE> sends back a "301 moved permanently"
message,
redirecting the client to a slash-terminated version of the original
URL. For example, the URL
<PRE>
http://clark.lcs.mit.edu/~shivers
</PRE>
would be redirected to
<PRE>
http://clark.lcs.mit.edu/~shivers/
</PRE>
<LI> If the filename names a regular file, it is served to the client.
</UL>
<!---------------------------------------------------------------------------->
<H3>Support procs</H3>
The source files contain a host of support procedures which will be of utility
to anyone writing a custom path-handler. Read the files first.
<!---------------------------------------------------------------------------->
<H3>Losing</H3>
Be aware of two Unix problems, which may require workarounds:
<OL>
<LI>
NeXTSTEP's Posix implementation of the <CODE>getpwnam()</CODE> routine
will silently tell you that every user has uid 0. This means
that if your server, running as root, does a
<PRE>
(set-uid (user->uid "nobody"))
</PRE>
it will essentially do a
<PRE>
(set-uid 0)
</PRE>
and you will thus still be running as root.
<P>
The fix is to manually find out who user nobody is (he's -2 on my
system), and to hard-wire this into the server:
<PRE>
(set-uid -2)
</PRE>
This problem is NeXTSTEP specific. If you are using not using NeXTSTEP,
no problem.
<LI>
On NeXTSTEP, the ip-address->host-name translation routine
(in C, <CODE>gethostbyaddr()</CODE>; in scsh,
<CODE>(host-info addr)</CODE>) does not
use the DNS system; it goes through NeXT's propietary Netinfo
system, and may not return a fully-qualified domain name. For
example, on my system, I get "amelia-earhart", when I want
"amelia-earhart.lcs.mit.edu". Since the server uses this name
to construct redirection URL's to be sent back to the Web client,
they need to be FQDN's.
<P>
This problem may occur on other OS's;
I cannot determine if <CODE>gethostbyaddr()</CODE>
is required to return a FQDN or not. (I would appreciate hearing the
answer if you know; my local Internet guru's couldn't tell me.)
<P>
If your system doesn't give you a complete Internet address when
you say
<PRE>
(host-info:name (host-info (system-name)))
</PRE>
then you have this problem.
<P>
The server has a workaround. There is a procedure exported from
the httpd-core package:
<PRE>
(set-my-fqdn name)
</PRE>
Call this to crow-bar the server's idea of its own Internet host name
before running the server, and all will be well.
</OL>
</BODY>
</HTML>