pymdl

The pymdl book

28. The package system

The portion of the MDL environment which provides a uniform facility for lexical blocking is known as the Package System. In one sense it is the most basic part of the environment, since it enables many programmers to use each other's code without identifier conflicts.

In addition, the Package System is interfaced to a library facility (see section 4) by which MDL code may be stored and later loaded as needed.

The Package System is so basic to use of the MDL environment that (with a few exceptions) every subsystem or family of MDL functions described in this document is a 'package'.

pymdl's own. The package system here is not an approximation of the manual's: it is the manual's, MIT's LIBMUD;NPCK (revision 26), loaded from its own source at boot and given the few things it expects to find around it (the PACKAGES and COLLECTIONS oblists, the L- control globals of chapter 30, a NULL-OBLIST). What pymdl adds is what the source calls out to and the era got from its disk -- how a USE finds a package that is not yet loaded (28.4) -- and the OBLIST machinery underneath, which was measured against MDL 55 case by case (chapter 18).

2.1. The Theory of Lexical Blocking in MDL

Lexical blocking is implemented in MDL by means of OBLISTs and LISTs of OBLISTs. Changes of lexical context are performed using the SUBRs BLOCK and ENDBLOCK. The Package System provides a high-level interface to these low-level constructs.

The primary goal of a lexical blocking scheme is the prevention of identifier conflicts. Specifically, when your program references the variable X, it should be your X and not that of some other program. At the same time, it should not be necessary for a programmer to search every program previously written to verify that an identifier he wishes to use is not already 'taken'.

It should be clear that the simplest solution, a single OBLIST, will not satisfy either of these goals. With only one OBLIST there would necessarily be identifier conflicts, necessitating exhaustive searching for unique identifiers.

Obviously, programmers could put their program's identifiers on an OBLIST unique to that program. Unfortunately, such a solution addresses only half the problem. What happens when some other programmer wishes to use some of this code? He could insert the unique OBLIST for that program into the OBLIST path for his program; but the moment that is done he gets all the identifiers for that program, including local variables, internal data structures, and so on.

Consequently, we move to a situation where each program uses two OBLISTs: one for the identifiers that are local to the program, and one for the identifiers that are to be used by other programs. In the Package System, these are known as the 'internal' OBLIST and the 'entry' OBLIST.

Most of the identifiers in a program are local to it, and want to be placed on the internal OBLIST.

Therefore, in terms of an argument to the BLOCK SUBR, when a program is being loaded into MDL, the OBLIST path wants to be:

( internal-OBLIST
  entry-OBLIST
  <ROOT> )

With this OBLIST path, most ATOMs (identifiers) will be on the internal OBLIST (as READ puts unknown identifiers on <1 .OBLIST>), but the ATOMs for the entries and the ATOMs for the usual SUBRs will be available.

The only issue yet to be addressed is that of using an entry of a different program in your program. This is accomplished by adding the entry OBLISTs of any such programs to the path after ROOT:

( internal-OBLIST
  entry-OBLIST
  <ROOT>
  other-program-entry-OBLIST
  yet-another-program-entry-OBLIST )

As only the entry OBLIST, and not the internal OBLIST, of the program being used is added to the path, the chance of identifier conflict is lessened.

All that remains is to introduce the functions by which these various operations are performed.

2.2. Package System Overview

The functions which make up the Package System are:

  • PACKAGE. This indicates the start of a package of functions.
  • ENDPACKAGE. This indicates the end of the package of functions.
  • ENTRY. This indicates an ATOM which is to be made available outside the definition of this package of functions. All other ATOMs will not be directly available outside the package.
  • USE. This indicates a reference by name to another package of functions.
  • USE-DATUM. This indicates a reference by name to a data set.
  • DROP and L-UNUSE. These undo the effects of USE and USE-DATUM.

These functions are themselves part of a package named "PKG", which is preloaded into MDL.

2.2.1. Sample PACKAGE

A sample MDL PACKAGE is given with comments in order to demonstrate the usage of these functions.

<PACKAGE "HOUR-STRING">
"PACKAGE begins the package called HOUR-STRING."
<ENTRY TIME-STRING>
;"The atom TIME-STRING is an entry to this package;
  it may be referenced by other packages by
  USEing HOUR-STRING."
<USE "DATIME">
"Indicate that the package DATIME is
  used within the current package."
<DEFINE TIME-STRING ()
  <STRING <UNPARSE <HOURS>> " o'clock">>
;"Define this little function which returns a string
  telling the last hour in a strange format."
<DEFINE HOURS () <1 <RTIME>>>
;"Define an internal function which is available
  only within the HOUR-STRING package, since its
  name is not in any ENTRY statement.
  Note that this function refers to RTIME,
  which is an ENTRY in the DATIME package."
<ENDPACKAGE>
;"The end of this little demonstration package."

The same package, typed at pymdl's listener with its entry renamed (the reason is in the box below), and what each step answers:

<PACKAGE "HOUR-STRING">                                        ⇒ HOUR-STRING!-PACKAGE
<ENTRY HOUR-TEXT>                                              ⇒ HOUR-TEXT!-HOUR-STRING
<USE "DATIME">
<DEFINE HOUR-TEXT () <STRING <UNPARSE <HOURS>> " o'clock">>    ⇒ HOUR-TEXT!-HOUR-STRING
<DEFINE HOURS () <1 <RTIME>>>                                  ⇒ HOURS!-IHOUR-STRING
<ENDPACKAGE>                                                   ⇒ T
<LENGTH .OBLIST>                                               ⇒ 4
<USE "HOUR-STRING">
<TYPE <HOUR-TEXT>>                                             ⇒ STRING
<GET <OBLIST? HOUR-TEXT> OBLIST>                               ⇒ HOUR-STRING!-PACKAGE
<GET <OBLIST? HOURS!-IHOUR-STRING!-HOUR-STRING!-PACKAGE> OBLIST> ⇒ IHOUR-STRING!-HOUR-STRING
<LENGTH <GET HOUR-STRING!-PACKAGE OBLIST>>                     ⇒ 19
<LENGTH <GET IHOUR-STRING!-HOUR-STRING!-PACKAGE OBLIST>>       ⇒ 23
<L-UNUSE "HOUR-STRING">                                        ⇒ "PACKAGE REMOVED"

Inside the package, DEFINE answers the atom with the trailer the path gives it: the entry on the HOUR-STRING oblist, the internal function on IHOUR-STRING. After ENDPACKAGE the path is back to the four oblists a session starts with (chapter 27), and only a USE puts HOUR-STRING's entries within reach; HOURS is reachable only by its full trailer. The two oblists are 19 and 23 buckets, the defaults of 2.3.

Found along the way. The manual's sample cannot be typed in as it stands, here or on an era system with the library loaded, because its entry name is taken. TIME-STRING is an entry of TIMFCN, the era's time-formatting package (vendored from the 1983 tape, chapter 30), and a bare <TIME-STRING> at the listener does not reach HOUR-STRING's at all: the dynamic loader of chapter 30 finds TIMFCN's entry in the library, USEs TIMFCN, and the call lands there -- with TOO-FEW-ARGUMENTS-SUPPLIED, since TIMFCN's takes a time. That is exactly the conflict of 2.3.7, and 2.3.7's remedy, the full trailer, resolves it. The example above renames the entry instead.

2.3. PACKAGE

This function delimits the beginning of a package of functions. It takes one required argument, a STRING, which is the name of the package. This STRING uniquely identifies the package within a library of packages (see section 4).

In a PACKAGE those ATOMs which are specified as entries live in a separate OBLIST of their own, called the entry OBLIST. The ATOM naming this OBLIST is on the PACKAGE OBLIST and has the same name as the PACKAGE itself. Thus, an entry 'X' of a PACKAGE 'Y' would have as its 'full-trailer' name: X!-Y!-PACKAGE!-.

PACKAGE blocks (sets up) the current OBLIST path so that the ATOMs which are internal to the PACKAGE fall into an OBLIST which is not otherwise used. The ATOM naming this OBLIST is on the entry OBLIST of the PACKAGE, and is by default given a name created by putting the character 'I' at the beginning of the PACKAGE's name. An internal ATOM 'Z' in the PACKAGE 'Y' previously mentioned would have as its 'full-trailer' name: Z!-IY!-Y!-PACKAGE!-.

PACKAGE also keeps track of the fact that the particular PACKAGE named has been defined in this MDL process, by putting its name on the PACKAGE OBLIST.

<PACKAGE name:string
         iname:string
         size:fix
         isize:fix>

PACKAGE takes three optional arguments in addition to the required one (the optional arguments are ignored if name is already a PACKAGE):

iname is the name of the internal OBLIST of the PACKAGE; by default it is the name of the PACKAGE with the letter 'I' prefixed.

size is the number of buckets in the entry OBLIST; by default 19.

isize is the number of buckets in the internal OBLIST; by default 23.

In addition to PACKAGE, there exists the obsolete function RPACKAGE, documented here only because some programs still use it. The difference between them is that the entry OBLIST for an RPACKAGE is the ROOT OBLIST. The implication of inserting an entry into the ROOT is that this requires that the name of the entry be unique over all PACKAGEs, because the entry is, in effect, being promoted to the status of a SUBR. It is (in rare cases) useful to do this, but the correct way is with the function RENTRY (see section 2.3.1).

pymdl's own. The era's own sources use the second argument: the assembler is <PACKAGE "CODING" "IC">, so its internal oblist is IC and an assembler-private name is X!-IC!-CODING!-PACKAGE (chapter 35). The library's PACKAGE (npck.26) also consults TRANSLATE? first, the renaming table of chapter 30, so a package loaded under a translated name is defined under it.

2.3.1. ENTRY

The ENTRY function applied to one or more ATOMs declares that these ATOMs are to be put into the OBLIST reserved for entries in this particular PACKAGE. Only ATOMs declared in this way will be accessible (in the normal course of events) to functions outside this PACKAGE.

It is possible to place some entries of a PACKAGE on the ROOT OBLIST using the function RENTRY. It is recommended that instead of using RPACKAGE in those rare cases where entries must go on the ROOT, RENTRY be used instead.

All ENTRY statements should appear immediately after the PACKAGE or RPACKAGE statement. Note: never put a USE statement before the ENTRY statements; if you do, you may get the ERROR message ALREADY-USED-ELSEWHERE, meaning that the name of an entry is conflicting with an ENTRY in one of the PACKAGEs you USEd. ENTRY will also give an ERROR if it is used outside the body of a PACKAGE.

Found along the way. ALREADY-USED-ELSEWHERE is also what a ROOT atom that already has a value earns a package trying to ENTRY the same name, and pymdl met it from the other side. A pymdl built-in that the era did not have on ROOT -- GET-NAME, &, &1, and the library names PURE? and its kin -- made the era package that defines the same name fail at its ENTRY; the loader clears such placeholders before loading the package that owns the name (FR& supplies the real &), and a sweep of pymdl's 426 default globals against a bare MDL 55 found 76 the era's ROOT does not hold at all (chapter 37). The other direction bit too: a boot SETG had left a global assigned with its atom interned on no oblist, which MDL cannot express and npck's DO-ENTRY reads as "interned somewhere else"; <USE "PP"> stopped on ENTRY NULL. Every ROOT global's atom is interned on ROOT now, as MDL 55's are (<GET <OBLIST? INCHAN> OBLIST> is ROOT there, and for all of them).

2.3.2. USE

This function takes as arguments one or more STRINGs which are the names (as given to PACKAGE) of other PACKAGEs. EXTERNAL is a synonym of USE. USE causes the entry OBLISTs of the PACKAGEs named to be spliced into the current OBLIST path. Thus, references to entries of those PACKAGEs may be made after the USE, until the next ENDPACKAGE (or the next DROP or L-UNUSE if USE is being invoked outside a PACKAGE to load a file).

USE is consequently the mechanism for sharing code. If the PACKAGE being used is already loaded, its entries are made available; if not, the PACKAGE is loaded first (see section 4.1 for details on how this is accomplished).

2.3.3. USE-DATUM

USE-DATUM requires one STRING argument, the name of a data set. If the data set is not loaded, USE-DATUM loads it and creates an ATOM of the same name, on the USE-DATUM OBLIST, whose GVAL is the data set. USE-DATUM always EVALs to the data set named, regardless of whether it had to be loaded or not.

2.3.4. DROP and L-UNUSE

These functions take the same arguments as USE and USE-DATUM and undo their effects.

DROP simply splices the named PACKAGEs out of the current OBLIST path. A USE of a DROPped PACKAGE will not reload the PACKAGE but simply splice it back into the OBLIST path.

L-UNUSE splices the PACKAGE out and removes its name from the PACKAGE OBLIST, which will cause the entire PACKAGE to be reloaded if it is USEd again. L-UNUSE of a data set will remove its ATOM from the USE-DATUM OBLIST.

<PACKAGE "P1">                        ⇒ P1!-PACKAGE
<ENTRY ONE>                           ⇒ ONE!-P1
<SETG ONE 1>                          ⇒ 1
<ENDPACKAGE>                          ⇒ T
<USE "P1">
<LENGTH .OBLIST>                      ⇒ 5
,ONE                                  ⇒ 1
<DROP "P1">
<LENGTH .OBLIST>                      ⇒ 4
<USE "P1">
<LENGTH .OBLIST>                      ⇒ 5
<L-UNUSE "P1">                        ⇒ "PACKAGE REMOVED"
<LOOKUP "P1" <GET PACKAGE OBLIST>>    ⇒ #FALSE ()

2.3.5. ENDPACKAGE

The ENDPACKAGE function of no arguments terminates the definition of the current PACKAGE and undoes the lexical blocking done by the PACKAGE function. The ENDPACKAGE statement should be the last one in the file.

Found along the way. An FLOAD that errors out of a file before its ENDPACKAGE leaves the file's PACKAGE blocking in place. MDL 55 does the same -- measured with a file that leaks a BLOCK and a file that errors inside the FLOAD, the top-level path going from four oblists to three on both machines -- so it is not an engine fault, but it has consequences: the compiler image built here once ended thirteen oblists deep, and the assembler's <TITLE> instructions then resolved to a redefinition of TITLE read through the wrong path (34.7.2). A file whose load failed has left the package open; ENDPACKAGE by hand closes it.

2.3.6. PACKAGE Restrictions

There are some restrictions on what the user may do inside a PACKAGE. These are enforced by the Library System when the user attempts to submit a PACKAGE to a library.

A PACKAGE should not FLOAD or LOAD any file to obtain parts of itself. All such environment setup should be done with USE and USE-DATUM.

A PACKAGE may not reference any ATOM whose OBLIST path goes through the INITIAL OBLIST. All of a PACKAGE's non-entry ATOMs should fall naturally into the PACKAGE's internal OBLIST.

As mentioned before, the RENTRYs of a PACKAGE have the same OBLIST status as SUBRs, i.e., they must be unique among both all SUBRs and all PACKAGE entries.

Found along the way. The second restriction is enforced by the era's librarian exactly as stated, and it rejected an era file. The SQUOZE package's source on the 1983 tape ends, after its <ENDPACKAGE>, in five control characters and the letters COU -- a tape artifact -- and LUP refused to add it: ATOM, COU, GOES THROUGH INITIAL. The vendored copy stops at the ENDPACKAGE (chapter 38). The era's own tools break the first restriction freely: the assembler FLOADs its ATOSQ binary from inside <PACKAGE "CODING">, and the compiler's modules FLOAD each other's binaries, because those files were never meant for a library.

2.3.7. ENTRY Name Conflicts

It is possible to have two or more PACKAGEs (not RPACKAGEs) which have entries (not RENTRYs) with the same PNAME. If the user needs both PACKAGEs at the same time, he may USE them both and refer to the ambiguous entries by their 'full trailer' names. All of the non-ambiguous entries in both PACKAGEs may still be referenced by PNAME only.

28.4 Where a USE finds a package

The manual defers this to its library chapter (30 here), and on the era the answer was a disk: the LIBMUD library, then the directories on the search path. pymdl keeps that order and adds the places its own tree puts packages, so that USE finds a package wherever this checkout has it:

  1. A package already defined in this process is spliced in and nothing is loaded.
  2. pymac packages -- this machine's compiled code, the peers of era FBINs (33.14) -- are preferred, as the era's L-SECOND-NAMES put FBIN before source: the compiled form when there is one.
  3. The library search path, L-SEARCH-PATH, in order, exactly as the era's USE walked it: a STRING element is a binary library database, read by the era's own LIB reader (ENTRY-FIND, DEFER-FIND); a VECTOR element is a source directory. LIBMUD; and PS:<MDLLIB> on that path resolve first under the file root and then to the library tools/build_mdllib.py builds from the vendored sources (chapter 2).
  4. Only when the path yields nothing do the vendored substitutes serve: the MUDBUG tools by name, with the loads they depend on; then the source directories of the tree, mudbug/ and mprog/, the era's own home for the small programs.

A name that is not defined at all, referenced at the listener, goes through the dynamic loader of 30 (4.1.2): the current libraries are asked which package has an entry of that name, and it is USEd. L-NO-MAGIC turns that off.

Measured, MDL 55, 2026-08-30 and 2026-09-09. Two OBLIST facts the package system rests on, both once wrong here. Two atoms sharing a PNAME on different oblists are different variables with different value cells -- <SET FOO 1> <SET FOO!-INTERRUPTS 2> leaves .FOO 1 and .FOO!-INTERRUPTS 2 -- and a BLOCK that hides INITIAL makes a bare name a new atom, so a SET inside it is UNBOUND-VARIABLE outside (chapter 18 has the five measurements). And REMOVE takes the atom off its oblist for good: a bare read afterwards interns a fresh atom, and the removed one keeps its own values under no name. That second fact is what lets the assembler <REMOVE "TITLE" <ROOT>> and redefine TITLE for its own use without touching the compiler's (35), and pymdl gave the removed atom back until the 9th of September.