pymdl

The pymdl book

30. The library system

The library system connects package and entry names to the files that define them. USE requests a package explicitly; the dynamic loader can find an entry when it is first referenced. Both use the library database, which the Library Update Program (LUP) maintains.

pymdl runs MIT's MDL library reader and librarian with Python support for their low-level operations. tools/build_mdllib.py uses LUP to build a library from the vendored package sources, following the workflow in section 4.1.10. This chapter retains the environment manual's account of that workflow. Section 30.3 then explains the installed library, search paths, and Python packages specific to pymdl.

Chapter 4. The Library System

A coherent unified library system serves to facilitate the sharing of algorithms and data by imposing a discipline appropriate for the particular environment. The MDL Library System provides:

  • A uniform access method for referring to functions and data outside of the current logical group;
  • Lexical blocking, eliminating difficulties arising from overlap of names between different logical groups;
  • Automatic loading of functions for the user who knows only the name of the function which is wanted;
  • A facility whereby functions which may be necessary only in unusual situations are loaded only in the event that they are needed.

The MDL Library System may be divided into distinct parts. These are:

  • The Package System, the collection of routines used to provide lexical blocking for a logical group (see section 2);
  • The 'explicit' loading facility, the routines used to explicitly indicate that references are being made to a particular logical group;
  • The 'implicit' (or 'dynamic') loading facility, the machinery for automatically loading functions when they are needed during console interaction.

4.1. Program Libraries

In the previous discussion of the Package System and USE (see section 2.3.2), we glossed over the mechanism by which a PACKAGE is loaded when another PACKAGE (or the user at his terminal) refers to it. We will now give the details.

There are two types of loading common in MDL programming: 'explicit' loading, such as USE may initiate, and 'implicit' or 'dynamic' loading, initiated by attempting to call or examine a function that is not currently loaded.

In the case of 'explicit' loading, it is necessary somehow to map the name of a PACKAGE into a file name which contains the body of that PACKAGE. The mechanism for doing so must be flexible enough to allow both 'installed' programs (those that have been debugged and submitted to the library) and developmental programs to be loaded. It must also be tailorable for special needs, such as libraries for specific systems and personal libraries for individual users.

In the case of 'implicit' loading, the further mapping from the specific ENTRY of a PACKAGE referenced to the PACKAGE itself must be performed. It must deal with the case of two or more PACKAGEs each containing an ENTRY with the same PNAME.

For programs that are 'public' or 'installed', both of these mappings are performed by a library. A library is a file which contains pointers between the names of ENTRYs of PACKAGEs and the PACKAGEs containing them, and from PACKAGE and DATUM names to the files containing them.

The standard library is named LIBMUD and lives on a directory named LIBMUD (on ITS) or MDLLIB (on Tenex/Tops-20), but other libraries, personal or special purpose, may also exist; the mechanisms for creating and maintaining them are the same in both cases.

pymdl's own. The standard library is here under both names. LIBMUD;LIBMUD and PS:<MDLLIB>LIBMUD resolve, first under the file root so that a site's own library shadows the installation as it did on the era, and then to the one pymdl ships, src/pymdl/mdl/mdllib/libmud.nlib with its data file beside it (chapter 2). It is a real era library -- the same format the 1976 and 1983 LIBMUDs in tests/data/ are in, read by the same code -- written by the era's librarian over pymdl's vendored packages when the tree is built (30.3).

4.1.1. Library Searching

When a PACKAGE is USEd, MDL first checks to see if the PACKAGE is already loaded, by looking up the PACKAGE name on the PACKAGE OBLIST. If the PACKAGE is not yet loaded, MDL must search for the file containing the body of the PACKAGE.

When MDL searches, it does so under the direction of a search path stored as the LVAL GVAL of the ATOM L-SEARCH-PATH. This value is a LIST, each element of which specifies 'a place to look' for the PACKAGE. These elements may be:

  • "file-name" -- A STRING refers to a library file; "LIBMUD;LIBMUD" for example.
  • [] -- An empty VECTOR refers to the <SNAME> directory. The directory will be searched for files whose names are the name of the PACKAGE being loaded (truncated to six characters on ITS) and second names from the LVAL GVAL of the ATOM L-SECOND-NAMES, which is a VECTOR of STRINGs which are possible second names for the file.
  • [dir:string-or-false] -- A non-empty VECTOR specifies a directory. The first element of the VECTOR gives the directory as a STRING or a FALSE, the latter case meaning <SNAME>. If that is the only element, L-SECOND-NAMES specifies the file names to look for. If there are other elements, they should be STRINGs to use in place of L-SECOND-NAMES.

A search path may consist of any number of such elements. The loader will examine them sequentially, attempting to find the PACKAGE being loaded.

The initial value of L-SEARCH-PATH (on ITS) is

~~("LIBMUD" "LIBMUD;LIBMUD" [] ["MBPROG"] ["MPROG" ">"])~~
("LIBMUD" [] "LIBMUD;LIBMUD" ["DSK" "MBPROG"] ["DSK" "MPROG" ">"])

and on Tenex/TOPS-20, it is

~~("LIBMUD" "<MDLLIB>LIBMUD" [] ["MDLLIB"])~~
("LIBMUD" [] "PS:<MDLLIB>LIBMUD" ["PS" "MDLLIB"])

This instructs the loader to first search the user's personal library (if it exists), then the 'public' library. Next, search the user's directory the user's directory for a file whose first name is the PACKAGE name, and whose second name is specified by L-SECOND-NAMES, then the 'public' library. If that fails, perform the same search on the library directory, and finally (on ITS), look for a source version of the PACKAGE on the source directory.

The initial value of L-SECOND-NAMES (on ITS) is

["FBIN" "GBIN" "NBIN" ">"]

and on Tenex/TOPS-20, it is

["FBIN" "GBIN" "NBIN" "MUD"]

To give a simple example of how this mechanism may be tailored for individual needs, consider a programmer debugging a subsystem. If he wants his debugging versions of various PACKAGEs to be loaded before the installed versions, he CONSes a new element onto L-SEARCH-PATH so that it contains

([] "LIBMUD" "LIBMUD;LIBMUD" [] ["MBPROG"] ["MPROG" ">"])

(assuming the files with his debugging versions are on the <SNAME> directory).

,L-SEARCH-PATH     ⇒ ("LIBMUD" [] "LIBMUD;LIBMUD" ["DSK" "MBPROG"] ["DSK" "MPROG" ">"])
,L-SECOND-NAMES    ⇒ ["FBIN" "GBIN" "NBIN" ">"]

Found along the way. The struck lines are what the package system's own source says against the manual. npck.26 SETGs both variables -- they are GVALs, and USE, L-LOAD and the rest read ,L-SEARCH-PATH -- and its ITS path puts the user's directory [] before the public library, with the device spelled out in the directory elements (["DSK" "MBPROG"]), while the manual's order has the library first and the elements bare. The Tenex path in the file is ("LIBMUD" [] "PS:<MDLLIB>LIBMUD" ["PS" "MDLLIB"]), chosen on <G? ,MUDDLE 100>, which is how pymdl's Tenex personality gets it. The manual describes the search correctly and the file's order is the one that ran.

pymdl's own. A STRING element names a binary library, and pymdl's USE reads it with the era's own LIB package -- PACKAGE-FIND over the database's hash tables -- exactly as the era did; the file the record points to is then resolved under the file root, and if the record is a data-file entry (4.1.6) that is what loads. A VECTOR element is a directory under the file root, searched for the second names. So a checkout with a LIBMUD; directory of its own, or a MPROG; of sources, is searched the way the manual says. What follows when the whole path yields nothing is 30.3: the places pymdl's own tree keeps packages. PYMDL_PREFER_SOURCE (Appendix D; implied by PYMDL_NO_PDP10) reorders L-SECOND-NAMES so ">" comes before "FBIN": on the era compiled code ran natively and belonged first, and here an FBIN has to be emulated where a source can be interpreted. That is 4.1.1's own tailoring, the file untouched and only its value reordered, and only when asked.

4.1.2. Dynamic Loading

To ease the use of 'top level' routines from the console, a feature is provided whereby the Library System can load a PACKAGE of functions automatically when one of the functions which is an ENTRY in that PACKAGE is invoked by name. This facility is not available for use by other PACKAGEs of functions, which must refer explicitly, via USE, to PACKAGEs which they require: while a human can resolve the difficulty of possible multiple PACKAGEs with ENTRYs of the same name, a program cannot.

When an error is generated because a FORM is evaluated, and the first element of that FORM is an ATOM which has no value, and the particular ATOM is in the INITIAL OBLIST, an error handler established by the Library System determines if there are any PACKAGEs in the current libraries which contain an ENTRY with the same name as the PNAME of that ATOM. If there is one such PACKAGE, it is loaded, and the evaluation which got the error is continued with the correct value. If there is more than one such PACKAGE, the possible choices are displayed, the user is asked which is the desired PACKAGE, and it is loaded. If there are no PACKAGEs with ENTRYs of the correct name, the error is not handled, and so it will fall into the standard error mechanism. This same procedure is also invoked when GVAL is applied to an ATOM on the INITIAL OBLIST and the ATOM has no value.

pymdl's own. The entry-to-package question goes to the current libraries first, through the era's ENTRY-FIND, and only when no library answers does pymdl's own index of the tree's package sources answer. A name two packages export asks the user which, as the manual says, and L-ALWAYS-INQUIRE makes it ask always. Chapter 28 has the case this produces in this checkout: the manual's own sample package names its entry TIME-STRING, which the library's TIMFCN exports too.

4.1.3. USE-DEFER

It is sometimes desirable to have available functions that are rarely invoked, but are nonetheless available. (One example would be certain error handling routines.)

The USE-DEFER function sets up the OBLIST path so that, when a reference is made to an ENTRY in the specified file, the correct ATOM is found, but the PACKAGE is not actually loaded at that time. When a function at a later time tries to call the function which is the value of one of the entries in this PACKAGE, the whole PACKAGE will be automatically loaded. USE-DEFER has two constraints which USE does not. First, the PACKAGE must be in one of the currently active libraries; it may not simply be a file as in the case of USE. Second, no reference may be made to ATOMs which are entries but do not have values which are applicable. In other words, ATOMs which are entries because they are data (rather than functions) may not be referenced when USE-DEFER is employed instead of USE.

Because USE-DEFER utilizes the dynamic loader, which utilizes the ERROR interrupts, USE-DEFER will not work in a demon or any other MDL program which sets up its own error handlers. All such MDL programs should SETG the ATOM L-NO-DEFER to a non-FALSE, which (as explained previously) will cause USE-DEFER to behave exactly like USE. Then, PACKAGEs containing a USE-DEFER can be used without modification in demons and the like.

<USE-DEFER "FINDATOM">                          ⇒ T
<TYPE ,FINDATOM>                                ⇒ RSUBR
<LOOKUP "FINDATOM" <GET PACKAGE OBLIST>>        ⇒ FINDATOM!-PACKAGE

The entry is applicable before its package has loaded -- the stub the library's DEFER-FIND record makes for it is an RSUBR -- and the first call loads FINDATOM and runs it.

4.1.4. USE-TOTAL

USE-TOTAL is analogous to USE, but instead of splicing in only the ENTRY OBLIST of the PACKAGE, it additionally splices in the internal OBLIST. This is useful in some debugging situations, as it reduces the number of trailers printed and also makes the internal identifiers of the PACKAGE more accessible.

4.1.5. Translations

It is occasionally useful to have more than one copy of a particular PACKAGE loaded at once. One example that comes to mind is the case of debugging a debugging PACKAGE. The Library System contains a mechanism for 'translating' a PACKAGE name into another one. More specifically, it is possible to tell USE: 'If you ever load the PACKAGE named foo, pretend it was named bar instead.' Note that this does not change the searching and loading procedure described above, only the names of the OBLISTs and so on used to store the ATOMs in the PACKAGE.

<TRANSLATE old:string new:string-or-false>

causes the PACKAGE old, when it is USEd, to behave as if it were named new. If new is FALSE, it means that old should be loaded as though it were not a PACKAGE at all: its ATOMs will appear on the DEFAULT OBLIST or <1 .OBLIST> (normally INITIAL).

<UNTRANSLATE old:string>

causes any translation of old to be removed.

<TRANSLATIONS>

lists all translations currently in existence.

.L-TRANSLATIONS

is a LIST containing all the translations.

<TRANSLATE "TRACE" "TRACE2">        ⇒ "TRACE2"
<TRANSLATIONS>                      ⇒ T
<UNTRANSLATE "TRACE">               ⇒ "TRACE"
,L-TRANSLATIONS                     ⇒ ()

(TRANSLATIONS prints TRACE-->TRACE2 between the first two answers, and No translations when there are none. L-TRANSLATIONS, like the search path, is a GVAL in the source.)

4.1.6. The Library Data File

In addition to its ability to map between PACKAGEs, ENTRYs, and the files which contain them, the library serves another purpose. If a user is compiling a function which USEs a given PACKAGE, that PACKAGE is not usually going to be run. All that is necessary is to examine the calling sequences of its functions, and make sure that all 'side-effects' (such as the definition of new TYPEs) occur. If only these necessary parts of the PACKAGE are loaded, a great saving of time and space is effected.

The library data file provides a way of achieving this end. When a PACKAGE is added to the library, more information than the list of ENTRYs and the file containing the PACKAGE is collected. In particular, MANIFEST GVALs, NEWTYPE definitions, some MACROs, and RSUBR DECLs are stored. Since this is the information used by the compiler, one can save a great deal of space and time by using information from the library where possible.

If .L-USE-DATFILE is true, USE of a PACKAGE will load from the data file if possible. It is impossible if the PACKAGE has changed since the data file entry was created. In those cases, the PACKAGE itself is loaded instead. If .L-ALWAYS-DATFILE is true, an ERROR will result if the data file entry is outdated: one can ERRET T to cause the real PACKAGE to be loaded.

USE-DATFILE is just like USE, except that it temporarily SETGs L-USE-DATFILE and L-ALWAYS-DATFILE to T.

The data file contains, for each PACKAGE, information for each interesting ENTRY: MANIFEST GVALs, NEWTYPE definitions, RSUBR DECLs, and MACROs. It also has, of course, the lists of ENTRYs and RENTRYs needed by the dynamic loader. It does not contain other structures, nor does it contain functions. When a PACKAGE is loaded from the data file, it is effectively USE-DEFERed: if you end up needing to run part of the PACKAGE, it will be loaded dynamically.

Some PACKAGEs can not have data file entries. If a PACKAGE defines MACROs that use data not stored in the data file (if the MACRO calls a FUNCTION, for example), the PACKAGE will not get a data file entry: it would normally end up being loaded from the file anyway.

It is possible for a data file entry to become obsolete (if a new version of a PACKAGE is created without the library entry being updated). For this reason, the library is examined periodically for such entries and an attempt is made to update the appropriate entries.

Measured (tests/test_nlib.py), on MIT's 1976 library. The data file mechanics run here end to end on the era's own files: DEFER-FIND supplies a package's record from libmud-1976.nlib, and the record's forms are READ and EVALed out of libmud-1976.dat, an evacuated-format text whose access pointers count era characters. <DATFILE-LOAD "DMX" "LIBMUD-1976 NLIB"> defines DMX.MAKE's entry stub as an RSUBR-ENTRY carrying the 1976 DECL, ("VALUE" DMX!-DMX FIX FIX "OPTIONAL" FIX). For an interpreted package the era's record is the ENTRY and RENTRY lists alone -- the stubs with DECLs are written only for compiled entries -- and that is what pymdl's librarian writes for the packages it administers.

4.1.7. Run-time Switches

There are a number of variables which may be set dynamically to tailor the Library System's performance.

.L-SEARCH-PATH as described above (see section 4.1.1) is a LIST specifying the libraries and directories to look in, and the files to look for when trying to load a PACKAGE. This variable is used by USE, USE-DEFER, USE-DATUM, and the dynamic loader.

.L-SECOND-NAMES as described above (see section 4.1.1) is a VECTOR of the second names of files to look for when attempting to load a PACKAGE from a directory.

.L-NOISY -- If the GVAL of L-NOISY is non-FALSE, the names of PACKAGEs and DATUMs are printed whenever they are loaded, dynamically or otherwise. This feature may be turned off by SETGing L-NOISY to #FALSE (). L-NOISY has an initial GVAL of T.

.L-NO-MAGIC -- Dynamic loading may be disabled by SETGing L-NO-MAGIC to a non-FALSE. L-NO-MAGIC has an initial GVAL of a FALSE.

.L-ALWAYS-INQUIRE -- If the GVAL of L-ALWAYS-INQUIRE is non-FALSE, the dynamic loader will always ask the user before it loads anything. The GVAL of L-ALWAYS-INQUIRE is initially a FALSE.

.L-NO-DEFER -- If the GVAL of L-NO-DEFER is non-FALSE, USE-DEFER will work exactly like USE. L-NO-DEFER is initially SETGed to #FALSE ().

,L-NOISY              ⇒ T
,L-NO-MAGIC           ⇒ #FALSE ()
,L-ALWAYS-INQUIRE     ⇒ #FALSE ()
,L-NO-DEFER           ⇒ #FALSE ()
,L-USE-DATFILE        ⇒ #FALSE ()

L-NOISY is why a USE at pymdl's listener prints the loaded package's name after a slash -- /TRACE -- as the era's did.

4.1.8. Library Utility Functions

A number of functions exist which allow the user to examine libraries, list their contents, and retrieve their entries. All of the functions below except L-PATH and L-OBL accept an optional STRING argument, a library specification. If it is defaulted, they operate on the public library, specified by the string "LIBMUD;LIBMUD" or "<MDLLIB>LIBMUD".

<L-LOAD package:string library:string>

L-LOAD requires a STRING (the name of a PACKAGE or DATUM) and attempts to load it from library (if given) or the current libraries, as per L-SEARCH-PATH.

<L-FIND function-name:string library:string>

L-FIND requires a STRING (the name of an ENTRY), returning a UVECTOR of two-element VECTORs of the form:

[package-in-which-function-exists:string
 library-in-which-package-exists:string]

This finds all of the entries which have the same PNAME but are in different PACKAGEs.

The remaining functions are in the PACKAGE "L", rather than in the PACKAGE "PKG". For each of these, the optional library argument is by default the library: that is, "LIBMUD;LIBMUD" or "<MDLLIB>LIBMUD".

<L-FILE package:string library:string>

L-FILE requires a STRING (the name of a PACKAGE or DATUM) and returns a STRING which is the file specification of the file, pointed to by the library, which contains the body of that PACKAGE or DATUM.

<L-WHERE package:string library:string>

L-WHERE is similar to L-FILE but returns a VECTOR of STRINGs which is the actual complete file specification of the file containing the PACKAGE (i.e., the 'real' slots in a CHANNEL open to the file).

<L-LISTE library:string>

L-LISTE prints the names of all of the entries of all of the PACKAGEs in the library.

<L-LISTP library:string>

L-LISTP prints the names of all of the PACKAGEs and DATUMs in the library.

<L-COUNTE library:string>

L-COUNTE returns a FIX, the number of entries defined by all of the PACKAGEs in the library.

<L-COUNTP library:string>

L-COUNTP returns a FIX, the number of PACKAGEs and DATUMs in the library.

<L-LISTPE package:string library:string>

L-LISTPE requires a STRING (the name of a PACKAGE) and prints the names of all of its entries.

<L-PATH>

L-PATH prints a list of the names of all of the OBLISTs in the user's current OBLIST path.

<L-OBL atom>

L-OBL requires an ATOM and returns an ATOM, the name of the first ATOM's OBLIST. L-OBL is in fact

<GET <OBLIST? atom> OBLIST>

<USE "L">
<L-COUNTP "LIBMUD;LIBMUD">                ⇒ 43
<L-FIND "TRACE" "LIBMUD;LIBMUD">          ⇒ ![["TRACE" "LIBMUD;LIBMUD"]!]
<L-FILE "TRACE" "LIBMUD;LIBMUD">          ⇒ "DSK:MDLLIB;TRACE MUD"
<L-FILE "COMPILE" "LIBMUD;LIBMUD">        ⇒ "DSK:MDLLIB;COMPILE PYM"
<L-LISTPE "TRACE" "LIBMUD;LIBMUD">        ⇒ T
<L-OBL PACKAGE>                           ⇒ ROOT
<L-OBL L-OBL>                             ⇒ L!-PACKAGE
<L-LOAD "PINFO">                          ⇒ "DONE"
<TYPE ,PCK-INFO>                          ⇒ FUNCTION

L-LISTPE prints Entries: and the seventeen names of TRACE's entry oblist; L-PATH prints INITIAL, ROOT, PP!-PACKAGE, EDIT!-PACKAGE, L!-PACKAGE, one per line, in a session that has USEd "L".

pymdl's own. Given a library, these utilities answer from it: the shipped one, or MIT's own 1983 index copied under the file root as MDLLIB;LIBMUD NLIB, over which L-COUNTP answers more than two hundred and L-FILE "PP" answers "DSK:MUDBUG;PPRINT FBIN", where it lived (tests/test_library_system.py). Defaulted, they answer over the catalogue of pymdl's own compiled packages -- the pymacs, 30.3 -- which is this installation's personal library in 4.1.1's sense: the bare "LIBMUD" that the search path names first. That is a choice, and an earlier version got it wrong in a way worth recording: the utilities once printed the same nine catalogue names for every library on earth, because they never looked at their argument. L-WHERE answers FALSE here; there is no CHANNEL open to an ITS file to read the 'real' slots from.

4.1.9. Internal Library Functions

There are several internal functions used for searching libraries (which is, after all, all the Library System ever does).

<PACKAGE-FIND package:string library:string>

searches library for package. If there is no such PACKAGE or DATUM in library, it returns a FALSE. Otherwise, it returns a STRING, which is the name of the file containing package.

<ENTRY-FIND entry:string-or-atom library:string>

searches library for PACKAGEs containing entry. It returns a FALSE if there are none, otherwise a LIST some multiple of four elements long, where each set of four elements describes a package containing an ENTRY with that PNAME. These elements are:

  • package:string is the PACKAGE being described.

  • file-name:string is the file-name containing the package.

  • rpackage?:atom-or-false indicates, if non-FALSE, that the package is in fact an RPACKAGE.

  • rentry?:atom-or-false indicates, if non-FALSE, that the entry is an RENTRY.

returns a FALSE if the PACKAGE or DATUM is not found, or a VECTOR of five elements describing the package:

  • rpackage?:atom-or-false indicates, as above, whether the package is an RPACKAGE.
  • name:string is the name of the package.
  • file-name:string is the file containing the package.
  • entries:list is a LIST of the PNAMEs of the ENTRYs of the package.
  • rentries:list is a LIST of the PNAMEs of the RENTRYs of the package.

This is all the information about the package that the library contains.

<LENGTH <DEFER-FIND "TRACE">>       ⇒ 5
<1 <DEFER-FIND "TRACE">>            ⇒ #FALSE ()
<2 <DEFER-FIND "TRACE">>            ⇒ "TRACE"
<LENGTH <ENTRY-FIND "TRACE">>       ⇒ 4
<PACKAGE-FIND "NO-SUCH-PACKAGE">    ⇒ #FALSE ()

Measured (tests/test_nlib.py), on the 1976 LIBMUD. The three functions are the era's, in nlib.17, and over MIT's own library they answer what MIT's library held: PACKAGE-FIND of "PP" is "DSK:MUDBUG;PPRINT FBIN" and of "HISTORY" is "DSK:LIBRM2;HISTOR NBIN"; ENTRY-FIND of "PPRINT" is ("PP" "DSK:MUDBUG;PPRINT FBIN" #FALSE () T), an RENTRY of PP; DEFER-FIND of "HISTORY" lists its four entries. The reader runs on Python stand-ins for the handful of hand-assembled primitives at the end of nlib.mud -- DB-ACCESS, which mapped the file's pages, MAKE-STRING, HASH-NAME -- and the hash was verified against every entry of the 1976 file: 302 of 302 packages and 2380 of 2389 functions land in the bucket the era's own lookup would have searched. The other nine are mis-bucketed in the file, era bit-rot, unreachable by the era's lookup too. The functions on ROOT here, the ones the examples above call without a library argument, are pymdl's: they answer from the tree's own index and give host paths, which is why the examples show a LENGTH rather than the string.

4.1.10. Library Maintenance

The PACKAGE called "LUP" contains functions used to modify libraries, and to add, update and delete PACKAGEs and DATUMs. It should be noted that libraries do not contain the bodies of PACKAGEs and DATUMs. Rather, they point to files which contain these.

<LUP-ACT library:string>

requires one argument, a library specification STRING, and activates the library so specified. If the library doesn't exist, it is created. In order to protect the library from loss due to system or MDL crashes, activating a library for modification copies the library data files and locks the library so that no one else may modify it. Modifications are made to the copies, which are renamed back over the originals only when the library is explicitly deactivated. Obviously, PACKAGEs added to a library aren't available, even to the person adding them, until the library is deactivated.

<LUP-DCT>

deactivates the currently active library.

<LUP-ADD-PACK package-file:string
              update?:boolean
              datfile-entry?:boolean>

package-file is a file specification of the file containing the body of the PACKAGE to be added. LUP-ADD-PACK will find the PACKAGE statement within the file (or complain if it can't).

update? is optional, and if non-FALSE, it allows the PACKAGE to update an older version of itself, something which is not otherwise allowed. Note that, since the library points to the file which contains the body of the PACKAGE, that file should not be deleted later, else the library won't be able to find it.

datfile-entry? is by default T, but if it is FALSE, no entry will be created in the datfile for this PACKAGE. Since datfile entries are generally useful only in the compiler (and similar environments), it doesn't do much good to have them for PACKAGEs that are only called from top level (e.g., FINDATOM).

When adding a PACKAGE to the public library, the PACKAGE's object file should be copied to the appropriate library directory ("LIBMUD" on ITS, or "" on Tops-20) and the library pointed at that copy of the file. If no library is activated when LUP-ADD-PACK runs, it will activate "LIBMUD;LIBMUD" or "<MDLLIB>LIBMUD".

<LUP-ADD-DATUM name:string
               file:string
               update?:boolean>

is analogous to LUP-ADD-PACK, adding a DATUM to the active library. LUP-ADD-DATUM requires two STRING arguments, the name of the DATUM and the specification of the file which contains the body of the DATUM. LUP-ADD-DATUM will accept the same optional argument that LUP-ADD-PACK accepts, with the same meaning and default. The same restrictions concerning the file which contains the DATUM also apply.

<LUP-DEL package:string>

LUP-DEL requires one STRING argument, the name of a PACKAGE or data set, and deletes that PACKAGE or DATUM from the currently active library. LUP-DEL does not touch the file containing the body of the PACKAGE or DATUM.

<LUP-MOVE package:string file:string>

causes the file pointer of package to be changed to point to file. This is a faster operation than re-adding the PACKAGE, and it is intended for situations in which an existing library file has been moved for some reason.

<LIB-GC library:string>

garbage-collects the library in question, if this is required. Garbage-collection is occasionally useful since it causes all the elements of each hash bucket to live near each other in the library file, thus improving performance during searches. It also allocates some free storage in each page of the file.

Measured (tests/test_nlib.py). LUP is the era's libmud/nlup.72 -- it was a Python stand-in until 2026-09-10, and the stand-in went when the era's MDL was found to run -- and it writes real libraries here. <LUP-ACT "TLIB"> creates a 1024-word database whose first word is the SIXBIT for LIBMUD, 545142556544 octal, with the hash tables and the free chain, and locks it through the copy-and-rename dance the manual describes; LUP-ADD-PACK loads the package source, finds its PACKAGE statement and adds its entries with the bucket and string surgery the source performs; LUP-DCT answers "DONE" and renames the new version over the old; a fresh interpreter's reader then answers from the file. LUP-ADD-PACK and LUP-DEL run on a copy of the real 1976 LIBMUD, and LIB-GC compacts it, every entry still found afterwards. The librarian refuses what the manual says it refuses: a file that interns an atom through INITIAL is answered ATOM, name, GOES THROUGH INITIAL, which is how a tape artifact in the era's SQUOZE source was caught (chapter 28).

4.2. The Pure-mapping Library

The basic idea behind MDL pure mapping is to separate out the code part of RSUBRs in compiled programs. The RSUBRs themselves are kept in a file known as an FBIN (see 6.3). These RSUBRs do not contain the code but instead point to a file which contains the code. This scheme has several advantages. First, the code can be dynamically mapped in when needed. This allows MDL to use more code than will fit in the virtual address space of the machine it is running on. Secondly, since the code is pure it can be shared between several MDLs using it. Finally, the FBIN file itself is smaller than a corresponding NBIN file and therefore FLOADs more rapidly.

In the most basic implementation of FBINs, there are three files: the FBIN, the SAV file (which contains the code), and the FIXUP file, which contains the information necessary to update the SAV file for new releases of MDL. As is obvious, this entails a lot of files, and potentially a lot of file directories. The MDL Pure-mapping Library reduces this storage overhead by collecting all of the SAV and FIXUP files together.

The scheme uses two large data bases, each contained in one file. The data bases are called "SAV" and "FIXUP". These files store all currently existent SAVs and FIXUPs for all existing versions of MDL. Each data base is structured like a file system. There is a main 'directory' that points to a number of other 'directories', each of which points to a number of 'files' inside the data base. In this section the word 'file' or 'directory' in quotes refers to an object inside a data base. The files containing the data bases are named (on ITS) "MUDSAV;SAV FILE" and "MUDSAV;FIXUP FILE". On Tenex/TOPS-20, they are "<MDL>SAV.FILE" and "<MDL>FIXUP.FILE".

pymdl's own. Chapter 33 (33.13) is the pure-mapping library as pymdl reads it: the 1983 MUDSAV;SAV FILE and FIXUP FILE from the tape, the <PCODE "name" offset> that an FBIN's RSUBR points into a block with, and SAV-BLOCK, which hands a block back as a CODE. The compiler of chapter 34 runs from those blocks in its raw configuration, and the assembler's lost sources were read back out of one (chapter 35). The whole round trip -- GLUE a group, PDUMP it into SAV/FIXUP files, the demon's add, SAV-BLOCK reading it back -- is tested end to end (chapter 31).

4.2.1. The Demon

While all MDLs can read from the Pure-mapping Library, there is only one program which can write into it. This is a maintainer demon which runs once a day to keep the Library updated. This demon can add 'files', delete 'files', and add 'subdirectories' to both data bases.

To facilitate updating of the Library there is a directory on which to put files to be added as well as files to indicate what is to be deleted. This is the "MUDTMP" directory on ITS and the "" directory on Tenex/TOPS-20. Any file on it with the second name of SAVnnn or FIXnnn (where nnn is a 2 or 3 digit MDL release number) will be added to the appropriate data base. If the files "DELETE SAVS" or "DELETE FIXUPS" exist, then they will be used to delete 'files' from the data bases. These files must be ASCII files of the form

filename 1 [SPACE] filename 2 [CRLF]

An example of a valid delete file is as follows

INCODGE SAV53

The demon will ignore any deletion requests for 'files' not in the data base.

The demon does its work in several passes. The basic passes are the delete pass, the planning pass, the update pass, and the salvage pass. The delete pass deletes 'files' if either a "DELETE SAVS" or "DELETE FIXUPS" file exists on its working directory. The planning pass builds a plan file by examining the working directory and calculating where new 'files' will be placed in the data bases. The planning pass builds two files using a special internal format. These files will be used by the update pass to add 'files' to the data bases. The planning pass also enlarges the data base files as much as necessary to accommodate the new 'files'. The update phase reads the plan files and adds new SAV and FIXUP 'files' to the data bases. If a 'directory' overflows, a new 'directory' is added during this pass, and all the 'directories' are recreated (i.e., all the 'files' have to be rehashed, since they were originally placed in a 'directory' according to a hashing algorithm based on the number of 'directories'). The salvage pass is used to pick up any free storage that has been lost through system crashes or lost through holes created during the updating of the data bases.

Throughout the entire processing of the data bases attempts are made to keep the data bases in a consistent state. 'Directories' are updated only after 'files' are guaranteed to be in the data bases. The plan files described are used to keep the data bases consistent in case the system crashes while the demon is in the update pass.

A major goal in the design of the data bases is to allow recovery in case of demon errors or system disk crashes. To this end the data bases are backed up on tape every other week. (It would be dumped more often but the file is currently over two million words long). This of course leaves the problem that 'files' added to the data bases between dumps could be lost in a disk crash. To aid in recovery from such a crash, all 'files' added between dumps are copied to the "MUDRST" directory (on ITS) or the "<MDL.SAV>" directory (on Tenex/TOPS-20). Moreover a file is kept listing all the 'files' added during the previous week. This file is called "ADDED FILES". All this information is deleted once the data base is dumped to tape.

4.2.2. User Programs

Occasionally it is useful for a user to list the data base 'directories', to see if certain 'files' are in it, and copy 'files' out of the data base. DBMAIN is a program which allows the user to do these things.

The following are functions available to the user.

4.2.2.1. Listing Functions

<CLISTF data-base:string>

is used to list all the 'files' in a data base. It takes one optional argument which is the name of the data base (either "SAV" or "FIXUP"). If no argument is supplied, "SAV" is used by default. (This is always the default whenever a function takes an optional argument specifying the data base.) CLISTF prints each 'file', its length, and where it is located. The format of a line of listing is as follows:

fn1 fn2 size block

where fn1 is the first 'file' name, fn2 is the second 'file' name, size is the length of the 'file' in blocks (1024. words for SAVs, 256. words for FIXUPs), and block is the block at which the 'file' starts. This is the format used whenever listing 'files'.

<LISTF data-base:string directories>

is used to list all the 'directories' of an entire data base. It takes two optional arguments, the data-base to be listed, and a specification of which 'directories' to list. The 'directories' may be:

  • a FIX: list the 'directory' specified by the FIX;

  • a LIST of FIXes: list the 'directories' specified in the LIST;

  • the ATOM ALL: list all the 'directories' (this is the default).

lists free areas of storage in the data base. It lists the free storage in the form:

length block

where length is the length of the area of free storage and block is the block number of the starting block. This function takes one optional argument which is the name of the data base to be examined. At the end of the listing it will tell the total amount of free storage.

4.2.2.2. Find Functions

<FIND-FILE file:string data-base:string>

is used to find a specific 'file'. It takes as its argument a 'file' specification and prints the 'file' name along with the information printed by the listing functions if the 'file' exists, otherwise it returns an object of type FALSE. The 'file' specification must be of the form:

"dir fn1 fn2"

where dir is either SAV or FIXUP and fn1 and fn2 are the first and second 'file' names respectively.

<SPEC-FIND fn1:string data-base:string>

is used to find all 'files' with the same basic name, disregarding the leading digit(s) which are added to make 'file' names unique. It takes one required argument which is the fn1 to look for. It takes an optional second argument which is the data-base to look in. For example the call

<SPEC-FIND "MAIL">

might print:

1MAIL SAV53 8 360

4.2.2.3. Other Functions

<DELETE file:string data-base:string>

allows the user to delete a 'file' from a data base. It takes the same type of 'file' specification that FIND-FILE takes. The 'file' you specify will be deleted the next time the demon that maintains the data base runs.

<GET-FILE file:string output:string data-base:string>

allows the user to retrieve a 'file' from the data base. It takes two arguments. The first is the 'file' specification of the file to retrieve out of the data base and the second is the output file you wish to copy it to.

<STATUS>

gives the information about the state of the data bases. It tells the number of 'files' and the amount of free storage in each data base. STATUS takes no arguments.

4.2.3. Using DBMAIN

There are several ways to use DBMAIN. It can be used by typing

:DBMAIN function arg1 ... argn

to DDT. The jcl-line is of the form function arg1 ... argn, where function is the name of the function to be used. For example

:DBMAIN FLIST "FIXUP"

will list the free storage block for the "FIXUP" data base. DBMAIN will kill itself after finishing and can be killed earlier by typing ^S.

The jcl-line mentioned above can be modified to allow output to be routed to a file. This can be done by preceding the normal jcl-line with a string specifying the file name of the output file.

:DBMAIN "LISTOF SAVS" CLISTF

will produce a listing of the files in the SAV data base and will print this information to the file "LISTOF SAVS".

pymdl's own. DBMAIN is two things here. With a pure-mapping database pair under the file root -- a MUDSAV;SAV FILE and FIXUP FILE -- <USE "DBMAIN"> is the era's own user program, newlf.mud from the tapes, listing and finding 'files' in the real databases (tests/test_nlib.py). Without one, DBMAIN is a pymac package that answers the same questions over pymdl's compiled-code base, which is the pymac library: a 'file' is a pymac module, fn1 its package name and fn2 "PYMAC"; a 'directory' is a library directory; the "SAV" data base is the library and the "FIXUP" data base is truthfully empty, since Python modules need no release patches; sizes are in 1024-byte blocks and the location shown is the module's path; free storage is truthfully zero, and DELETE takes effect at once because there is no demon. It runs as a program too:

python -m pymdl.dbmain function arg ... ["output-file" function arg ...]

4.2.4. Garbage Collection

One problem of the MDL Pure-mapping Library is that many useless SAV and FIXUP 'files' remain as new revisions of user programs are created. To alleviate this problem there is a garbage collection system for the data bases.

The major goal of this scheme is to determine which 'files' in the data bases are no longer useful. To do this all files in the system are scanned to see what SAV files are still pointed to (not including those pointed to only from within ITS archive files). A SAV 'file' can be pointed to from FBIN files and SAVE files. A SAVE file contains pointers in its PURVEC (Pure VECTOR). All FBIN files should begin with something of the form

<PCODE file:string>

where file is the name of the SAV 'file' associated with this FBIN. If an FBIN has more than one SAV 'file' associated with it then there can be several PCODE FORMs at the beginning of the file. For purposes of garbage collection, this FORM (or FORMs) must be retained whenever an FBIN file is edited. If these PCODE FORMs disappear, their pointers to the SAV 'files' will go with them, and the SAV 'files' might be garbage collected.

Garbage collections proceed by looking at every file on the disk, building a list of all 'files' pointed to. The program then examines the data bases and any 'files' which are not pointed to are deleted.

It is possible that deletions can fragment the free area in the data bases. If compaction becomes necessary, there exists a routine to do in-place compaction of the data bases.

Found along the way. The <PCODE file> at the head of every FBIN is exactly what tools/vendor_era_data.py reads to know which blocks of the eleven-megabyte SAV FILE to vendor: the compiler's FBINs name forty-four, collected from both stagings that load them, and the vendored database holds just those (chapter 38). The manual's warning that the FORMs must be retained is the same fact from the other side.

4.2.5. Internal Structure

The "SAV" and "FIXUP" data bases have similar formats. The 'files' in the data base are pointed to by entries in what is essentially a hash table. Associated with each data base is a main 'directory' (the hash table). This 'directory' is located in the first 1024 words of the file. This main 'directory' points to other 'directories' in the data base (the hashing buckets). Each of these 'directories' is 1024 words long. The first 'file' name is used to determine which 'directory' the 'file' is on. The structure of the main 'directory' is as follows.

word 0/      number of entries in the main 'directory'
words 1-n/   block number of each 'directory'

There can be up to 1023 'directories' and each of these can contain approximately 500 'files'. This provides a virtually unlimited 'directory'.

Word 0 of each 'directory' gives its length in words. From Word 1 on are 'directory' entries. All entries have the same two word format. The first word contains the first 'file' name in SIXBIT. The second word contains the following fields:

  • length of the 'file' in blocks (a block for a SAV 'file' is 1024 words long while a block for a FIXUP 'file' is 256 words long) (bits 1-6)
  • version revision of MDL this 'file' belongs to (bits 8-17)
  • block in the data base where this 'file' starts (bits 18-35)

The 'directories' are sorted by strict numerical order (e.g., AAA SAV53 comes before 1AAA SAV53).

Each data base contains a free storage table. This table occupies the second 1024 words of the data base. The first word of the table is the number of entries in the free storage table. The remaining entries define areas of free storage. These are of the form

length,,block

where length is the number of blocks for this free area, and block is the block number at which it starts.

There are two major differences between the "SAV" data base and the "FIXUP" data base. The first deals with block sizes. In the "SAV" data base the block size is 1024 words. In the "FIXUP" data base the block size is 256 words. This smaller size allows for more compaction of these small 'files'.

The second major difference is that while there can be many versions of the same 'file' in the "SAV" data base (e.g. NCODGE SAV53 and NCODGE SAV54), there can only be one version in the "FIXUP" data base.

30.3 pymdl's library

What is in it. The shipped library is built by tools/build_mdllib.py (PYTHONPATH=src python tools/build_mdllib.py, about forty seconds; the test suite builds it when it is missing). The build loads the era's LUP, and for each vendored package -- the MUDBUG tools of chapter 29 and the small programs of mprog/ -- runs the 4.1.10 workflow, one activate, add and commit session each: <LUP-ACT "MDLLIB;LIBMUD">, <LUP-ADD-PACK "DSK:MDLLIB;TRACE MUD">, <LUP-DCT>. A file the librarian refuses to load-bless -- one that FLOADs inside itself, or errors while loading -- is administered through DO-ADD, the maintainer's direct add, also an exported LUP entry. The result is libmud.nlib (the hash-table database) and libmud.dat (the data file, holding for each interpreted package its ENTRY and RENTRY lists), with the package sources installed beside them as 4.1.10 prescribes, and libmud.list, a text manifest used only as a fast miss-probe. Nothing in that directory is original, so nothing in it is tracked; it is rebuilt from nothing each time, never grown.

<USE "L">
<L-COUNTP "LIBMUD;LIBMUD">        ⇒ 43

The Python packages. Seven packages are this machine's compiled code, the pymacs of 33.14: COMPILE (chapter 34), DBMAIN, SORTX, PRIMHK, TEMPLATE, TEMHLP, TEMHAK. They are administered into the same library as .pym 'binaries', "DSK:MDLLIB;COMPILE PYM", because Python is the peer of era assembly and a compiled package belongs in the library like any other; and USE prefers a pymac when there is one, as the era's L-SECOND-NAMES preferred FBIN to source -- the compiled form first, and here also the one that needs no emulator. A pymac's entries are RSUBRs, not SUBRs: an era program can tell, since an RSUBR is PRIMTYPE VECTOR and answers LENGTH and <TYPE? x RSUBR> where a SUBR refuses, and a SUBR was something else, a primitive assembled into the interpreter.

How a USE finds a package. In order: a package already loaded; a pymac; the search path of 4.1.1, libraries by the era's reader and directories by their second names; and only then the tree's own places -- the MUDBUG tools by name, with the loads each depends on and the environment each expects (chapter 29), and the source directories mudbug/ and mprog/, the era's own home for the small programs. The dynamic loader's entry-to-package question goes to the libraries first (ENTRY-FIND) and to the tree's index second.

Found along the way. MIT's own 1983 index, libmud-9006255.nlib with its 257-package data file, was once served here as the system library, ahead of pymdl's. An index holds pointers, and those pointed into a purified group this checkout does not have: <USE "DATIME"> answered with RSUBR-ENTRYs that died on the first call. It is era material now, read by the era's reader from tests/data/ when a test asks, and it is the only surviving record of the LSRTNS package's vintage. The shipped library is pymdl's own, under both of the era's names.