Doing RMS IO while in Executive mode is OK

 I write a lot of VAX/VMS programs that change mode to Exec, to get their work done. Lots of interesting VMS data structures are visible (and some are changeable) when you are in Exec mode.

  When your program is working with  these data structures, it's only natural to want to display some of the results. You could collect the data in Exec mode, and return to User node to output it, and that's what a lot of folks do. But that can be a pain in the sitz-platz, switching back and forth all the time. Instead , I don't return to User mode to do that, I just open a file or call lib$put_output (which uses RMS for its IO) and output the data, while still in Exec mode

  "But wait!" many of you are thinking, while you clutch your pearls and recoil in horror. "You can't do RMS IO from Exec mode!!!"  Suffused with a sense of their superiority, people (often otherwise very knowledgeable people) often attempt to correct me  about my ignorant transgressions, of using RMS where it's not welcome. After all, EVERYONE knows that Executive mode is reserved for the workings of RMS - it doesn't make sense for a user's program to do RMS operations in that mode - it would hit its head on the ceiling! Or something...

  But here's the shocking, but, for some reason, not very well known fact - using RMS while in Exec mode is perfectly fine, documented and supported.  In the "OpenVMS Record Management Services Manual", section 2.5, it says...

  2.5 Allowable Program Execution Modes

RMS should not be called from kernel mode, from executive AST mode, or from executive mode when executive-mode ASTs are disabled.

  This spells out when you CAN'T use RMS. This list of places you can't use RMS DOES NOT include non-AST, AST enabled EXEC mode - and that's how I use it. On a serious note - if you can find any documentation from DEC, COMPAQ, HP  or even VSI to the contrary on this issue - I'd sure like to hear about it.

  I don't know how the urban legend, that you can't use RMS while in EXEC mode, got started - but, boy howdy, I can sure tell you it is widespread. I'm thinking about quoting that RMS manual section in all of the code that I distribute, that uses this technique, to try and cut down on the number of times I have to argue the issue.

  Really, if the complainers would stop and think a minute, they'd realize VMS is riddled with examples of RMS activity initiated and performed while in Exec mode. SYS$GETUAI and SYS$PUTUAI for instance, or the opening of process permanent files.

  So, if you spot this occurring in any of my programs, save your breath - I don't need to be "educated" about this.

  Interestingly enough, a few test programs have shown me that RMS file IO and lib$put_output work from Kernel mode as well - but this is documented as not supported, and not a good idea, so I don't do it and don't recommend it. But, if you want to see what it's like, just change the $cmexec_s call in the example program to $cmkrnl_s...it seems to work just fine...

  Anyway, here's an example program.

rmsfromexec.mar

  To use, first make sure you have CMEXEC priv, and then...

$ mac rmsfromexec
$ link rmsfromexec
$ run rmsfromexec
Command me>testfilename.txt

  You'll see the contents of the input file on your screen, and it will be copied to a file called data.dat. The file IO will have been done in EXEC mode, and the skies didn't split. Since it does change mode, however, I have to include a disclaimer- use at your own risk - file IO from EXEC mode is documented and supported, but, no program is perfect. 

RSX Radix-50 conversion utility

  Lately, I've been working on some RSX programs that do a lot of spawning of assorted tasks.

  There were some bugs involved, and I needed to check the task name arguments to the SPWN$S calls.

  These arguments are in RADIX -50 format. RADIX-50, for those of you who haven't had the pleasure of working with it is a way to squeeze three characters into two bytes. It sounds like a lot of trouble, but, it dates from the days when saving a byte here and a byte there was a big win. RADIX-50 has a limited character set - Capital letters, 0 through 9, space, $, . and %. Encoding things in RADIX-50 is why so many things in the RSX world come in multiples of threes - like filenames and extensions, and task names, for example.

  Anyway, I had reason to think some of these variables were not correct in my code. I needed to check them out - but it's hard to look at a 16 bit word and say just what three letters it represents in RADIX-50. I could have written some debug code to evaluate them, but that sounded like even more error prone work - and I didn't go into programming to look for more work to do. I also could have evaluated the words using ODT - it has a function in it that will let you see a word in RADIX-50 format. But, I really really hate ODT. I think it really stands for Odious Debugging Technique. It requires you to have up to date listing and map files handy (preferably printed out - and I hate printing things out). To be honest here, my favorite two debugging techniques for Macro-11 are the diagnostic QIO printout (if you wonder what a value is, add a print QIO to the code and display it), and the much maligned IOT trick (if you are wondering what's happening at some point in your code, put the values of interest in the registers, and then add an IOT instruction - when it gets executed, the task exits and you get to see the register values). A lot of people hate the IOT trick - it can cause problems, for instance, when  fast supervisor mode mapping gets involved. But, it's always handy and is easy to use, so I use it.

  But, like I say, RADIX-50 variables don't lend themselves to these sort of caveman's debugging techniques. So, I figured, time to write a command line utility. I like doing that - it makes me think I'm really accomplishing something.

  So, I wrote R5A. It's a utility that can translate 16 bit words into their three letter RADIX-50 format, and vice versa. It's nothing special - it uses GCML to get a command line, and a few System Library routines to translate octal to binary, binary to ascii, and RADIX-50 back and forth.

r5a.mac
  To build

>MAC R5A=R5A
>TKB R5A=R5A

To use
>RUN R5A
R5A>50712/AS
MCR
^Z

and the other way...
>RUN R5A
R5A>MCR/R5
50712

  There are two switches, that indicate what is on the command line - an R50 string, or an octal string in ascii - /R5 and /AS. /AS is the default, since that's mostly what I need to do - convert words to R50 strings. It doesn't use CSI or TPARS to evaluate the switches, like most of my utilities do (I was in a bit of a hurry, no time for niceties this go round) so the / in the switch has to immediately follow the argument.

  Natch, you can install it and invoke as a whole command line
>INS R5A/TASK=...R5A
>R5A MCR/R5
50712
R5A>




  

Basic and Fortran programs to read temperature and pressure from the BMP driver on the PiDP-11/70

    In my last entry, I wrote about a program to read the BMP180 temperature and pressure sensor connected to a PiDP-11/70, and running the BM device driver.

  That example was written in Macro-11. I understand that not every RSX enthusiast is proficient in that language, so I coded  examples in BASIC-Plus 2 and Fortran 77.

  They do the usual - assign a LUN, and do a QIOW (via WTQIO). The results are formatted and printed out - what could be easier? The conversion of the data was a lot easier in BP2 and F77  than it was in Macro, that's for certain. It did actually take me a few hours to write this - I've written thousands of lines of BP2, but that was circa 1982 - the language has changed quite a bit in the ensuing decades. And, I thought I had written my last FORMAT statement decades ago, but, here one is in the F77 example.


  Here they are....

getoneb.b2s

getonef.ftn

  To use them, compile and link like any other BP2 or F77 program. They will return the temperature and pressure in Celsius, Fahrenheit, Bars and Inches of mercury.

>run getoneb
Temp C 22.60, Temp F 72.68, Press B 1.0171,Press Hg 30.03

>run getonef
Temp C 22.60 Temp F 72.68 Press B 1.0171 Press Hg 30.03

Program to read temperature and pressure from the BMP180 driver on a PiDP-11/70

 In our last chapter, we discussed adding a BMP180 to a PiDP11, and adding a device driver to RSX to read it (and when I say RSX, I'm talking RSX11 M-Plus).

  If you followed all that and got the thing assembled and built, loaded and connected, you were rewarded with an online, ready device...

>DEV BM0:
BM0: Loaded
>

  But, that's not all that useful, or interesting. One presumes that if there is a device there, it would be good to see some data out of it.

  So it's time for a demo program. All you have to do is assign a channel to the device, and then do an IO.RLB QIOW to it. What could be easier?

  I wrote getone.mac to do just that. It also converts the data to ASCII, in units we are familiar with - the BMP180 returns data as Celsius*100 , and Bars*10000 - I converted to degrees F and Bars, then write 'em out for your viewing pleasure.

  So, here's the code....reading the values from the device is about 4 or 5 lines of code - the rest was arithmetic and formatting.

getone.mac

  To use it, assemble and link it...

mac getone=getone
tkb getone=getone

Then run it...

>run getone
Temperature 76.20 degrees F. Pressure 1.0116 bar.

  Whew...starting to get a little warm in here...time to open the door and let a little winter air in. Next entry, I'll show a couple of high level language examples, for people who aren't familiar with Macro-11.

RSX11 M-Plus device driver for the BMP180 sensor on the PiDP-11/70

  I mentioned in the previous post that I now have a PiDP-11/70. This is a scale model of the front console of a PDP-11/70. It's controlled by  SIMH, running on a Raspbery PI. 

  The creator of this product, Oscar Vermuelen, has detailed how to extend the hardware of the PiDP-11, by connecting things to the I2C bus of the Raspberry Pi (most of the other GPIO pins are used to drive the console, but I2C was not used).

  As an example, he detailed what it would take to connect a BMP180 temperature and pressure sensor to the device, and how to add support for it to SIMH, so it could be "seen" by the virtual PDP-11 and the front panel. All of this work is detailed in http://obsolescence.wixsite.com/obsolescence/pidp-11-temp-barometer-hack.

  I undertook to do this, and, with a bit of a struggle allocating vectors that SIMH was happy with (SIMH was unhappy with a lot of the unassigned vectors I tried - I finally found it was happy with 360 and 364) I got it to work. Oscar's example used the vectors used by the paper tape devices, but I had to use new ones, since I wanted the paper tape devices to also be available, and devices can't share vectors. 

  At this point, it was possible to read the temperature and pressure, both using the lights and switches on the console, and from the SIMH command line.

  But that's only the first step of the journey. I'd like to be able to read the temperature from a program running under RSX, with an eye to displaying it and logging it.

  There are several ways to access device data in RSX. The easiest and simplest way is to write a privileged program. Privileged programs (usually) have their highest addresses mapped to the IO page of the system, so they can access device registers with simple memory reads and writes. But this is a limited trivial sort of approach, not very flexible or general, and fraught with potential for problems, if a bug causes reads and writes to go where they shouldn't. 

  The best way to access a device under RSX is a device driver. A device driver accepts requests for access to a device, and then it handles the actual IO in a (hopefully) controlled manner. A device driver also does not consume an eighth of the address space in your program to map the IO page - and lack of memory in your address space is often a problem on RSX,  since you only get 64KB of it (neglecting I&D space and supervisor mode for the moment). It also presents a uniform interface for IO, and does some  processing of the data, so it doesn't have to be included in every program that wants to access the device.

  This is a simple device, so it doesn't take a very complicated device driver to deal with it. BMDRV is a minimal implementation of a driver that can read the pressure and temperature. It only acts on read logical block requests, and returns 2 words - the first is the Celsius temperature*100. The second word is the pressure, in Bars*10000.

  Here's the data table part of the driver

bmtab.mac

  And here's the code part of the driver. 

bmdrv.mac

  To use this, first, connect the BMP180 chip to the I2C bus on your Raspberry PI. Like I said above, how to do this is documented on Oscar's page.

  Then, modify the SIMH sources, per Oscar's document mentioned above. However, use vectors 360 and 364 for the new devices you create - don't use the existing vectors to the paper tape devices.  Remember to include the attach statement in your SIMH .ini file.

  Copy the two files above to your RSX system, and assemble them

mac bmdrv,bmdrv=lb:[1,1]exemc/ml,[11,10]rsxmc/pa:1,sy:[200,200]bmdrv.mac
mac bmtab,bmtab=lb:[1,1]exemc/ml,[11,10]rsxmc/pa:1,sy:[200,200]bmtab.mac

  Substitute the directory that the files are in for [200,200] above.

  Then, time to task build

>TKB

TKB>bmdrv/-mm/-hd/sq,bmdrv/-sp,bmdrv=
bmdrv,bmtab,lb:[1,54]rsx11m.stb/ss,lb:[1,1]exelib/lb

 /

stack=0

par=gen:120000:40000

//

  Then copy bmdrv.stb and bmdrv.tsk to [1,54].

>pip [1,54]=bmdrv.tsk/nv
>pip [1,54]=bmdrv.stb/nv

  Now load the driver 

>loa bm:

  Then bring the controller and unit online

>CON ONL BMA
>CON ONL BM0:

  Have a look and see if it all loaded and connected OK

>DEV BM0:

  You should see

BM0:     Loaded

  If you see something else...then something went wrong....

  Now, you should be able to do a QIO to the BM0: device, to read a temp & pressure value.

  I'll post an example program in my next post - this is already getting too long...

PiDP11 display register and RSX11

  Recently, a fellow in Switzerland, Oscar Vermuelen,  has been producing some scale models of the PDP-11/70 front panel.  Driven by SIMH running on a Raspberry Pi, and using software from Jorg H )

  It comes in kit form, but only takes a few hours to assemble. The build quality on the components of this kit is really great. I've really enjoyed having mine - it's been a long time since I've had access to the lights and switches of a PDP-11 console. 

  Here's a really terrible phone camera picture of it. It looks much better in person.

   PDP-11 consoles (both real and PiDP11 emulated) came with a switches and display register. A read to this register would read the values set in the console toggle switches. A write to it would cause the value written to be displayed in the DATA lights, if the console display select switch was set to DISPLAY REGISTER.

  RSX comes with a utility for reading the console switches (the SWR utility), but doesn't include any facility for setting the display register lights from a program.

 I wanted a demo program to show how to read  the switches and write to the display from an RSX program. The display register's address is 17777570. But, on RSX, it's not a simple as just writing a program to store a value at that address. The PDP-11/70 (and the PiDP11 simulator) has a 22 bit address memory  space. Programs running on a 70 under RSX  have a 16 bit address space. They access real memory in 8KB chunks, that are mapped to  8KB chunks in the 22 bit memory space.

  The top of that 22 bit space is called the IO page. It's where the CPU and device registers are (IO is mostly memory mapped on the PDP11, mostly). In order for a program to access the IO page, it has to be task-built as a privileged task. Among other things, this causes  the  top 8KB chunk of the task to be mapped to the IO page. When this is done, the 16 bit addresses in the task that are in that top 8KB chunk correspond to the addresses in the IO page. Thus, 17777570 in the IO page, in 22 bit space,  can now be accessed as 177570 in the program. And this is the sketchiest bare bones explanation of PDP-11/70 memory management you'll ever find - don't get me started about I&D space, or user, supervisor and kernel mode mapping registers. 

  Here's a trivial example of how to do that. It reads the switch settings, and writes them into the display register. To use, switch the PiDP display to "DISPLAY REGISTER" from "DATAPATHS", set a value in the switches, and then run the program. The value in the switches will then be shown in the console data lights.

File readwrite.mac

******************************
        .mcall  exit$s

        .psect  cod,ro,i,gbl,rel,con

readwrite:
        mov     @#177570,r0     ;read the switches
        mov     r0,@#177570     ;write the display

        exit$s                             ;donesville

        .end    readwrite
[EOB]



********************************

  To build it, save the above as readwrite.mac, then assemble and link it:


>tkb readwrite/pr=readwrite,[1,1]exelib/lb,[1,54]rsx11m.stb

  Substitute the directory that readwrite.mac is located in for the [200,200] on the first mac line above.

  The above MAC and TKB command lines work fine, but Hugh Sparks pointed out that they are overkill for this project. The commands above are the boilerplate command lines I use when doing more complicated privileged programs that make use of symbols in the exec. For this project, since we don't do anything with symbols, actually all you need to do is...

>tkb readwrite/pr=readwrite
>


 Anyway, it's privileged, and mapped to the IO page, so be careful when you modify it...mistakes here will give you an excellent opportunity  to learn how to use XDT to analyze a crash dump.

VAX/VMS Logical Name Hash Table Statistics

            Logical Name Hash Table Statistics

  VMS (and RSTS, and RSX, for that matter) provides a logical
name service, by which a string is mapped into one or more
"equivalence" strings. These logical names can be used in place
of a literal file or device specification to provide device
independence. This allows programs and data to be moved from one
device or system to another without requiring extensive
modification. They are also used to present the user with a
shorter, more manageable "abbreviation" of an actual device or
file specification.

  Everyone is, by now, pretty much sold on the value of logical
names in VMS. Many  years ago, at one of my larger sites, the
Technical Support group (the group responsible for production
system standards at this site) set a policy that ALL access to
files and devices in production systems would be by logical
names. This seemed like a reasonable idea to me, since the tape
and disk farm there tended to change configuration every few
months or so, and the users' systems were wont to drift around
from disk to disk, displaying a tropism for free space. A strict
logical name policy allowed them to change the locations of
files that were used by thousands of users without causing any

problems.

  When we started out, we elected to create any logical names

used by more than a few individuals as system wide logicals,

created at system startup. This, we reasoned, would prevent

wasting processor time creating and deleting them as process

logicals each time a user entered an application. It would also

save space, since we would have one copy of each logical,

instead of hundreds. At that time, I made a cursory

investigation of the possible impact of such a tactic. All

logical names are allocated in Paged Pool. No problem there - we

would just make sure we SYSGENed it to be large enough. I also

found that the names were located by hashing, so I reasoned that

a large number of them should not affect the time required to

locate one, since hashing tends to be much less affected by the

number of items to be searched than, say, a linked list.

  But, that was several years ago, and a great many applications

have been written there since then. With thousands of logical

names being maintained, I thought that a more detailed look for

any potential bottlenecks in this process might be in order. An

examination of the data structures involved brought to light

some interesting facts. It turns out, that all of the many

logical name tables (both system and user provided) are, in a

sense, an illusion. Typically, we think of a table as a

structure that is used to locate and select something. Logical

names are located independently from the logical name table they

reside in - AFTER being located, the system checks what logical

name table they are in, to see if it has located the correct

logical name.

  Logical names are actually located by means of hash tables (do

not confuse logical name hash tables with logical name tables).

There is a system wide hash table used to locate shared logical

names - those accessible by more that one process. Each process

has its own process private hash table. The residency of a

logical name in a logical name table (eg, LNM$_PROCESS) can be

considered to be "virtual". The logical name is first located in

one of the hash tables, and after being located, its "residence"

in a logical name table is evaluated.

  A hash table is a "semi" random access technique for storing

and retrieving data. The principle behind it is simple - that it

is very efficient to access an entry in an array, given the

index of the desired item. In the real world, some elaboration

on the idea is necessary for two reasons. The first is that most

data do not have a simple numeric quality that can be used as an

array index, so it becomes necessary to generate one. The

second, is that, in most cases, the set of possible items to be

stored is much larger than the set that actually occurs, so

allocating space in an array for every possible item would be

very wasteful.

  The index into the table is generated by applying some

algorithm to the data, to produce an integer. Typically, the

algorithm "hashes" up the data, in an attempt to produce a

random distribution of indices. Since the table is smaller that

the number of possible items, it is inevitable that some items

will generate the same index (will "hash" to the same address,

thus causing a "collision").

  A method often used to resolve these collisions (and, indeed,

the very one used in the logical name hash tables) is to have

each entry in the hash table contain the listhead of a linked

list. As items are entered in the table, all entries that hash

to the same index are linked onto the same linked list. When

looking up an item, the hash address will direct you to the

appropriate listhead. The list can then be traversed to find the

item you want. This is referred to as collision resolution by

chaining.

  The efficiency of a hash table using collision resolution by

chaining is measured by how many "probes" are necessary to

locate an item. A probe is considered to be a comparison of an

item in the linked list with the search criteria to determine if

it is the one we are after. Thus, if we locate a linked list and

find the item we want is the first item in it, then we found it

with one "probe". If we hash to a location and discover that

there are no items in this linked list, then that is zero probes

- we know immediately that it is not in the table, with no

compares performed.

  My dusty old college Data Structures textbook pointed out that

the efficiency of a chained hash table is a function of two

things. One is the ability of the hashing algorithm to hash

items to a good random distribution of indices. An algorithm

that hashes to the same address for many different items will

cause the linked list for that index to be very long, and the

speed of access will degenerate to that of searching a linked

list. Testing showed that the one used for logical name hash

tables does a reasonably good (but, not perfect) job of

distributing different logical names over the hash table.

  The other factor affecting performance of a hash table is its

fill factor. The fill factor is the ratio of the total number of

entries in a hash table to the size of the table. For an

unsuccessful lookup (that is, the item we want is not in the

table) the number of probes tends to be exactly the fill ratio.

For a successful search, the number of probes is approximated by

1 + 0.5*fill factor.

  Armed with this knowledge, and a sinking feeling that the hash

tables on this system were very full, I wrote the LNMFILL

utility. This utility locates the hash tables, and travels down

their entries. For each entry in the hash table, it traverses

the linked list it points to. The utility counts slots used,

slots empty, and maximum and average lengths of the linked

lists. LNMFILL does its work in KERNEL mode, since it needs to

acquire the LNM MUTEX. It is compatible with VMS V5 and up as

well as VMS V4, since the call to SCH$LOCKR will acquire the

needed spinlocks automatically. All calculations are rounded off to

integral values, to make everything easier...

  A run of the utility on one of the systems in question

produced the following results...

   *  Shared Tables *

Total Listheads : 128

Free Listheads  : 0

Logical Names   : 2207

Ave. Chain Len. : 17

                   Top 16 Max. Chain Lens.

   157    154    151     68     64     58     54     54

     43      43      42     39     36     26     21     20

   *  Process Table *

Total Listheads : 127

Free Listheads  : 109

Logical Names   : 20

Ave. Chain Len. : 1

                   Top 16 Max. Chain Lens.

     2      2      1      1      1      1      1      1

     1      1      1      1      1      1      1      1

  The process private hash tables were in pretty good shape.

They had a fill factor of less than 1 (logical names/total

listheads), and the average chain length was 1. The maximum

chain length was 2, indicating that there was an occasional

collision, causing two items to hash to the same address. We

decided that the size of this table didn't need to be increased.

If necessary, it could be increased by SYSGEN parameter

LNMPHASHTBL. It is in units of table entries, and defaults

to 128.

  It was pretty clear that the hash table size for the shared

table needed to be increased, to get the fill factor down to

less than 1. This would cause fewer items to be linked onto

linked lists, and speed up both successful and unsuccessful

searches. Its size is controlled by SYSGEN parameter

LNMSHASHTBL, and it also defaulted to 128. This was too small by

over an order of magnitude, for this system. Fortunately, this

is not too "expensive", in terms of memory, to increase. We

raised it to 2048 - from .25 pages to 16 pages of paged pool.

After a reboot, the improvement was definitely noticeable.

   *  Shared Tables *

Total Listheads   : 2048

Free Listheads   : 1028

Logical Names   : 2203

Ave. Chain Len. : 2

                   Top 16 Max. Chain Lens.

   131    130    130     18     12     11      9      9

       9        9        9       8      8        8      8      8

   *  Process Table *

Total Listheads   : 127

Free Listheads   : 109

Logical Names   :   22

Ave. Chain Len. :     1

                   Top 16 Max. Chain Lens.

     3      2      2      1      1      1      1      1

     1      1      1      1      1      1      1      1

  This was much better, with a load factor of about 0.5, but,

the top three maximum chain lengths were surprisingly high,

about 131. A little inspection with SDA showed why there where

so many collisions at these three indices. Each job references

three logicals with the same names - SYS$SCRATCH, SYS$LOGIN, and

SYS$LOGIN_DEVICE. Since the hash address is a function of the

logical name, and since all JOB logicals are in the system

shareable hash table (since they are potentially accessed by

more than one process), each of these three hash to the same

three indices. There will be one entry per logged in user on

these three linked lists. In any case, the average chain length

was down to 2, so, for most logical names, access times were

much improved.

  If you have a small to medium site, then you probably do not

have to worry about changing the table sizes. If you have much

more than 128 logical names (the default hash table size), then,

you should consider increasing the size of LNMSHASHTABL and/or

LNMPHASHTBL, as needed to improve the speed of logical name

translations.

lnmfill.mar

  To use LNMFILL, just assemble, link, and run it. Use of

LNMFILL requires CMKRNL privilege...

$ MAC LNMFILL

$ LINK LNMFILL

$ RUN LNMFILL