OK, everybody make yourself comfortable - this is a long one....
Next I needed to write routines to read and write to the DEQNA and RQDX3. It sounds simple enough. I've written device drivers for RSX11 and VMS before. This time I won't even have an OS in the way. There are no rules! But, my previous drivers were all either software only devices (that is, using a device driver to get a QIO interface for my kernel mode routines), or simple devices that had a simple CSR and input and output data buffer registers. DEQNAs and RQDX3s are much smarter devices than that (for a while, I was thinking they were both much smarter than I was). They expect complex data structures to be set up in memory, and they DMA data in and out of them as they see fit, whenever they feel like it, with no rhyme or reason At least that's the way it seemed. But, I managed to puzzle them out.
The key for both of them was to just implement the minimum subset of their functions that I needed for this project. Both devices have complicated support for having queues/lists of multiple IOs and data buffers active at one time, with attendant complexity for managing them. This application doesn't need that - if it works faster than asynch RS232, it's a win, so I could omit a lot of code dealing with multiple simultaneously active IOs.
There was information about how to program the DEQNA in the DEQNA Ethernet Users Guide - EK-DEQNA-UG-001. And some of the info in the DELQA manual (EK-DELQA-UG-002) applied to the DEQNA, and parts of it were explained better there. If you choose to follow this twisted path to getting the DEQNA to work, you'll likely want to read both of them. There were also some clues to be found in the RUST-11 project by Ian Hammond. Another place I looked for examples was the DEQNA driver in 2.11 BSD (yes, I was so determined to get this to work I even read UNIX code). I had a look at the VMS and RSX drivers, but they were a little too...recondite...to help much. The PDP11 boot code contained in the DEQNA roms was of some help as well.
So, on to the DEQNA. Let's discuss some of the problems I ran into getting it to send and receive.. One pitfall I ran into here was figuring out why the DEQNA wasn't interrupting. I described this as well in Part 3. I tried a lot of things before it occurred to me that the default machine IPL after power on might not be below 4, the interrupt level. Code for this is in Part 3.
Some other unexpected problems - The DEQNA routines took a while to get working, since I didn't notice some things I wasn't expecting. To be fair, these were mentioned in the doccos. Two bits in the CSR (RI and XI) are cleared by writing a 1 to them. One bit in the CSR (IL) is considered set when it is 0 and clear when it is 1 (a curious definition of set and clear, nicht wahr?). And longword alignment of all the data structures is absolutely necessary.
Additionally, all reads and writes to device registers on the MicroVAX 1 have to be words or bytes - longwords aren't welcome there.
I also had forgotten that ethernet packets had to be a minimum length to be considered valid, so sending was a problem when I had them too short. These problems would have been apparent sooner, but with no OS, there were no friendly informative error messages - just a lack of results, or even worse, cryptic mis-results. When it doesn't work...change something and try again...again and again. Anyway, I padded the disk block request messages that were too short with some nonsense text, and those sends started working. The data messages all had a disk block in them, so they were plenty long already.
But enough about pitfalls (and pratfalls) - let's talk about what you actually have to do to send and receive on the ethernet using a DEQNA.
First. let's talk about the device registers of the DEQNA. There's 8 of them. How do we find them? The doccos say that the first DEQNA in a system has these registers located starting at 174440. That's the octal 16 bit IO page PDP-11 style address. That ain't gonna do any good here. But let's think about what it means. 174440 means it is located 14440 octal bytes into the IO page on a 16 bit PDP-11, which begins at 160000. We have to translate that to a 22 bit MicroVAX1 IO page address. According to the MicroVAX 1 architecture manual, the MicroVAX 1 IO page starts at ^X20000000 (bit 29 set and above). So, ^X20000000 + 14440 octal is ^X20001920. What could be simpler? If you have two DEQNAs (DEQNAae? DEQNAnen?), the second one will be at ^X20001950, but, I got my hands full on this project with just one, so we'll speak no more of a second DEQNA, OK? If you're the kind of person who needs two ethernet adapters on a MicroVAX, you're the kind of person who can figure out how to get it to work..
Anyway, to make things more interesting, the DEC engineers made some of these registers have multiple functions. If you do a simple read of the the first 6 registers (word registers, not longwords, remember?), you get the 6 octets of the ethernet address of the card, in the six low bytes of the words read.. This gave me something simple to do to check that I had found the card in the IO page.
15 0
+----------------------------------------------+
| reserved | address byte 1 | ^X20001920
+----------------------------------------------+
| reserved | address byte 2 | ^X20001922
+----------------------------------------------+
| reserved | address byte 3 | ^X20001924
+----------------------------------------------+
| reserved | address byte 4 | ^X20001926
+----------------------------------------------+
| reserved | address byte 5 | ^X20001928
+----------------------------------------------+
| reserved | address byte 6 | ^X2000192A
+----------------------------------------------+
Simple enough, nicht wahr? Six reads and you're done...
adrbuf: .blkb 6
movl #^X20001920,r6
moval adrbuf,r7
movl #6,r8
rloop: movb (r6),(r7)+
addl #2,r6
sobgtr r8,rloop
dump #6,adrbuf
halt
Here's what it looks like - tested on a MIcroVAX 1 that has a DEQNA with an address of 08-00-2B-26-2F-84
>>>B XQA0
ATTEMPTING BOOTSTRAP
08002B262F84
And we got the 6 bytes of the ethernet address.
While we're talking register addresses, let's save a little memorizing of hex numbers and define some symbols for them.
dadr1 = ^X20001920 ;address byte 1
dadr2 = ^X20001922 ;address byte 2
dadr3 = ^X20001924 ;address byte 3
dadr4 = ^X20001926 ;address byte 4
dadr5 = ^X20001928 ;address byte 5
dadr6 = ^X2000192A ;address byte 6
drbdll = ^X20001924 ;receive bdl addr low register
drbdlh = ^X20001926 ;receive bdl addr high register
dtbdll = ^X20001928 ;transmit bdl addr low register low
dtbdlh = ^X2000192A ;transmit bdl addr high register
dvec = ^X2000192C ;interrupt vector register
dcsr = ^X2000192E ;csr
OK, that's when you're reading them. When it comes to writing them, the first 2 registers aren't used. The third and fourth registers, at ^X20001924 and ^X20001926 are where you store the 22 bit address of the receive buffer descriptor list, whatever that is. Just kidding, I'll discuss it later. Also, writing the second of these registers is the "Go" signal for a read to start.
15 05 0
+-----------------------------------------------+
| low 16 bits of RBDL address | ^X20001924
+---------------------------+-------------------+
| |high 6 bits of addr| ^X20001926
+---------------------------+-------------------+
In like wise, the next two registers, at ^X20001928 and ^X2000192A are where you write the address of the transmit buffer descriptor list. More to come on that as well. As well, writing the second of these registers is the "Go" signal for transmitting a packet.
15 05 0
+-----------------------------------------------+
| low 16 bits of TBDL address | ^X20001928
+---------------------------+-------------------+
| |high 6 bits of addr| ^X2000192A
+---------------------------+-------------------+
The next register is read/write, at ^X2000192C. It's where you write the value of the DEQNA's vector. At power on time, it's 774, the standard diagnostic vector. We leave it at that (it's as good a value as any, in this case).
15 10 09 02 0
+---------------+-------------------------+--+--+
| Reserved | |RR|RR| ^X2000192C
+---------------+-------------------------+--+--+
RR means reserved. Reserved reads back as 0.
And finally, the good ole CSR.
15 0
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
CSR |RI|RR|CA|OK|RR|SE|EL|IL|XI|IE|RL|XL|BD|NI|SR|RE| ^X2000192E
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
RR still means reserved. Let's run down what the rest of these abbreviations mean - it'll help give you a false sense that you know what this card is doing...
RI - R/W - Receive Interrupt Request. Set when the DEQNA has fully received a packet. The Interrupt Service Routine must clear this bit, if it ever wants to run again. Treacherously enough, to CLEAR this bit, you have to write a 1 to it, which any sane person would think is the way to set a bit - but...not this bit, not here...
CA - RO - Carrier. This reflects the state of Carrier Sense. We don't use it.
OK - RO - Fuse OK. You know that fuse in the bulkhead of the Ethernet connector, that you've always wondered what the point of it was? This signal tells you if it's blown or not. No fooling. Now you know why it's there.
SE - R/W - Sanity Timer Enable. This timer can be used to reboot the whole danged machine if the Ethernet gets all hung up. Since the last thing I need is the system rebooting while I'm trying to get something done, I don't mess with this.
EL - R/W - External Loopback - This bit and IL are used to put the DEQNA into an assortment of loopback modes for testing. I don't need these - if Wireshark doesn't see traffic, I'll know there's a problem.
IL - R/W - Internal Loopback - see above explanation. This bit has the additional wrinkle of being set at boot and reset time, to prevent a troubled DEQNA from machine gunning the network. It has to be cleared to allow transmits.
XI - R/W - Transmit Interrupt Request - When set by the DEQNA, this bit indicates that a transmit has completed all the way. You have to clear this bit (note well - this is another treacherous bit of design, that is cleared by writing a 1 to it) to enable any more transmit interrupts.
IE - R/W Interrupt Enable - .Set this bit to enable transmit and/or receive interrupts, when XI and RI are set as well.
RL - RO - Receive List Invalid - When set, it indicates that the receive list is empty. It is set at power up and cleared by reset. We are only using one item receive lists, so this bit isn't as important as it could be.
XL - RO - Transmit List Invalid - Like RL, only for transmit.
BD - R/W - Boot Diagnostic ROM - The DEQNA has a boot/diagnostic ROM on board. Since it's written for the PDP-11, it's not much use here on a MicroVAX. I read it while testing, since moving it to a buffer was something I could get the card to do, to see if I was talking to it at all. There's a few gyrations to go through to get it to read it out - if you have any interest in doing that, may I recommend the DEQNA User's Guide?
NI - RO - Nonexistent Memory Interrupt - This gets set when you use nonexistent memory. A bunch of side effects happen. It's a bit of a mess. I recommend avoiding it, by being careful where you read and write.
SR - R/W - Software Reset - Set this bit to reset the DEQNA. Clear it when you want to come out of reset.
RE - R/W - Receiver Enable - When set it enables receiving packets. It is cleared by power up and reset.
OK, this list of complex and interacting bits should be enough to tell you, this ain't gonna be a matter of loading a data buffer and watching a ready bit, like a DL11 or a line printer. And we haven't even discussed setup of the card yet, which is it's own special challenge.
Let's talk about how you really talk to the card - buffers. Buffers are read and written from/to the card using DMA. Each buffer has a Buffer Descriptor that describes the buffer. The address of the Buffer Descriptors gets stored in the registers described above, so's the DEQNA will know where to get data to and from. Buffer Descriptors are kept in lists.
The format of the transmit and receive buffer descriptors is thus...for each buffer, there is a buffer descriptor like this...
15141312111009080706 0 Bit #
+-+-+---------------------------+
Flags | | | |
+-+-+-+-+-+-+-+-+-+-+-----------+
Descr. Bits |V|C|E|S|x|x|x|x|L|H|AdrHi6bits |
+-+-+-+-+-+-+-+-+-+-+-----------+
| Buffer Address Low 16 bits |
+-------------------------------+
| Buffer Length |
+-------------------------------+
| Status Word 1 |
+-------------------------------+
| Status Word 2 |
+-------------------------------+
Additional Buffer Descriptors )if any)follow
There are only two bits defined in the Flags word, and, irritatingly enough, they don't have official names - we'll call them bits 15 and 14. They, along with bits 15 and 14 in Status Word 1 are used as a handshake between the MicroVAX and the DEQNA about who's got control of the buffer. 14 and 15 values are...
15 14
1 0 DEQNA not yet using this buffer
1 1 DEQNA using this buffer
The Descriptor bits ...
V - 15 - Valid - when set, indicates that this descriptor is valid
C - 14 - Chain - when set indicates that this descriptor points to another descriptor instead of a buffer
E - 13 - End of message - (transmit descriptors only) - When set indicates that this descriptors buffer contains the end of the packet and it's good to be transmitted.
S - 12 - Indicates that this descriptor points to a setup packet
11:08 - undefined
L - 07 - Low byte only - (transmit descriptors only) - when set indicates that that this buffer ends on a byte vs word boundary
H - 06 - High byte only - (transmit descriptor only) - when set indicates that this buffer starts on a byte boundary.
You can have a big long list of descriptors, one after another in the list, describing lots of buffers, and can even indirect off to another list of them. but, we're not doing any of that here. One buffer at a time each for transmit and receive will be plenty for this project. To indicate that this is the last and only buffer descriptor in play, throw in a few empty longwords, so the DEQNA will know there aren't any more.
Note the the buffer length field is the size of the buffer, in words, as a negative number. Man, those hardware guys...
As well, to simplify things, I'll only transmit word aligned buffers with an even number of bytes, so we don't have to worry about setting Descriptor bits H and L
Wow, this is a lot of typing, and we haven't even discussed initializations and setup yet. And the RQDX3 is even worse than this. Well, can't stop now, let's reset and init the card. The card is in the reset state when it is powered on. We clear the whole CSR, which, among other things, clears the S/R bit, which takes us out of reset.
clrw @#dcsr ;reset the card
Then we set the bits we need for normal operation. This includes setting bit IL, which, annoyingly enough, clears it (which is how we need it).
movw #^X141,@#dcsr ;set RE, IE and IL (to clear it)
We set Receive Enable and Interrupt Enable, since here at the start we want to be able to pseudo receive the setup packet, and we want an interrupt when it completes. We clear IL by setting it, since we don't want any loopback going on.
Speaking of setup, it's time for that part of the movie. Setup consists of telling the card what addresses to enable. We only need to use two addresses. One is FF-FF-FF-FF-FF-FF - all one's, the broadcast address. This is used while we are looking for a load host. The other one is the legit address of the card, which for these examples is 08-00-2B-AA-BB-CC.
Setting this up on the DEQNA is...funny. You have to lay the addresses out that you are interested in, in a data structure with a less than obvious pattern. Then you create a transmit buffer descriptor that points to that structure as a buffer, and has bit 12 (the S (setup) bit in the above diagram) set in the address descriptor bits. When you signal the DEQNA to pick it up (by setting the buffer descriptor's address in the transmit descriptor registers, it DMAs it in and "pretends" to transmit it. Instead of putting it on the wire, though, it sets up the addresses in the structure as addresses it will use. Then it acts like it received it and you have to process it as a received packet before you can get to any real traffic. This was definitely designed by a hardware engineer instead of a software guy.
Anyway, here's an example of how that structure looks, for addresses 08-00-2B-AA-BB-CC and FF-FF-FF-FF-FF-FF. You have to rotate 'em and lay them in vertically. Additionally, the first column is all 0's. As well, even though we only have two addresses we care about, it's recommended that you fill in all of the possible 14 addresses with valid addresses - so we just duplicate the one non-broadcast address we are using. Complicated, nicht wahr?
1st set of 7 addresses
07 00 Bit #
+--+--+--+--+--+--+--+--+
|08|08|08|08|08|FF|08|00| 0 addr offsets
+--+--+--+--+--+--+--+--+
|00|00|00|00|00|FF|00|00| 8
+--+--+--+--+--+--+--+--+
|2B|2B|2B|2B|2B|FF|2B|00| 16
+--+--+--+--+--+--+--+--+
|AA|AA|AA|AA|AA|FF|AA|00| 24
+--+--+--+--+--+--+--+--+
|BB|BB|BB|BB|BB|FF|BB|00| 32
+--+--+--+--+--+--+--+--+
|CC|CC|CC|CC|CC|FF|CC|00| 40
+--+--+--+--+--+--+--+--+
|00|00|00|00|00|00|00|00| 48
+--+--+--+--+--+--+--+--+
|00|00|00|00|00|00|00|00| 56
+--+--+--+--+--+--+--+--+
2nd set of 7 addresses
07 00
+--+--+--+--+--+--+--+--+
|08|08|08|08|08|08|08|00| 64
+--+--+--+--+--+--+--+--+
|00|00|00|00|00|00|00|00| 72
+--+--+--+--+--+--+--+--+
|2B|2B|2B|2B|2B|2B|2B|00| 80
+--+--+--+--+--+--+--+--+
|AA|AA|AA|AA|AA|AA|AA|00| 88
+--+--+--+--+--+--+--+--+
|BB|BB|BB|BB|BB|BB|BB|00| 96
+--+--+--+--+--+--+--+--+
|CC|CC|CC|CC|CC|BB|CC|00| 104
+--+--+--+--+--+--+--+--+
|00|00|00|00|00|00|00|00| 112
+--+--+--+--+--+--+--+--+
|00|00|00|00|00|00|00|00| 120
+--+--+--+--+--+--+--+--+
It's madness, but, ya gotta give 'em what they want . Here's the MACRO to set up that buffer.
sbuf:
.byte 0, 08,^XFF, 08, 08, 08, 08, 08
.byte 0, 00,^XFF, 00, 00, 00, 00, 00
.byte 0,^X2B,^XFF,^X2B,^X28,^X2B,^X2B,^X2B
.byte 0,^XAA,^XFF,^XAA,^XAA,^XAA,^XAA,^XAA
.byte 0,^XBB,^XFF,^XBB,^XBB,^XBB,^XBB,^XBB
.byte 0,^XCC,^XFF,^XCC,^XCC,^XCC,^XCC,^XCC
.byte 0, 08, 08, 08, 08, 08, 08, 08
.byte 0, 00, 00, 00, 00, 00, 00, 00
.byte 0,^X2B,^X2B,^X2B,^X28,^X2B,^X2B,^X2B
.byte 0,^XAA,^XAA,^XAA,^XAA,^XAA,^XAA,^XAA
.byte 0,^XBB,^XBB,^XBB,^XBB,^XBB,^XBB,^XBB
.byte 0,^XCC,^XCC,^XCC,^XCC,^XCC,^XCC,^XCC
.long 0,0,0,0
sbuflen = . - sbuf
So, to make this take effect, like I've been saying, you have to sort-of transmit it. To do this, create a transmit buffer descriptor for it.
.align long ;make sure its longword aligned
sbufd: .word ^X8000 ;flags - it's valid and unused
.word ^X8000!^X2000!^X1000 ;address descriptor bits
;it's valid, setup and end of message
;and the high 6 bits of bufadr are 0
.word sbuf ;low 16 bit addr of setup buffer
.word -sbuflen/2 ;length of the setup buffer in words
ss1: .word ^X8000 ;status word 1
ss2: .word ^X8000 ;status word 2
.word ^X8000 ;flag of next buffer
.word 0 ;which is null
.long 0 ;some empty longwords...
.long 0 '...to indicate...
.long 0 ;...the end...
.long 0 ;...of the list
OK, we're getting there. There's one little wrinkle in the data structure above. Remember way back in Part 1, where I mentioned that we had to write all this as Position Independent Code? Well, just writing
.word sbuf
Will cause the assembly time value of SBUF to be deposited at that address. But we need the run time address of SBUF there, since we don't load at a normal address. Not a real problem - we add the lines
movaw sbufd,r6 ;load addr of buffer desc to r6
movaw sbuf,r5 ;get run time addr of xbuf
movw r5,bfd$w_loa(r6);store low 16 bits of addr in bd
These three lines get the relocated, run time address of sbuf, and truncate it to 16 bits as it is stored in the buffer descriptor (bfd$w_loa is set elsewhere to be 4, the offset to the low 16 bits of the address in the descriptor).
In order for setup to complete, like I've been saying, you have to pseudo transmit the setup buffer. But in order for it to complete all the way, you have to pseudo receive it. For that we need a receive buffer, a receive buffer descriptor, and set the two receive buffer descriptor registers on the DEQNA. This will get it ready to "receive" the setup buffer when we sort of send it.
We'll need a receive buffer....nothing special about that.
.align long
rbuf: .blkb 2000 ;longer than we'll ever need
rbuflen = . - rbuf ;record its length
Then we'll need a receive buffer descriptor, to describe it.
rbufd: .word ^X8000 ;flags - it's valid and unused
.word ^X8000 ;addr desc bits - it's valid, and the... ;...high 6 bits of setup buff are 0
.word rbuf ;low 16 bits addr of setup buffer
.word -rbuflen/2 ;neg length of the setup buffer in words
ss1: .word ^X8000 ;status word 1
ss2: .word ^X8000 ;status word 2
.word ^X8000 ;flag of next buffer
.word 0 ;which is null
.long 0
.long 0
.long 0
.long 0
Then, just like the transmit descriptor, fix up the address of the buffer.
movaw rbufd,r6 ;load addr of buffer desc to r6
movaw rbuf,r5 ;get run time addr of xbuf
movw r5,bfd$w_loa(r6);store low 16 bits of addr in bd
Then load the address of the descriptor in the receive descriptor register of the DEQNA, and clear the high 6 bits register (in that order). The manual says, load the low order 16 bits first, and then the high order 6 bits last, which is the receive GO signal to the card. All throughout this project, it's assumed that we stay within the 16 bit address limit, even though the QBUS on the MicroVAX 1 is 22 bits large - this is a complicated program, but not a big one, so the upper 6 address bits will always be 0.
movw r6,@#drbdll ;load low addr of setup bfd into csr
clrw @#drbdlh ;clear high word and start it all up
OK, now to finally actually tell the DEQNA to do setup. To pseudo transmit the setup buffer, we have to load the two DEQNA transmit buffer registers with the address of the transmit buffer descriptor. Again, we load the low order 16 bits first, and then the high order 6 bits last, which is the GO signal to the card to "send" the buffer..
movaw sbufd,r6 ;load addr of buffer desc to r6
movaw sbuf,r5 ;get run time addr of xbuf
movw r5,bfd$w_loa(r6);store low 16 bits of addr in bd
movw r6,@#dtbdll ;load low addr of setup bfd into csr
clrw @#dtbdlh ;clear high word and start it all up
Now we can just spin until the receive interrupt is done, which signals the end of the setup process. Nothing to it, right? Piece of cake so far. We set a flag word in the interrupt receive routine to indicate that this setup receive is complete. We just blbc loop on that flag word until it's set.
setwat: blbc setupdone,setwat ;spin until deqna init completes
Additionally, in the receive interrupt routine, for this setup and for normal receives, we reset the CSR to ^X141, reset the fields in the receive descriptor, and reset bit RI in the CSR (by writing a 1 to it), to get ready for the next receive.
Now that setup is done,. we can send packets out on the wire. Set up the transmit descriptor just like we did for setup, but don't set the setup bit (^X2000). In the buffer for a normal transmit, enter the 6 byte destination address, the 6 byte address we're using for the card, and a two byte packet type code. We're using 60-06 - that's reserved for DEC customers )and that's us!).
The buffer looks like this....destination address, source address, and two bytes of type. Then the message we want to send, followed by some fillter to make the packet long enough.
xmbuff: .byte ^XFF,^XFF,^XFF,^XFF,^XFF,^XFF ;dest addr
.byte 08, 00,^X2B,^XAA,^XBB,^XCC ;source addr
.byte 06,^X60 ;type
.ascii /Payload to transmit goes here/
.blkb 48 ;filler - payload has to be at least this long
xmbufflen = . - xmbuff
To receive, it's just like the receive we described during setup.
OK, now we can send and receive. In broad strokes, what the program does is broadcast a message to all stations (to address of FF-FF-FF-FF-FF-FF) looking for someone who can load the file we need. A normal VMS program running on the load host is listening for that, and replies that it can provide that file. The program running on the MicroVAX then sends packets that contain the number of the block it wants next. The program on the load host sends back a packet containing that block. The MicroVAX program writes that block, and then requests the next one. We'll talk about what it takes to write the data blocks via the RQDX3 in the next section.
