FSC-0024 - A Proposal for a Type-3 Mail Bundle - Oliver McDonald

            Notes on Type three bundlers.

                 The first  important note is that without Wynn Wagner's
            work on FTSC-0014, none of this would have come to fruition.
            I owe him a great debt in this area, as well as the debt for
            Opus itself that got me into this.  Thanks Wynn.

                 Type 3  bundlers offer  opportunities for new levels of
            sophistication in mail processing.  As the first step Aurora
            Computer Technologies plans to provide the minimum specified
            by the  existing Type 3 bundle specifications with one minor
            addition.  This addition is the inclusion of the features of
            ReMapper.   This addition  is not  a required  inclusion for
            other software authors producing a Type 3 bundler.

                 To sum up, standard required features are:

              Must be able to create and unbundle Type 2 Bundles (See
              Must be able to create and unbundle Type 3 Bundles (See
              Must be able to Toss EchoMail from either Type 2 OR 3
              Must generate an update-required message for the sysop if
                the MinorVersion changes.
              Must generate an update-required message if it encounters
                a misc packet type it does not recognize.

                 Initial optional features are:

              May Duplicate the functionality of ReMapper.
              May automatically generate an F.Req. from source of bundle
                when the minor version changes.
              May generate an F.Req from source of bundle if it
                encounters a misc packet type it does not recognize.
              All error messages are placed in Matrix Mail messages to
                the Sysop.
              Will create outbound bundles on the fly from the inbound
                bundle.  Does not need to scan these messages.  Note, if
                this option is exercised it is IMPERATIVE that the areas
                are scanned prior to the unbundle process.

                 Type 3.0 proposal (preliminary)

                 This proposal allows for automatic updating of the Type
            3 bundle  to allow  for further  revisions and enhancements.
            Thus we will refer to it as a Type 3.0 with further versions
            becoming Type 3.1 etc.

                 All multi-byte  data forms (int/long) are considered to
            have the  MSB first and the LSB last.  Int is two bytes, and
            Long is four.
                      Bundle Header

              struct _BundleHeader {
                         struct _Address  B_Destination;
                         struct _Address  B_Origination;
                         unsigned nybble  B_BundlerVersionMajor;
                         unsigned nybble  B_BundlerVersionMinor;
                         unsigned byte    B_ProductCode;
                         unsigned byte    B_VersionMinor;
                         unsigned byte    B_VersionMajor;
                         unsigned long    B_CreationTime;
                         unsigned byte    B_Password[8];

                           Bundle Header Notes

                 This works out to 32 bytes which is a nice size to work

                 Here follows a short explanation for each field:

                 "B_BundlerVersionMajor/Minor"   provide   for   version
            numbers from  0.0 to  16.16, this  should be  enough for all
            except TJ.

                 "B_ProductCode" is  the  FTSC  assigned  product  code.
            This can  be used  to identify  just which  type  3  bundler
            created the  bundle; it should not be considered an error if
            this  is   unidentified,  and   need  not  be  processed  on
            unbundling but  MUST be included _correctly_ at the bundling

                 "B_VersionMinor"  is   a  version   number  that   will
            initially start  at Zero  and is  used to allow non backward
            compatible changes  to Type 3 bundles, such as header length
            change.     If  this   is  LOWER  in  the  bundle  than  the
            corresponding version  number in  the  unbundler  it  should
            abend.   It is  suggested that a short message be written to
            the Sysop  in NETMAIL  with as much information gleaned from
            the header  as possible.  (All  info  up  to  and  including

                 "B_VersionMajor", always 3.  This and all data prior to
            this point  is position  dependant and will never be changed
            in future Type 3.0 bundle revisions.

                 "B_CreationTime" is  an Unix  1970 based  creation time
            indicating the time/date the bundle was created.

                 "B_Password" is  a NULL padded character array that may
            contain uppercase  alpha bytes  or ASCII  digits.  It should
            not  contain   lowercase  characters,  punctuation,  control
            characters etc.  A maximum of 8 characters are significant.
                      Struct _Address

              struct _Address {
                         unsigned int     Zone;
                         unsigned int     Net;
                         unsigned int     Node;
                         unsigned int     Point;

              struct _AddressShort {
                         unsigned int     Net;
                         unsigned int     Node;

                      Bundle Footer

              Struct _BundleEnd {
                         Unsigned Byte    E_Packet_Type  /* Always 0 */

                           Bundle Footer notes.

                 All bundles  end with  this packet.  It is not optional
            and the  packet should  be  considered  grundged  if  it  is

                      Area header

              Struct _AreaHeader
                       Unsigned byte    E_Packet_Type  /* Always 1 */
                       Unsigned byte    E_NameLength   /* Actual bytes in E_NAME */
                       Unsigned Byte    E_Name[1]      /* Variable length field */

                           Area Header Notes

                 The area header packet marks the start of a sequence of
            messages destined  to the  same  message  area.    The  area
            indicated in  the Area Header will remain valid until either
            the end of the bundle OR another Area Header is encountered.

                 E_Name will  usually contain  the area name of the echo
            area that subsequent messages should go.  If E_NameLength is
            zero then  the subsequent  messages should  go  the  NetMail
            area.   Any messages  that occur  prior to  the  first  Area
            Header in a bundle should also go to the Netmail area.

                 The Maximum value for E_NameLength is 63.

                 E_Name is NOT null terminated.
                      Message Header

              Struct _MessageHeader {
                         Unsigned byte    M_Packet_Type  /* Always 2 */
                         Struct _Address  M_Destination  /* Final
                Destination */
                         Struct _Address  M_Origin       /* Where the
                message was entered */
                         Unsigned Long    M_CreationTime /* When the
                message was entered */
                         Unsigned int     M_Attributes   /* FTSC
                bitweights */
                         Unsigned byte    M_FromLength
                         Unsigned byte    M_ToLength
                         Unsigned byte    M_SubLength
                         Unsigned byte    M_From[1]
                         Unsigned byte    M_To[1]
                         Unsigned byte    M_Sub[1]

                           Message Header Notes

                 Every message  begins with a message header packet.  It
            should  be   created  by   the  system   where  the  message
            originated.   If there  are any intermediate stops along the
            way it  is the  responsibility of  the intermediate  systems
            along the  way to  maintain all  of the  information without

                 None  of   M_From,  M_To,  or  M_Sub  are  to  be  NULL

                      Message Body

              Struct _Text {
                         Unsigned byte    T_Packet_Type  /* Always 3 */
                         Unsigned int     T_ByteCount    /* # of bytes (< 0x1000 ) */
                         Unsigned byte    T_Data[1]      /* Variable length field */

                           Message Body Notes:

                 The message  body  is  considered  one  or  more  _Text

                 No _Text  packet will  be more  that 1000H  bytes  long
            (that's 4096  to the  terminally base  10 folks).  Of course
            there may  be a  near infinite  number of  _Text packets per
            bundle/Message header,  but you  are  absolutely  positively
            guaranteed that with the type 3.x method you will never need
            a buffer larger than 1000H.
                 In addition to ASCII values 20h through 7Eh (inclusive)
            the following  control codes  are legal for TEXT data.  Note
            that <CR> and <softCR> are NOT in this list, thus type three
            packers will eliminate spurious 0Dh's.

              <LF>                        0Ah            Forced
              <dle>      10h              Replicate

                 Other control  characters and  values 7Fh and above are
            symptomatic of a grundged message.

                 Replicate is  a  three  byte  sequence:  <dle>  <value>
            <length>.   For example  if a packet contains the bytes 10h,
            20h, 09h  it should  be expanded in the message body as nine
            <space> characters.

                 There is  no minimum  or maximum  line  length,  it  is
            assumed that  the reader  can supply  the  appropriate  line

                 One "line" of a message may cross from one _Text packet
            to another.


              Struct _EchoMailInfo {
                         Unsigned byte    EI_Packet_Type /* Always 4 */
                         Struct _EID      EI_Parent      /* Up message thread */
                         Struct _EID      EI_Child       /* Down message thread */
                         Unsigned byte    EI_SeenByCount
                         Unsigned byte    EI_PathCount
                         Struct _AddressShort            EI_SeenBy[1]
                         Struct _Address  EI_Path[1]

                           EchoMail notes:

                 The  EI_Child   and  EI_Parent   fields  are   used  to
            reconstruct the message thread.

                 Type 3 bundles uses binary seenby and path information,
            but should  convert to a normal seenby/path in the unbundled
            messages.   If  the  auto-rebundleing  is  used  it  is  not
            necessary  to   process  the   seenby's  into  the  unpacked
            messages.   It is suggested that if this approach is used it
            is HEAVILY tested prior to implementation, and that it still
            store the data somewhere for retrieval in case of unresolved
            dupe problems.

                      Cargo Info.

              Struct _PointInfo {
                         Unsigned byte    CI_Packet_Type /* always 05h */
                         Unsigned byte    CI_File Count  /* Number of files */
                         Unsigned byte    CI_FileName[1] /* Filenames (10 bytes */
                           Cargo Info Notes

                 The Cargo  info packet  will only  be found in a Type 3
            arcmail bundle  that contains  files.  It will always be the always
            second packet in a bundle.

                      Node info

              Struct _NodeInfo
                         Unsigned byte    NI_Packet_Type /* always 06h */
                         Unsigned int     NI_Flags       /* Flags for node */

                 This packet  is sent  to a  Type 3.x  node in the first
            bundle to  be sent  to that node.  The bundler should detect
            that the  node can  accept a Type 3.x bundle from a nodelist
            flag.   It will  automatically generate  this packet at that
            point.   Should a  type 3.x  bundle come from a node that is
            not identified  in the  nodelist as  type  3.x  capable  the
            bundler should  mark that  node  as  Type  3.x  capable  and
            generate a warning message.

                 NI_Flags, is  a bit  mapped field  that identifies  the
            characteristics of  the node.  Some of this information will
            duplicate that  information found  in the nodelist.  This is
            used as a check.

              Bit:       Meaning:

              0:                          Type 3
              1:                          Packing Protocol Bits.
              2:                             "        "      "
              3:                             "        "      "
              4:                          |Bits 3 & 4 are used together
              5:                          |to determine mail handleing.
              6:                          Ingate
              7:                          OutGate
              8:                          Net Host
              9:                          Net Hub
              A:                          <Reserved>
              B:                          <Reserved>
              C:                          <Reserved>
              D:                          <Reserved>
              E:                          Sent info
              F:                          Got Info

                 Bits 1,  2, &  3 are  used to determine how mail may be
            packed for this node (SEA, PKWare, ZOO)

              Bits 1 & 2 & 3:             Meaning:
              000:       No packing.
              001:       SEA Archive format.
              011:       PKWare LZW packing format.
              100:       ZOO Compression system.

                 Note that  these bits  were intended to be combined, so
            that if  a node  could handle  ZOO and SEA Archives it would
            set the  bits to  '101'.   Since by  definition  PKWare  can
            handle SEA  format Arc's  it is  considered standard  to set
            both bits for a PKWare capable system.

                 Bits 4  and 5  are used to determine how mail should be
            sent to this node (CM, hold, direct)

              Bit 4 & 5: Meaning:

              00:        Direct
              01:        Continous
              11:        Hold

                 The sysop  should be  able to  clear the  sent info bit
            should the status of his system change (ie becomeing an NC).
            Zone gates  may be  identified by  the fact that they are in
            Net 1  and they are both an ingate and an outgate.  The zone
            they are the gate for is identified by their node number.

                      MiscInfo (IFNA Kludge).

              Struct _MiscInfo {
                         Unsigned byte    MI_Packet_Type /* Always 06h -  FFh, assigned by FTSC */
                         Unsigned byte    MI_ByteCount  /* # of bytes of miscinfo */
                         Unsigned Byte    MI_WhoKnows[1] /* Misc Stuff */

                           MiscInfo Notes:

                 The Misc info packet(s) are the last packets associated
            with a  message, there  may be  more  than  one  in  extreme
            circumstances, but  this should  prove to  be unlikely.  The
            bundler  must   retain  any  information  in  these  packets
            unchanged if it is a routed message.
                 This is  a catch  all packet  that replaces the dreaded
            IFNA kludge.   It  is designed to only be used as an interim
            method.   At present  all IFNA  kludges are handled in other
            special purpose  packets, it  is foreseen  that any  further
            kludges will  be handled  only by miscinfo packets until the
            Type 3.x  bundle specification  can be  updated and coded to
            handle the new data.

                 MiscInfo packets  in the  range 80h  -  F0h  should  be
            preserved if  not understood.   It  should be  considered an
            error condition  to find a MiscInfo packet with in the range
            04h -  7Fh, this  range is reserved for future expansions on
            Type 3  packets.   Packets in  the range F1h - FFh should be
            unpacked AND  a warning  message should be generated.  These
            numbers will be used on an extremely temporary basis as they
            are designed  for ESSENTIAL  IFNA kludges  and will be added
            into the Type 3.x specification as quickly as possible.

                      Arcmail and Type 3.x

                 Type 3.x  bundlers support  arcmail much  the same  way
            that the type 2 bundlers do.  There are some enhancements in
            the arcmail  naming scheme  however, that help reduce system
            overhead for routed mail.

                 For arcmail  destined for  type 2 based systems the old
            reliable method of arcmail file naming will be used, IE:


                 Where NNNN  is a  four hex  digit net number, nnnn is a
            four hex  digit node  number, ww is a two character weekday-
            name identifier,  and #  is the  packet number for that day.
            Type 3.x  packers SHOULD  generate the  day  name  correctly
            rather than the OMMM 1.08 cyclic method.

                 Here  follows  a  suggested  Type  3.x  ArcMail  naming
            scheme, basically a modification of Roeland Meyer's original
            proposal.   I have  been made  aware that  Roeland has  some
            things  to   say  on   this,  but   there  seems   to  be  a
            communications break  between us, so until I can contact him
            I will stick with this.

                 For Arcmail  destined to  a Type  3.x system (with Type
            3.x bundles  internally), a  variation of  the method  first
            proposed by  Roeland Meyer  will be  used.   Here follows  a
            quick synopsis:

                 New address  specifier   (Re-edited by Oliver McDonald)
            This is  designed for the Type 3.0 Arcmail naming convention

              |  |  |  ||
              |  |  |  |`----> Incremental sequence number, base 10, max = 99d
              |  |  |  |           Starts at 00 and counts to '99' then wraps
              |  |  |  |           back to 00. No "Day-of-week" info.
              |  |  |  |           This is strictly to avoid bundle collisions.
              |  |  |  |           An 'empty' version of the bundle is kept
              |  |  |  |           around to help the router remember what the
              |  |  |  |           last sequence number was.
              |  |  |  |
              |  |  |  `-----> Flag to indicate bundle type
              |  |  |              Allowed values:
              |  |  |                 'All non-specified flags are reserved.
              |  |  |                 'U' - ZOO File bundle
              |  |  |                 'V' - ZOO Mail only bundle for a Point.
              |  |  |                 'W' - ZOO Mail only bundle
              |  |  |                 'X' - File bundle
              |  |  |                 'Y' - Mail only bundle for a point.
              |  |  |                 'Z' - Mail only bundle
              |  |  |               For files with the 'Y' flag it is
              |  |  |                sent as per normal until it reaches the
              |  |  |                node specified by the arcname.  At this
              |  |  |                point the node will unarc the FIRST bundle
              |  |  |                in the arc, and read the Message Header,
              |  |  |                and then attach the bundle to the point
              |  |  |                specified.
              |  |  |               For File bundles if the files are to be
              |  |  |                forwarded, the node will unarc the bundle
              |  |  |                in the arc.  It will check the message header
              |  |  |                for address (match against name), and will open
              |  |  |                the Cargo Info Bundle, and attach those files
              |  |  |                to the destination.
              |  |  |                 Note: If the addresses do not match it considered
              |  |  |                an error to forward the files.
              |  |  |                 Note: The point address is not considered for
              |  |  |                matching purposes.
              |  |  |
              |  |  `--------> Node address,  base 36, max = 56,654d
              |  |                 Allowed values: '000' to 'ZZZ'
              |  |                 This is the Node part of the destination
              |  |                 address of the bundle.
              |  |                 Special values:
              |  |                    '000'- Destination is the Net Host given by
              |  |                           ZNNN, not forwarded to any Nodes.
              |  |                    'ZZZ'- This a broadcast bundle to ALL Nodes
              |  |                           in the Net given by ZNNN, as
              |  |                           well as, the Net Host given by same.
              |  |
              |  `-----------> Net   address, base 36, max =  55,654d
              |                    Allowed values: '000' to 'ZZZ'
              |                    This is the Net part of the destination
              |                    address of the bundle.
              |                    Special values:
              |                       '000' - Destination is the ZoneGate given by
              |                               ZZ, not forwarded to any Nets.
              |                       'ZZZ' - This a broadcast bundle to ALL Nets
              |                               in the Zone given by ZZ, as well as,
              |                               the ZoneGate given by same.
              `--------------> Zone  address, base 36, max = 1,294d
                                   Allowed values: '00' to 'ZZ'
                                   This is the Zone part of the destination
                                   address of the bundle.
                                   Special values:
                                     '00' - Destination is the current ZoneGate.
                                     'ZZ' - This a broadcast bundle to ALL
                                            ZoneGates given by the NodeList, as
                                            well as, the ZoneGate given by same.

                 Note, Point  numbers are  specifically NOT  included in
            the file  name identifier.   There  were a couple of reasons
            for this;  first, we  wanted to  allow the  maximum range of
            Zone:Net/Node  numbers   to  be  available;  second,  anyone
            running points  should not  be doing  so on a minimal system

                 Note(2),  Special   bundle  names   (ZZZ  or  000)  are
            implemented optionally  by the  destination.  You should not
            assume that  it will  work.   A future  Type 3.x  spec  will
            include password protection for this.

                 The logic for providing for up to 100 packets allowable
            to a  specific node  is that  I have seen cases of a :CM Net
            Host generating  in excess  of 10 messages for a node in one
            day, and the next logical number is 100.

                 Should a  Type 3.x  destination fall  outside the range
            available to  the Type  3.x arcmail limits, then the bundler
            will fall back and use the Type 2 arcmail naming scheme.

                 Notes on Zone Gates

                 With type 3 bundles the ZoneGates software load is MUCH
            easier, all  it has  to do  is simply  forward the  Type 3.x
            bundle.   It is  suggested that  it may be VERY desirable to
            have Type  3.x bundlers  duplicate the  functionality of the
            Zone Gate  software.   At the  very  least  it  is  STRONGLY
            suggested  that   ZoneGates  upgrade  to  Type  3.x  capable
            bundlers as soon as they become available.

                 Notes for Type 3.x Developer's:

                 The latest  specs for Type 3.x will be available in the
            FTSC library  at all  times and at 1:342/1.  Developer's who
            register with  1:342/1 will  have upcoming changes netmailed
            to them  as they are confirmed.  Any upcoming change notices
            will have  a date  officially implemented.   This  date will
            always be in the future and should be considered an official
            release date of the new Type 3.x standard.

                 Every attempt  will be  made  to  allow  developer's  a
            reasonable time  period to  upgrade to the new standard.  It
            is important  that developer's  attempt to meet this date as
            these changes  are usually  NOT backward  compatible.   Code
            samples will  also be  F.Req'able from  1:342/1 as the magic
            file name  TYPE3x where x is the latest revision to the Type
            3 standard.   Should  a developer  be  unable  to  meet  the
            release  date   he  should  notify  the  FTSC  and/or  342/1
            immediately.   The release date is based on an estimate made
            by Aurora  Computer Technologies.  If there is a good reason
            the release  date will  be pushed  back, and  ALL developers
            will be  notified.  As the new Type 3.x standard will not be
            official until  the release  date no  developer will release
            his code early.

                 On that  subject, care should be taken by the developer
            to let  no new  format bundles escape his beta test systems.
            There are  a couple of approaches we recommend for this, the
            first is  to have  beta test  versions only generate the new
            format bundle  for  specific  zone:net/node  addresses,  the
            second is  to set  up a  completely separate private net for
            testing purposes.

                 Release method:   Since  the bundler will automatically
            spread news of itself with use, a simple zero effort release
            program may  be used.  As different versions of the Type 3.x
            bundlers will  require different operating environments, you
            should try  to get  your bundler  made  available  on  Echo-
            BackBone and  Echo Hub  systems.  The reasoning behind this,
            is that  it is  from these systems that the existence of the
            new bundler  will become  common knowledge.  The other place
            to send  it would  be the  Zone, Regional, and maybe Net Co-

                 Future of Type 3

                 Since the  Type 3  format proposed  provides for  a new
            level of  information exchange in Matrix mail I provide here
            a few advance hints of what is planned.

                      AutoEcho built in.

                 Replace AutoEcho/AreaFix with automatic security.  This
            security is  such that  the hub  will not  need  to  pick  a
            password and send it in netmail to the downstream node prior
            to the downstream node requesting echos.

                   Instead, the downstream node will request an echo, at
            which point  the  Hub's  bundler  will  generate  a  netmail
            message to  the Hub  Sysop.  Now the hub Sysop may decide to
            give it  to him.  If he does, he simply tells his bundler to
            start sending it downstream to him.

                 Now since  this last  paragraph  has  already  confused
            people, I  will provide  a scenario with names.  Here in Net
            342 we  have our  NEC as  Brian McCullough (BDMc for short),
            and our  REC is  Steve Barnes (SB).  We have BDMc requesting
            the echo NET_DEV from SB.  The sequence is as follows:

              BDMc requests NET_DEV.
              BDMc's bundler sees this and generates the echo request
              This packet is bundled and sent to SB.
              SB's bundler finds the bundle.
              SB's bundler sees that BDMc is authorized to have that
              SB's Bundler generates an acknowledge packet and starts
                sending the echo to BDMc.
              BDMc's Bundler gets the acknowledge and sets up the area.
              BDMc's Bundler will use the password it was sent for
                future requests.

                  If there  was a  problem with  access to  the requested
            echo Steve Barnes would have received a NetMail message from
            his bundler  and he would be able to make a decision at this
            point.  Other than that he need not even be in the country.

                 Minor details  on this,  the Hub (or upstream node) can
            specify levels  of  permission  for  this  autoecho  request
            process, and deny certain echos to certain downstream nodes.
            If a  downstream node  requests a  denied echo, the upstream
            node's bundler  will again generate a netmail message to the
            Hub Sysop informing him of what happened.

                 This  will   probably  be  implemented  as  a  downward
            compatible  upgrade   with  the  request  for  new  software
            triggered by  the first  request for  a new  echo.  Note, if
            standard distribution  applies this  should never generate a
            request.  However as things do not always work that way, the
            automatic notification  and  optional  file  request  should
            solve any major problems.

                 The Future of the Aurora Type 3 Bundler

                      Fowarding of bundles that costs money.

                 All forwarded  bundles that  will cost  money  will  be
            marked as  HOLD unless  either the  receiving OR the sending
            node are  marked as  send-to or  accept-from  appropriately.
            All keywords  will be  valid in  these cases.    This  is  a
            completely backwards compatible change.

                      Forwarding Cost bundles from Points.

                 The forwarding of cost bundles from points will be done
            on the  basis of  a credit  that the  point has.  The credit
            will be  monitored in  the USER.BBS  file, with  the  record
            number corresponding  to  the  point  number.    This  is  a
            completely backwards compatible change.

                 Final Notes:

                 Final Note:  Would all those planning on writing a Type
            3.0 bundler  please contact me (Oliver McDonald) via NetMail

                 Final  Note(2):   There  are   already   some   planned
            extensions to  Type 3.0,  they will not be strictly required
            and will  not create a new VersionMinor number, but will add
            functionality, and  will when used require an update.  It is
            my feeling that if you are aware of these plans, you will be
            able  to  integrate  them  better  at  the  point  they  are
            "officialized".  It is not my desire to become the only Type
            3.x developer  out there.  It is merely my desire to be able
            to be  one of  them, and also to be able to make Type 3.x so
            attractive to all that everyone will want to run it.

                 Final Note(3):  Send Code (tm, Bob Hartman).

                 Final Note(4):  Convince me(tm) of suggested changes.


                 Thanks to  all the  people in  Net_Dev  who  have  made
            suggestions and comments on this proposal as I worked on it.
            Your comments  are appreciated (even those I have not used).
            I would like to especially thank the following people:

              Wynn Wagner III             FSC-0014 and support.
              Roeland Meyer               Work on ArcName routing.
              Randy Bush Suggestions and support.
              Brian McCullough            Sounding board and Cattle