FSC-0028

FwdSpec - A Collection of Notes on Moving Files in FidoNet



Preamble

Copyright 1988 Greylock Software, Inc.

  POBox 730
  Gt Barrington MA 01230
  
  FidoNet>1:321/202.0


Synopsis

  This started as a reverse-engineered technical description of the
  core operations of Ron Bemis' Flea program, and an attempt to
  formulate a new specification which is a more symmetric superset
  of that functionality.  Specifications for Mr. Bemis software is
  available with that software, which is not freely distributed.
  
  This document ONLY addresses the format of files transferred
  between systems.  It does not address configuration information,
  which is really an implementation specific issue.
  
  This is currently only a base for discussion, which should be
  carried on in the SOFTWARE (SDS) and FTSC conferences.


Distribution

  This document may be freely distributed, so long as it is
  complete.
  
  Comments should be directed to:
  
  Barry Geller:    266/12
  Tom Hendricks:   261/662
  Harry Lee:       321/202
  Rick Moore:      115/333


1  General

1.1  Existing Tools

1.1.1  FileFwd

  FileFwd is a program by Joe Keenan whose primary purpose is to
  move consistently named files on a routed, regular basis.  It is
  extremely useful for routing echomail packets through intermediate
  nodes without unpacking and re-packing at each of the stations.








Copr 1988 Greylock Software, Inc       1                           Dec 2, 1988




FwdSpec - A Collection of Notes on Moving Files in FidoNet



1.1.2  Flea

  Flea is a program created by Ron Bemis which is used to broadcast
  files in a manner similar to EchoMail.  It is the primary tool
  used by the FidoNet Software Distribution System.
  
  Specifications for the Flea program are ostensibly available from
  the author.


1.1.3  GlueFwd

  GlueFwd is a distributed document control system from Greylock
  Software that was considered and rejected for use by the FidoNet
  Software Distribution System.
  
  Unlike Flea and Tick, GlueFwd uses messages to contain the
  associated routing information.


1.1.4  Tick

  Tick is a program by Barry Geller, which performs approximately
  the same functions as Flea, but uses a unique associated
  information file format.


1.2  Basics

1.2.1  Associated Routing Information

  There are a number of problems associated with file routing,
  either point to point, or broadcast.  The basic problem is how to
  handle the associated routing information.  The approaches involve
  a spectrum ranging from information contained ONLY on the systems
  handling the files to carrying the information WITH the files
  being handled.
  
  In addition, there is the choice of how this information is to be
  conveyed.  The choices range from associated files, to messages.


1.2.2  Name Collisions


1.2.3  Larva - starting the process

  The "Larva" process is usually invoked by the user at the command
  line.  This is how a file is put in motion.  It creates the
  appropriate outbound .Fle files and the file attach information
  required by the given mailer environment.






Copr 1988 Greylock Software, Inc       2                           Dec 2, 1988




FwdSpec - A Collection of Notes on Moving Files in FidoNet



1.2.4  Flea - moving stuff along

  The "Flea" process is the one that moves the files along.  It does
  the following:
  
  Check the inbound for .Pre file, and process any that are
  releasable as you would a normal .Fle file
  
  Check the inbound for .Fle files, and process each as follows:
  
  Parse the .Fle file, making sure its associate file exists, it
  comes from a valid source, and that it is not a pre-release.  If
  any of those conditions are violated, the file is renamed either
  to .Bad or .Pre.
  
  If all is well, move the file to the appropriate path associated
  with the area, and, if possible, update the FILES.BBS file.
  
  Using a Larva-like process, send the file along to any nodes in
  your echo list that have not seen the file.
  
  A Flea process is generally run whenever inbound mail is received.


1.3  Nomenclature

1.3.1  [Required]


1.3.2  {Optional}


1.3.3  Address: {Domain>}{Zone:}Net/Node{.Point}

  In the context of Flea 2.x, only Net/Node style addressing is
  supported.


1.3.4  Dates


2  New Forwarding Format (TICK)

2.1  General Goals

2.1.1   Removing order dependency

  The current structure of .Fle files is very order dependent.  In
  some cases, .Fle file lines have verbs, in others, they do not.  
  Presumably, Flea proper will have problems processing lines beyond
  the description that are not in the proper order.
  





Copr 1988 Greylock Software, Inc       3                           Dec 2, 1988




FwdSpec - A Collection of Notes on Moving Files in FidoNet



  This weakness should be eliminated, essentially by insisting on a
  verb per line, which makes possible free-form parsing, eliminating
  order dependency.  Within some groups of entries with the same
  verb, order dependency may be required.


2.1.2   Limiting the type of information contained in a given datum

  Flea 2.x very often carries different types of information on a
  given line.  While on the surface, this seems like an economical
  way to do things, it can lead to complications later on.
  
  Therefore, it is a general design goal to keep the type and use of
  a given datum associated with a given verb very clean.


2.1.3   Removing Case Sensitivity

  Flea is currently very case sensitive.  Software should be soft.
  
  An argument has been made that case sensitivity is a protection
  against bad files being inserted into the system.  If someone
  wants to generate a trojan horse, they will need passwords (the
  primary protection), and in all likelihood would use some sort of
  Larval tool to generate it anyway.  Case sensitivity makes it
  slightly more difficult for a developer to "enter the fray".


2.1.4   Removing Inconsistent Colon Usage

  Flea currently is haphazard in its usage of colons after verbs. 
  Colons should be made optional (or eliminated) on all verbs.


2.1.5   Optional Multiple DESC lines

  Flea currently supports a single description line, which is
  additionally position sensitive.  By creating a DESC verb, the
  position sensitivity can be eliminated, and multiple DESC lines
  can optionally be supported.
  
  At the current time, .Tic files use the DESC verb, but multiple
  DESC lines are not permitted.  Minimal compliance will be to
  handle one; multiple lines will be addressed later.


2.1.6   App (Application) line support

  In general, all mechanisms in FidoNet should allow for
  growth/variation by other developers in a non-harmful manner.
  
  In the case of Flea routing files, an APP verb with non-specific
  data should be provided for.  For example, let's assume that UPCL
  supports some sort of a "return receipt" functionality - when a



Copr 1988 Greylock Software, Inc       4                           Dec 2, 1988




FwdSpec - A Collection of Notes on Moving Files in FidoNet



  file hits you, so long as it's posted to your area, and with the
  sysop's consent (in the form of a configuration option), a message
  is sent to the Origin node.
  
  This might be done as follows:
  
  APP GREYLOCK Return-Receipt
  
  The "Greylock" sub-verb would keep APP conflicts from occurring.  
  
  Processors other than UPCL would pass the line through to any
  rebroadcast .Tic files intact.  (In fact, so would UPCL.)
  
  App lines, taken as a group, are order dependent.  A Tick
  processor should output App lines created during forwarding in the
  same order they read them, and if a Tick processor creates new App
  lines, they should be added to the end of the existing App line
  list.
  
  Once the majority of processors support a given APP functionality,
  it might be moved to the spec proper.
  
  Indeed, any lines with "unrecognized verbs" should be passed
  through intact, and in the order encountered.


2.1.7   Use of PATH construct rather than sby kludge

  Seenby information is more easily digested by humans (and
  programs) if it is sorted.  Unfortunately, such sorting removes
  the ability to use it for both seenby, and path information as it
  is in Flea 2.2.  In addition, the mechanism used by Flea 2.2
  precludes tiny seenby's, or Zone gating.
  
  Therefore, a PATH construct, much like an EchoMail PATH line
  should be used, instead of the current mechanism.  Once again,
  order dependency should be discouraged.  Within a group of path
  lines, obviously, order is important.


2.1.8   Multiple Sby's per Sby line

  The current seen-by construct, with one seenby per line, with the
  word seen-by required on each line is hideously inefficient.
  
  This should be changed to mimic echomail's seen-by handling, where
  multiple seenby's are contained on each line, up to 78 or so
  characters worth.
  
  A possible reason to keep the seenby down to a single entry per
  line is if information on how and when that node got the file is
  to be included.  While this might be worth considering, it will
  add considerable weight to the .Fle file.
  



Copr 1988 Greylock Software, Inc       5                           Dec 2, 1988




FwdSpec - A Collection of Notes on Moving Files in FidoNet



  At the current time, Tick files are assumed to have one seen-by
  per line.


2.1.9   Full (Optional) Domain, Zone, and Point support

  In order to allow for the future growth of the network, and
  interactions with other networks, addresses should be able to
  contain a fully qualified FidoNet address:
  
      Domain>Zone:Net/Node.Point.
  
  Further, given that many authors' primary machines are points, the
  result is as shown in the sample above: completely unknown
  addresses appearing in the .Fle files.
  
  Of course, these should not be required, but used as necessary.
  
  At the current time, Domains are completely unsupported, and
  should not be used.


2.1.10  Different extensions to avoid problems with Opus Style Outbound

  The extension .Fle was chosen because it leads to some expedient
  side effects in the form of file truncation/elimination by Opus or
  Binkley when the files reside in the outbound directory.
  
  On the other hand, both Opus and Binkley explicitly specify their
  outbound areas should be used ONLY for that.  A number of
  Binkley/Opus developers have expressed concern with this problem.
  
  For this, and other reasons, .Fle files should be given a new
  extension of some sort, one that is not closely related to the
  commonly used routing/message file extensions.  In addition,
  rather than the three divergent extensions now used by Flea (.Fle,
  .Bad, and .Pre), any and all extensions used by file routers based
  on this technology should use extensions that are more closely
  grouped.
  
  As an ancillary note, the FTSC should consider a "File
  Specification Pattern Registry".  This would not be limited to
  network tools, and it would not be an indication of ownership, it
  would simply be a reference.


2.1.11  RFC-822 Format

  It might make some sense to consider using an RFC-822 compatible
  format for these files.  In a future version of this document,
  I'll detail this possibility.
  
  It would also be nice from the point of view of implementing a
  similar system on UseNet/Internet flavored systems.



Copr 1988 Greylock Software, Inc       6                           Dec 2, 1988




FwdSpec - A Collection of Notes on Moving Files in FidoNet



2.1.12  Valid pairing of associated info file and file proper

  We need a mechanism to insure that the primary file and the
  associated information file are a valid pairing.
  
  Consider the following scenario ...
  
  System allows overwrites.  A file and associated .Tic arrive. 
  They are, for whatever reason, not processed.  A file by the same
  name comes in.  The pair is no longer valid, but given current
  technology, it would be passed along.


2.2  Considerations

2.2.1  Up and downness

2.2.1.1  Single Uplink


2.2.2  Table driven duplicate elimination


2.2.3  Mapping between distribution and on-line organization

  There is a problem in the current implementation in that the local
  organization of a system tends to defeat the duplicate catching
  aspects of the system.
  
  I.E., the SDS currently sends out ALL FidoNet files in one
  "channel".  Many systems move files of this category or that to
  unique directories.


2.2.4  Many features are intended for local optional implementation

  Many of the features in this specification obviously affect how
  individual sysops run their systems.  As such, these features
  should be optionally supported by each sysop, although the
  information should be passed through the associated information
  file regardless of whether or not they support the feature.


2.3  Schematic of .Tic file

  Area{:} [AreaName]
  {Release{:} [Time]}
  {Replaces{:} [FileName]}
  File{:} [FileName]
  DESC{:} [Description]
  {DESC{:} [Description]}
  {Size{:} [Bytes]}
  {Date{:} [FileDate]}
  {CRC{:} [Calculated CRC-32 (in hex?)]}



Copr 1988 Greylock Software, Inc       7                           Dec 2, 1988




FwdSpec - A Collection of Notes on Moving Files in FidoNet



  Origin{:} [Address]
  From{:} [Address] [Pwd]
  {Created{:} [Program Banner]}
  Seenby{:} [Address] {Address} ...
  {Seenby{:} [Address] {Address} ...}
  {APP{:} [Application Specific Information]}
  Path{:} [Address] {Address} ...
  {Path{:} [Address] {Address} ...}
  
  
  Note this file is NOT order dependent.  Some of the newer features
  are more for discussion than anything else.


2.4  Nomenclature and Rules

2.4.1  Address Format: Zone:Net/Node{.Point}


2.4.2  Don't Barf on appended or unknown stuff

  Lines that are unrecognizable (i.e., non-existent or non-supported
  verbs) should be passed through untouched.
  
  Lines that have additional data beyond the required data
  (separated by whitespace) should not cause the system to fail,
  although it is obviously difficult to pass this information
  through.


2.4.3  One or zero items of a given type unless otherwise specified


2.4.4  Simple ASCII Alphabet


2.4.5  Unix Date Time Formats

  All times are expressed as a long decimal in Unix format - the
  number of seconds since 1970.


2.4.6  [Required Data]


2.4.7  {Optional Data}


2.5  Detail

2.5.1   App [Ref] {Info}

  This is a "pass through" line to allow developers some room for
  development without breaking other developer's work.



Copr 1988 Greylock Software, Inc       8                           Dec 2, 1988




FwdSpec - A Collection of Notes on Moving Files in FidoNet



  
  An APP line should have the following form:
  
    APP [AppRef] {App Information}
  
  or
  
    APPLICATION [AppRef] {App Information}
  
  Application lines should have their order preserved, and
  applications adding lines should do so at the end of the existing
  application list.


2.5.2   Area [Name]

  Area names should probably be limited to 8 characters, with
  alphabet restrictions, to simplify their implementation.
  
  This is a mandatory line, and only one should exist in the file.


2.5.3   Author [Name]

  This is an item for discussion.


2.5.4   CRC [Decimal CRC Value]

  As .Fle files stand, it is possible to "slip something in" to the
  pipe, particularly if .Fle files are processed only once in a
  while as opposed to after each inbound call.
  
  A number of the proposed (and optional) features here provide
  safeguards against this.  Specifically, computing the file CRC,
  and preserving the original file date and size in the .Tic file.
  
  This has some value as a verification tool, without the legal
  encumbrances of PKSCrypt, etc.
  
  This probably should be a CRC-32 value.  This would also closely
  follow some of the ideas that are being considered for echomail
  processing.
  
  This is currently a point for discussion.  It probably should be a
  mandatory field.


2.5.5   Created [Program Banner]

  This should contain some program identification information of the
  program that generated the attach information.
  




Copr 1988 Greylock Software, Inc       9                           Dec 2, 1988




FwdSpec - A Collection of Notes on Moving Files in FidoNet



  There might be some standard format for the first part of this
  line, allowing for variant information after this.
  
  This is an optional line.


2.5.6   Date [Date/Time of creation]

  This is a check for valid file pairing between the associated
  information file and the primary file.  It is the file date stamp
  of the primary file in Unix format.


2.5.7   Desc [File Description]

  This is a description of the file.  There is as yet unspecified
  length restriction on this line.
  
  At the current time, exactly one of these lines should appear in
  the Tick file.
  
  In the future, more than one line may be supported.


2.5.8   Dest [Address]

  This is related to Route (qv)


2.5.9   Encrypted [PKS Key]

  Read the section on "GARBLE", and change it as follows:
  
  The file is initially encrypted using a PKS style encryption. 
  This would be the ONLY time the file is encrypted.  The FTSC or
  someone would have to collect a list of valid public keys of
  authors (and probably eventually everyone).  The file would then
  be of "known-quality", or at least from a known source.  The key
  would be included in the .Tic file for ease of operation.
  
  The ramifications of this are considerable.  First off, PKSCrypt
  is something the spook types in the world are bothered by. 
  Secondly, the source is not available, and the program does not
  work on some machines (i.e., my 386.)  Large keys would probably
  have to be used so a large number of possible keys will exist,
  which means considerable encryption and decryption processing
  time.  Finally, there is the question of a "Key registry", and how
  you verify them.
  
  I am not sure if this and Garbled are and/or or either/or.







Copr 1988 Greylock Software, Inc      10                           Dec 2, 1988




FwdSpec - A Collection of Notes on Moving Files in FidoNet



2.5.10  File [FileName]

  ONLY a filename (no path information) is contained on the FILE
  line.  No wildcards.
  
  Exactly one of these lines must exist in a Tick file.


2.5.11  From [Address]

  This is the address of the system sending the file on the current
  leg.


2.5.12  Garbled

  This is really just a thought for consideration than anything
  else.
  
  If this is present, the file referenced by the .Tic file is
  assumed to be archived (we'd have to address the issue of
  "deviant" archivers") by an agreed upon password between the
  sender and the sendee.
  
  The ramifications of this are considerable.  It would mean that
  individual archives would need to be created for any node so
  protected, which would need to be deleted after sending.  This
  implies a considerable expenditure of time and resources to create
  and store these archives.


2.5.13  Log [Comment]

  This is another one for consideration.  Any such lines would be
  displayed on the console and/or the system log.


2.5.14  Magic [FileName]

  This is food for thought.
  
  In order to resolve and standardize version numbering in file
  names, and magic file names, this might be used to distribute a
  "magic file name" with a given file.
  
  More than one of these lines might exist.


2.5.15  Origin [Address]

  Where the file originally entered the system.






Copr 1988 Greylock Software, Inc      11                           Dec 2, 1988




FwdSpec - A Collection of Notes on Moving Files in FidoNet



2.5.16  Path [Address] {Arrival}

  Path lines are, among themselves, order dependent.  However, they
  need not be contiguous.
  
  The current path specification allows for only one address per
  path statement.
  
  It might make sense to leave it this way, and add an "Arrival
  time", which would be the time the file was processed.
  
  I.E., the file would start out with the path for this node and the
  next node with the time of creation.  When it gets to the next
  node, he changes his time to the time of processing, and puts out
  a similar line for the node(s) he sends to.


2.5.17  Pw [Password]

  This is the password between the sender and the sendee.  This
  password is not case sensitive.
  
  Exactly one of these lines must exist in a Tick file.
  
  It would be nice to have some method of password securing that did
  not require the password to be exchanged in clear text.


2.5.18  Release [DateTime]

  This is an optional line used to contain a Unix Date Time (seconds
  since 1970) of the release of the file.
  
  The handling of this is really murky as far as I can tell.  A
  brief digression into "political structures."
  
  Let's consider the case of the SDS.  In SDS, it has generally been
  assumed that ONLY nodes that are a part of the SDS get their files
  using Flea/Tick technology.  However, whether it is aware of it or
  not, this is not the case.
  
  Here's what I think was intended: a file comes in with a
  Pre-release time set.  That is the time at which the file is moved
  to the publicly available area.  I am not sure whether it is
  passed along the chain until that date, or if it is simply not to
  be made "publicly available" until that date.


2.5.19  Replaces [FileName]

  Only a filespec, no path information, is contained on this flavor
  line.
  




Copr 1988 Greylock Software, Inc      12                           Dec 2, 1988




FwdSpec - A Collection of Notes on Moving Files in FidoNet



  A REPLACES line is used to optionally (at each given node) dispose
  of older versions of the file being sent out.  For instance,
  Binkley releases are named:
  
  BEXE_XXX.Arc
  
  Assuming the next version of Binkley was 2.10, and assuming
  REPLACES was enabled for the given area, the file named on the
  REPLACES line would either be erased or moved if found.
  
  I.E.:
  
  FILE BEXE_210.Zoo
  REPLACES BEXE_*.Arc
  
  If these lines are encountered, and replacement is allowed, and
  BEXE_200.Arc was found, it would, in some way, be removed from the
  access directory.
  
  Wildcards should be allowed, but should also be used with care.
  
  Multiple REPLACES lines should be allowed.


2.5.20  Route [Address]

  This is just thinking out loud.
  
  These would have to be order dependent.  They would be set up at
  the point of creation, and there would have to be agreements all
  along the way.
  
  A political nightmare, but very useful in a corporate environment.
  
  Collisions are a very real problem here.


2.5.21  RtRcpt {Address}

  This is an item for discussion more than anything else.  It would
  be nice to have a means to find out how far your files have
  moved.  On the other hand, there are significant Policy type
  considerations for such a functionality.
  
  If the optional address is omitted, the ORIGIN is used.


2.5.22  Seenby [Address] {Arrival}

  The current seenby specification allows for only one seenby per
  line.
  





Copr 1988 Greylock Software, Inc      13                           Dec 2, 1988




FwdSpec - A Collection of Notes on Moving Files in FidoNet



  Seenby's are NOT order dependent.  Seenby information is more
  useful in "alphabetical" than encountered order, although it is
  not a requirement.


2.5.23  Size [File Size in Bytes]


2.5.24  Source [Address]

  Where the file actually came from.
  
  This is a point for discussion.  Let's consider the SDS again.
  
  In theory, SDS is a controlled system.  Files are only supposed to
  enter it from a very limited subset of FidoNet.  Currently, the
  Origin is the location the file was "launched" from, a very
  different thing than the author's address.
  
  The Source address, if present, is the address of a primary system
  used by the actual author.
  
  For instance, consider Binkley.  Binkley is supposed to enter the
  system at the region 16 SDS node, although it is written by nodes
  that do not participate in SDS.


2.5.25  Topo {Address}

  This feature, if enabled, can be used to generate a topology
  report for the area specified to the given node.  If no node is
  specified, the report should be sent to the Origin node.


2.5.26  Unidentified Verb Handling

  Lines with unrecognized verbs should be passed through.  Order is
  a critical issue here.  Unknown lines should be output in the same
  order they were input.


2.6  Feature Table

Feature                  Status       Count
                                  
Area [Name]                               1
File [FileName]                           1
Path [Address]                          >=1
Created [Text]                          0-1
From [Address]                    
Origin [Address]                  
SeenBy [Address]                  
Path [Address]                    
                                  



Copr 1988 Greylock Software, Inc      14                           Dec 2, 1988




FwdSpec - A Collection of Notes on Moving Files in FidoNet



                                  
                                  
                                  
                                  
                                  
                                  
                                  
                                  
Unidentified Verbs                


2.7  TK123456.Tic (Updated and amended slightly from Barry's Orig)

  Area TICKTEST
  File TEST.TXT
  Desc This is the file description Line!
  Origin 1:266/1
  From 1:266/13
  Created by TICK v1.00 - Copyright (C) 1988 by I. Barry Geller
  Release 59000000
  Path 1:266/21
  Path 1:266/13
  Path 1:150/1
  Seenby 1:266/21
  Seenby 1:266/13
  Seenby 1:150/1
  Pw TESTPW


2.8  Notes

2.8.1  The primary file should be sent before the associated file

  The actual file should be sent before the associated information
  file.  Consider this was not done in the following scenario:
  
  Associated file sent
  Primary file partially sent - session fails
  System processes associated files, and fails to find last primary
  During next session, primary is sent, with no associated

















Copr 1988 Greylock Software, Inc      15                           Dec 2, 1988