EDI

is a scary word to some! Like many of the standards developed around the same time, a simple concept got lost under the weight of diverse and sometimes competing interests. Standards try to be all things to all people and become complex as more special interests are accomodated. An entire industry sprang up around EDI, with specialized service providers such as VANs (Value-Added Networks) touting their capabilities. It reminds me of DCE (Distributed Computing Environment,) another fine idea which became so complex that relatively few companies ever got around to offering commercial products.

The underlying concept of EDI (Electronic Document Interchange) is quite elegant. Rather than exchanging mountains of paper (purchase orders, invoices, etc.) just exchange data files containing the requisite information! Of course, everyone has to agree on a standard format of these electronic documents and, as is commonly the case these days, the detailed specifications are only available from standards organizations. Don't expect to find free documentation on the web; you can expect to pay something like US$500 to ISO in order to get the documents needed to implement a solution.

A client of mine recently asked me to look into EDI solutions in order to send files to one of their customers. A quick search of the web took me to the home page of OBOE. Since this solution is written in Java and is Open Source, it seemed to be a good fit. Code portability is very important to me (platform independence) and I've recently become a convert to the Open Source approach. The robustness of applications like Apache and the utility of packages such as JacORB has not been lost on me.

Downloading the product for evaluation was a different experience than I'm used to on the web. You have to fill out an in-secure form containing your e-mail address and submit it. At some later time, the software arrives as an attachment to an e-mail message restating the license terms. They apparently have some kind of address filtering in place since when I used a free account I hadn't received a reply for three days. When I used the address at the client site, the software arrived within hours. This approach concerns me since it's not as open as has become common: I'm used to clicking a link on a site and saving the software right from my browser.

That being said, the software does work, albeit this is definitely not a project for beginners! The documentation is thin (granted, it's a work in progress,) and the few sample programs included with the package didn't come close to addressing the application I had in mind. I couldn't access the documentation supposedly included in the package and had to use the web site version. I didn't want to spend too much time figuring out the documentation and I have high-speed internet access from home anyway. Also, there are some additional Java packages required: see below.

The "binary" version of the package consists of the aforementioned documentation and sample programs as well as the core: a JAR file containing all the required classes. Installing the prerequisite products and adding the various JAR files to your CLASSPATH is standard practice. You have to import the com.americancoders.edi.* classes as well as com.americancoders.edi.x12.* if you plan on generating X12 documents. Other than that, development was fairly straightforward: I keep a browser window open to the documentation and a couple of X windows for firing up vi and java. I don't know how easy it would be to import everything into an IDE (Integrated Development Environment) since I find using one to be a chore.

Here are the required downloads:

Java Cryptography Extensions (JCE)
JavaMail
Java Activation Framework (JAF)
Apache xerces

The first three of these packages are available from the Sun web site. You can download xalan from the Apache site. In order to install and configure the requisite files, follow these steps:

create a new directory to hold the jar files (I use a subdirectory of my home directory named lib)
change to that directory
untar the downloaded file
add the following lines to your .profile:
JARDIR=jar directory CLASSPATH=$CLASSPATH:$JARDIR/oboe.jar:$JARDIR/jce1_2-do.jar:$JARDIR/mail.jar:$JARDIR/activation.jar:$JARDIR/xerces.jar export CLASSPATH
NOTE: If you cut directly from this page then your browser might split the CLASSPATH line above. Just edit it to make it all one line.
create a new file in your home directory called OBOE.properties containing the following line:
xmlPath: jar directory/xml/
NOTE: You can't use environment variables (like $HOME) in this file; a fully qualified path is required. Also, the trailing slash is essential.
logout and log back in in order for the change to take effect

You are almost ready to create your EDI application, but first we need to address the EDI document architecture. The diagram to the left shows the nested architecture of an X12 EDI document. Working from the outside in, the envelope contains an interchange header, one or more functional groups and an interchange trailer. A functional group consists of an optional header, one or more transaction sets and an optional trailer. A transaction set consists of a header, detail records and a summary (a trailer by a different name.) Each of these elements consists of one or more segments, which can be nested and which typically contain data elements.

The heirarchical nature of a document is significant insofar as the implementation of OBOE. It's very similar to the DOM approach and involves "walking the tree" in order to create, access and modify the document contents. Since DOM is typically used to parse a received document, it's in some ways the logical inverse of what I am going to describe, namely the creation of an EDI document. XML is also tightly integrated with OBOE so some experience in that area is almost essential for configuring new transaction sets. We'll defer that discussion until we dig down to the transaction set level.

In order to use OBOE to create a new EDI document, you'll have to include the following imports in your Java source:

com.americancoders.edi.*
com.americancoders.edi.x12.*

The first object you need to instantiate is the X12 envelope. Here's how you code it:

X12Envelope	env = new X12Envelope();

The next step is to set the fields in the interchange header. The following code snippet (taken from a functional program) performs the task:

Segment seg = Interchange_Control_Header.getInstance();
setElement( seg.getDataElement( 0 ), "0", ZERO_FILL, 0, MIN_LENGTH );
setElement( seg.getDataElement( 1 ), " ", BLANK_FILL, 0, MIN_LENGTH );
setElement( seg.getDataElement( 2 ), "0", ZERO_FILL, 0, MIN_LENGTH );
setElement( seg.getDataElement( 3 ), " ", BLANK_FILL, 0, MIN_LENGTH );
setElement( seg.getDataElement( 4 ), "1", ZERO_FILL, RIGHT_JUSTIFY, MIN_LENGTH );
setElement( seg.getDataElement( 5 ), fromID, BLANK_FILL, 0, MIN_LENGTH );
setElement( seg.getDataElement( 6 ), "1", ZERO_FILL, RIGHT_JUSTIFY, MIN_LENGTH );
setElement( seg.getDataElement( 7 ), toID, BLANK_FILL, 0, MIN_LENGTH );
setElement( seg.getDataElement( 8 ), transDate, 0, 0, 0 );
setElement( seg.getDataElement( 9 ), transTime, 0, 0, 0 );
setElement( seg.getDataElement( 10 ), "U", 0, 0, 0 );
setElement( seg.getDataElement( 11 ), "200", ZERO_FILL, RIGHT_JUSTIFY, MIN_LENGTH );
setElement( seg.getDataElement( 12 ), interchangeControlNumber, ZERO_FILL, RIGHT_JUSTIFY, MIN_LENGTH );
setElement( seg.getDataElement( 13 ), "0", 0, 0, 0 );
setElement( seg.getDataElement( 14 ), "T", 0, 0, 0 );
setElement( seg.getDataElement( 15 ), ":", 0, 0, 0 );
env.setInterchange_Header( seg );

For now, don't concern yourself with the setElement function; as you can probably tell, it controls how a string is inserted into a data element. What's more important is how I know what DataElement index corresponds to each field. OBOE is very familiar with the fields in the interchange and functional group layers. The following code will "walk" through the fields in a segment and display the salient information:

for( int i = 0; i < seg.getDataElementSize(); i++ ) {
	DataElement	elem = seg.getDataElement( i );
	System.out.println( i + ": " + elem.getID() + " (" +
	  elem.getDescription() + "): " + elem.getMinLength() + "/" +
	  elem.getMaxLength() + " = " + elem.get() );
}

I'll warn you again later that this loop only works on segments created by OBOE. Once you get to the transaction sets, you're going to have to cross-correlate to the XML definitions in order to determine the document structure and data elements.

Now that we've specified the interchange header fields and added the header to the envelope (env.setInterchange_Header,) we need to create a functional group. I use the following code:

FunctionalGroup	fg = null;
try {
	fg = X12FunctionalGroup.getInstance();
}
catch( Exception e ) {
	e.printStackTrace();
	System.exit( 12 );
}

Here you see a couple of idiosynchracies in the package. First off, the X12FunctionalGroup class name is inconsistent with the setInterchange_Header method of the X12Envelope class; it should be X12Functional_Group or setInterchange_Header should be setInterchangeHeader. A minor gripe to be sure, but it does point out that you have to keep a close eye on these implementation details. Secondly, this is one of only two methods I've used which is defined as throwing exceptions. As you know, with Java if a method indicates that it throws an exception then you either need to catch it or define your method as throwing the exception. Since there were only two anomolous methods, rather than defining my methods as throwing exceptions I just catch it here. Besides which, what it the real likelihood of this method throwing an exception? It's just instantiating a populated functional group.

As I mentioned, the functional group is pre-populated. The next code snippet gets a handle to the header and sets the value of some of the data elements:

seg = fg.getHeader();
setElement( seg.getDataElement( 0 ), "PT", 0, 0, 0 );
setElement( seg.getDataElement( 1 ), fromID, 0, 0, 0 );
setElement( seg.getDataElement( 2 ), toID, 0, 0, 0 );
setElement( seg.getDataElement( 3 ), transDate, 0, 0, 0 );
setElement( seg.getDataElement( 4 ), transTime, 0, 0, 0 );
seg.getDataElement( 5 ).setMinLength( 1 );
setElement( seg.getDataElement( 5 ), functionalGroupControlNumber, 0, 0, DATA_LENGTH );
setElement( seg.getDataElement( 6 ), "T", 0, 0, 0 );
setElement( seg.getDataElement( 7 ), "004010", 0, 0, 0 );

Here's another departure from the norm. For this particular project, the sample files we were provided showed the functional group control number as being minimal length, even though OBOE (and hence the standard) insists on a minimum length. This workaround was necessary to generate the files in exactly the specified format.

Before getting into the description of the transaction set, let's have a quick look at populating the functional group trailer. Here's the code:

seg = fg.getTrailer();
setElement( seg.getDataElement( 0 ), "1", 0, 0, DATA_LENGTH );
seg.getDataElement( 1 ).setMinLength( functionalGroupControlNumber.length() );
setElement( seg.getDataElement( 1 ), functionalGroupControlNumber, 0, 0, DATA_LENGTH );
env.addFunctionalGroup( fg );

The final statement adds the functional group to the envelope. As previously mentioned, we can add multiple functional groups to the envelope. My application was fairly straightforward so the body is complete at this point. The final step involves setting the fields of the interchange trailer. Here's the code:

seg = Interchange_Control_Trailer.getInstance();
setElement( seg.getDataElement( 0 ), "1", 0, 0, DATA_LENGTH );
setElement( seg.getDataElement( 1 ), interchangeControlNumber, ZERO_FILL, RIGHT_JUSTIFY, MIN_LENGTH );
env.setInterchange_Trailer( seg );

Now that the document is complete, it's time to generate the file which will be transmitted. Here's the code which accomplishes that function:

System.out.print( env.getFormattedText( Envelope.X12_FORMAT ) );

As can be seen, the creation of the interchange and functional group headers and footers is straight-forward. The same can't be said for the transaction set. There are two reasons for this: you have to explicitly create each element and you have to deal with the XML specification of the form.

Here's the code for creating a new (empty) transaction set:

TransactionSet	ts = null;
try {
	ts = TransactionSetFactory.buildTransactionSet( "867" );
}
catch( Exception e ) {
	e.printStackTrace();
	System.exit( 12 );
}

The transaction set factory can throw exceptions for a variety of causes:

if the OBOE.properties file cannot be found in the usual places
if the xmlPath doesn't exist or the directory to which it points doesn't exist
if the file 867.xml (in this case) doesn't exist
if errors occur while parsing the XML file
if certain DTD (Document Type Definition) files don't exist

The XML file name is created through the concatentation of the xmlPath, the argument to the buildTransactionSet and the string ".xml". The DOCTYPE definition in the XML file specifies transactionSetRules.dtd. A major limitation of the OBOE product is that the only document definition supplied is for a Request for Quotation. Since we needed to generate a Product Transfer and Resale Report, we had two choices: we could either purchase the definition from American Coders ($150/document) or create our own.

I chose the second option, primarily due to time constraints. In some ways, it's fortunate that the document type 840 XML file was provided since it contains just about every segment type imaginable. I was able to cut and paste the 840 file and generate a functional 867 file in a matter of hours. I had to perform some minor structural alterations in order to support both of the files we needed to generate, and it's certainly not a complete implementation, but then we didn't have to spend $500 for the standard or $150 for the file from American Coders.

I believe that this is one of the trickiest parts of the project. So many different skills are involved. It's not just familiarity with XML but the ability to visually parse structure and make intelligent guesses as to the meaning of tags without ever seeing their specification. Being able to infer the meaning of the sequence and occurs attributes, determining that a value of -1 for the occurs attribute implies infinity. Simply being able to read the definition in order to know how to structure the application is quite difficult.

That being said, let's look at how we go about building the transaction set. We need to obtain a reference to the header table (a form of a segment container) and start to create segments. The code follows:

tbl = ts.getHeaderTable();

seg = tbl.createSegment( "ST" );

elem = (DataElement) seg.buildDE( 0 );
elem.set( "867" );
seg.addDataElement( elem );

elem = (DataElement) seg.buildDE( 1 );
elem.set( transactionSetControlNumber );
seg.addDataElement( elem );

tbl.addSegment( seg );

The table reference is used to create a named segment ("ST" in this example.) The segment reference is used to build data elements which are then populated and added to the segment. Finally, the segment is added to the table. Here is the corresponding XML code:

<segment name="Transaction Set Header" id="ST"
  sequence="10"
  occurs="1"
  required='M'
  xmlTag="TransactionSetHeader">
    <dataElement name="Transaction Set Identifier Code" id="143"
      sequence="1"
      description="Code uniquely identifying a Transaction Set"
      type="ID" required="M"
      minLength="3" maxLength="3"
      xmlTag="transactionSetIdentifierCode"/>
    <dataElement name="Transaction Set Control Number" id="329"
      sequence="2"
      description="Identifying control number that must be unique within the transaction set functional group assigned by the originator for a transaction set"
      type="AN" required="M"
      minLength="4" maxLength="9"
      xmlTag="transactionSetControlNumber"/>
</segment>

The segment definition specifies the relative sequence of this segment within the enclosing segment, how many times it may occur, whether or not use is mandatory (it is in this case) and the XML tag associated with the segment. The XML tag is used in applications which need to parse an incoming document into OBOE objects. This is an interesting application which could provide an elegant bridge between XML and EDI.

Within the ST segment are two data elements definitions. The attributes extend those in the segment tag. The additions are the type (ID=identifier, AN=alphanumeric) and minimum and maximum length. All of these attributes, data elements and segments, including subsegments, are defined in the standards for the common document types. Returning to the Java source code, we can now see that the code is setting the transaction code and sequence number. The transaction code is used by communicating partners to indicate the purpose of the transaction set. In our project, we had different codes for the two different purposes.

This single example demonstrates most of what you need to know to code your application! The steps can be reduced to the following:

obtain a reference to a table (header, detail or summary)
create a segment
build a data element
set the value of the data element
add the data element to the segment
repeat steps 3-5 as many times as needed
add the segment to the table
repeat steps 2-7 as many times as needed

The only variation is when a segment contains a subsegment. In this case, instead of invoking createSegment on the table reference we invoke it on the segment which contains the subsegment. Look at the following code segment:

Table tbl = ts.getDetailTable();
Segment detail = tbl.createSegment( "PTD" );
...
Segment names = detail.createSegment( "N1" );
...
detail.addSegment( names );
...
tbl.addSegment( detail );

We obtain the reference to the table and create the segment ("PTD" in this case) in the usual manner. We create the "N1" segment by invoking the createSegment method on the "PTD" segment as opposed to the table. Similary, the "N1" segment is added to the "PTD" segment before the "PTD" segment is added to the table. Here is an extract from the XML file:

<table section="detail">
    <segment name="Product Transfer and Resale Detail" id="PTD"
      description="To specify identifying information"
      sequence="10"
      occurs="-1"
      required='M'
      xmlTag="ReferenceIdentification">
        <dataElement name="Product Transfer Type Code" id="521"
          sequence="1"
          description="To indicate the start of detail information relating to the transfer/resale of a product and provide identifying data"
          type="ID" required="M"
          minLength="2" maxLength="2"
          xmlTag="code"/>
            ...
        <segment name="Name" id="N1"
          description="To identify a party by type of organization, name and code"
          sequence="40"
          occurs="5"
          required='O'
          xmlTag="name">
            <dataElement name="Entity Identifier Code" id="98"
              sequence="1"
              description="Code identifying an organizational entity, a physical location, property or an individual"
              type="ID" required="M"
              minLength="2" maxLength="3"
              xmlTag="entityID"/>
                ...

This particular example also raises an important point: use indentation in your XML files to enable others to recognize elements at the same depth in the heirarchy. That's all there is to it!

I've made a couple of complete examples available:

Click here for the complete source code for an application
Click here to see the complete XML file for document type 867 (not a complete implementation but it works for our purposes)