Author’s note: Link to a Romanian translation of this article  (by Alexander Ovsov)

Data versus Information

As part of open government initiatives, government agencies are creating guidelines and goals for making more of the information held by them more accessible to the public. See for example, the Open data initiative in the UK and the EU directive on Public Sector Information.

Of course, we advocate the judicious use of PDF for disseminating that government information. Well, in most situations.

We use the terms data and information to distinguish at least two ways government agencies are being asked to provide information to the public in electronic form. Data is just the raw numbers, names, places, etc. The stuff one might pull out of a database.  But data can be shaped into information by providing interpretations, drawing conclusions, pointing out inconsistencies and then packaging it in an attractive form. And then there is the most extreme view that the medium is the message!

When we go to government websites we usually want information not data. When an analyst goes to a government website she might want just the raw data so that it can be interpreted, shaped and analyzed, and turned into a specific document as information not provided by the government website.

Hammers

There is the saying that if you have a hammer, then everything looks like a nail. And we have to confess that, from our view, almost all needs are best addressed using PDF. PDF is our metaphorical hammer. We have some justification for this position as the most significant PDF software provider in the world.

Government agencies can, and do, use PDF for effective information distribution. Information distributed as a PDF can be downloaded, read in its electronic form, saved for later reference, shared and printed. Everyone has a PDF reader. PDF documents can also be infused with, what we at Adobe like to call, “rich document” features. The final representation of the information can be very important. As noted earlier, the medium is the message, or it certainly can make an important contribution.

However, for the person who wants raw data, PDF isn’t the right choice. See, we are willing to refrain from hitting everything in sight with our PDF hammer!  But raw data isn’t of much use unless you have metadata (data about data) describing, how to access the data elements, the ranges and meanings for the individual data items, where the data comes from, how old the data is, how authentic, and so many other important properties.  So we will argue later that, in fact, PDF can play a very important role in distributing raw data, offering a clever means to accompany that data with meaningful and accurate metadata. (We are using the term metadata here in a very general way as data about data, and maybe a little differently from what you are used to.)

XML

The government initiatives have surfaced people who have XML hammers. Some XML enthusiasts, but certainly not all, go overboard. We think it hurts their cause. As I have blogged earlier (XML for ••• ), XML  is one of the most misunderstood, yet useful, technologies we have in today’s toolkit. We need to drop our hammers and consider the facts.

Using XML for raw data is the kernel of a good idea. But there are some limitations of XML that need to be addressed when considering it as the only option for data distribution:

1. Those who are not familiar with XML, need to realize that XML isn’t a single markup language for a single use, but it is a method for defining and using specialized markup languages. That is why we have to say XML for business cards, XML for invoices, XML for classifying political action committees and so on. There are thousands of such XML markup languages and there will be thousands more to cover all those government datasets where XML is appropriate. I also have a blog on this topic.

2. Large raw datasets can be prohibitively large when expressed in an XML markup language. Unnecessarily large, from an information theoretic view. For example, here is an XML data file that can be found on www.data.gov at this webpage. Note that, when you download this file, it is a ZIP file whose size is 11,903,362 bytes.  When you unzip it though, you get a 220,655,917-byte XML file. In this case, the EPA personnel know that XML files can be very large and have packaged it in a ZIP for downloading to reduce the transmission time by a multiple of over 18. In other words, if it takes a minute to download the ZIP’ed version, it will take over 18 minutes to download a raw XML version. After unzipping, the XML file is identical to the original. So any advocacy for XML, should always be accompanied with a discussion on file size and considerations of using something like ZIP. To do otherwise would be irresponsible.

3. XML files need additional metadata in order to make use of the data that is found within them. If you are given three numbers (007, 56, 00010) you might not guess that they represent a birthday, which is more commonly supplied as 7-10-1956.  But even if it is provided in that form, it isn’t representative of how it is often expressed in other countries as 10/07/56. Unless given additional information, there is an ambiguity between the 7 and the 10 as to which is the month and which the day. And this is a trivial example. We need extra information, e.g., metadata, in order to be able to interpret data accurately and make proper use of it. The basic syntactic rules used for a markup language can and should be provided by offering an XML Schema (.xsd file) but even that does not go far enough. We also need to explain the semantics; that usually requires a technical document.

4. There are other raw data formats that might be more suited to particular needs such as standard spreadsheet files (.csv — character separated values) or the Microsoft and Open Office spreadsheet formats (.xlsx and .ods, respectively).  The latter can also include the formulae and rules needed to compute dependent and summary data. In addition there may be very application specific files, not in XML format, appropriate for specific needs. For example, shapefiles (.shp) for holding spatial feature information as in the http://www.data.gov/raw/6 webpage reference earlier.

Note:  There can be an argument made that compressing/decompressing files is so time consuming that the time lost there is not made up in reduced transmission time. With today’s lighting fast CPU’s the compression and decompression times are relatively minor, but the transmission times can be a problem if you don’t have the latest and greatest Internet connections. So nearly always, the choice to used compressed data is the right one for data expressed in an XML markup language.

PDF versus ZIP

In another InsidePDF blog (PDF File Attachments), the file attachment features of standard PDF (ISO 32000) are described. To summarize, any number of file attachments, in any format, can be embedded into any PDF file. They can be extracted for use by anyone receiving the PDF file. In addition, when files are attached/embedded into the PDF, they will be compressed using the same compression method that is used in ZIP files: deflate/flate. For the purposes of distributing government data, this is nearly ideal. The PDF file can carry the .xml or other raw data files as compressed attachments and that base PDF document, itself, can provide all the additional semantic information that would be needed in order to make use of the data — the metadata. If the raw data is in XML form, then a compressed XML Schema file (.xsd) can also be attached to the PDF document. So when using PDF, the points made above are addressed: file size, necessary metadata to define the XML markup language used, and formats other than XML.

Sample PDF envelope containing XML data

We have created a sample PDF envelope starting from this government dataset.  You might want to look it over. Note that both the XML dataset and the associated Schema file are attachments to the PDF that helps to define the XML markup language used for this file. We took the general introduction from the government web page and made up a brief description for each of the XML elements found in the file. Make sure to use a PDF reader that can display the attachment annotations and that can extract the attachments. Adobe Reader can do that.

Programmatic extraction of any data out of PDF

PDF attachments can be of any format and can also be organized hierarchically, just as you can with a ZIP file. And like ZIP, there are numerous open source projects devoted to the creation, modification and viewing of PDF. One very popular one is iText by Bruno Lowagie and iText Software. They also have an excellent example demonstrating how to use iText to create a PDF containing various data files. In addition, it also shows how someone could programmatically extract those contents.  This means that the PDF is not only usable for pure human consumption but also for pure machine consumption or a combination thereof.

Authenticating PDF envelopes by PDF Digital Signature

Another important benefit that just works out when using PDF envelopes, is that the digital signature technology available for PDF files also can cover the attachments automatically since they are an official part of the PDF file. Government agencies can send digitally certified PDF files containing data files and their customers can authenticate that the PDF, and all the attachments, came from that agency and have not been tampered with. See the InsidePDF blog about Authenticated PDF Documents.

Other ways to use PDF attachments for government information delivery

So we described how we use PDF to provide a complete package for raw data downloading.  But we can also augment richly formatted PDF information files with the raw data files that were used to create the PDFs.  For example, if a document contains, pie charts, bar charts or tabular information, the raw data that was used to create that formatted information can be attached to the PDF file. An annotation can be placed on the chart or table that allows the appropriate attachment to be extracted.  This is covered a bit more in an InsidePDF blog: PDF file attachments.

A third use for PDF attachments is to create hybrid files.  These are PDF files that attach the original editable source document (e.g., .odf or .docx file) that was used to create the PDF. In a sense, this make PDF editable, something that people have asked for. Quoting from the OpenOffice.org website: “A hybrid PDF/ODF file is a PDF file that contains an embedded ODF source file. Hybrid PDF/ODF files will be opened in OpenOffice.org as an ODF file without any layout changes. Users without this extension can open the PDF part of the hybrid file with their PDF viewer.” Adobe Acrobat’s Microsoft Office tools can also create PDF files with the Office file that created them as an attachment.

So hammer away, you PDF enthusiasts.

Data versus Information

Data versus Information

As part of an Open Government initiative the US Government has created guidelines and goals for making more of the information held by government agencies more accessible to the public. See for example, President Obama’s December 2009 directive.

Of course, I advocate the judicious use of PDF for disseminating that government information. Well, in most situations.

I will use the terms data and information to distinguish at least two ways government agencies are being asked to provide information to the public in electronic form. Data is just the raw numbers, names, places, etc. The stuff you might pull out of a database.  But data can be shaped into information by providing interpretations, drawing conclusions, pointing out inconsistencies and then packaging it in an attractive form. And then there is the most extreme Mcluhan view that the medium is the message! 

When I go to a government website I usually want information not data. When an analyst goes to a government website she might just want the raw data so that it can be interpreted, shaped and analyzed, and turned into a specific document as information not provided by the government website.

Hammers

There is the saying that if you have a hammer, then everything looks like a nail. I know for sure my 3 year old grandson thinks so when he has a real hammer in his hand.  And I have to confess that, from my view, almost all needs are best addressed using PDF. PDF is my metaphorical hammer. I have some justification for this position as the Project Leader for the ISO 32000 PDF standard.

For real, in most cases, government agencies can, and do, use PDF for effective information distribution. The information as a PDF can be downloaded, read in its electronic form, saved for later reference, shared and printed. Nearly everyone has a free PDF reader. PDF documents can also be infused with, what we at Adobe like to call, "rich document" features. The final representation of the information can be very important. As noted earlier, the medium is the message, or it certainly can make an important contribution.

However, for the person who wants raw data, PDF isn’t the right choice. See, I am willing to refrain from hitting everything in sight with my PDF hammer!  But raw data isn’t of much use unless you have metadata (data about data) describing, how to access the date elements, the ranges and meanings for the individual data items, where the data comes from, how old the data is, how authentic, and so many other important properties.  So I will argue later that, in fact, PDF can play a very important role in distributing raw data, offering a clever means to accompany that data with meaningful and accurate metadata. (I am using the term metadata here in a very general way as data about data, and maybe a little differently than you are used to.)

XML

The government initiative has surfaced people who have XML hammers. I think some XML enthusiasts, but certainly not all, go overboard. I think it hurts their cause. As I have blogged earlier (XML for •••), XML is one of the most misunderstood, yet useful, technologies we have in today’s toolkit. We need to drop our hammers and consider the facts.

Using XML for raw data is the kernel of a good idea. But here are major problems that arise because we do not go further into a more complete discussion:

1. Those who are not familiar with XML, need to realize that XML isn’t a single markup language for a single use, but it is a method for defining and using specialized markup languages. That is why we have to say XML for business cards, XML for invoices, XML for classifying political action committees and so on. There are thousands of such XML markup languages and there will be thousands more to cover all those government datasets where XML is appropriate. Please see my previous blog for more on this topic.
2. Large raw datasets can be prohibitively large when expressed in an XML markup language. Unnecessarily large, from an information theoretic view. For example, here is an XML data file that can be found on www.data.gov at this webpage. Note that, when you download this file, it is a ZIP file whose size is 11,903,362 bytes.  When you unzip it though, you get a 220,655,917 byte XML file. In this case, the EPA personnel know that XML files can be very large and have packaged it in a ZIP for downloading to reduce the transmission time by a multiple of over 18. In other words, if it takes a minute to download the ZIP’ed version, it will take over 18 minutes to download a raw XML version. After unzipping, the XML file is identical to the original. So any advocacy for XML, should always be accompanied with a filesize discussion and considerations of using something like ZIP. To do otherwise would be irresponsible.
3. XML files need additional metadata in order to make use of the data that is found within them. If you are given three numbers (007, 56, 00010) you might not guess that they represent a birthday, which is more commonly supplied as 7/10/1956 in the US.  But even if it is provided in that form, it isn’t representative of how it is often expressed in other countries as 10-07-56. Unless given additional information there is an ambiguity between the 7 and the 10 as to which is the month and which the day. And this is a trivial example. We need extra information, e.g., metadata, in order to be able to interpret data accurately and make proper use of it. The basic syntactic rules used for a markup language can and should be provided by offering an XML Schema (.xsd file) but even that does not go far enough. We also need to explain the semantics which usually requires a technical document.
4. There are other raw data formats that might be more suited to particular needs such as standard spreadsheet files (.csv — character separated values) or the Microsoft and Open Office spreadsheet formats (.xlsx and .ods, respectively).  The latter can also include the formulae and rules needed to compute dependent and summary data. In addition there may be very application specific files, not in XML format, appropriate for specific needs. For example, shapefiles (.shp) for holding spatial feature information as in the www.data.gov webpage reference earlier.

Note:  There can be an argument made that compressing/decompressing files is so time consuming that the time lost there is not made up in reduced transmission time. With today’s lighting fast CPU’s the compression and decompression times are relatively minor, but the transmission times can be a problem if you don’t have the latest and greatest Internet connections. So nearly always, the choice to used compressed data is the right one for data expressed in an XML markup language.

PDF instead of ZIP

In a previous blog (PDF File Attachments), I described the file attachment features of standard PDF (ISO 32000). To summarize, any number of file attachments can be embedded into any PDF file. They can be extracted for use by anyone receiving the PDF file. In addition, those file attachments may be compressed using the most common compression method that is used in ZIP files: deflate/flate. For the purposes of distributing government data, this is nearly ideal. The PDF file can carry the .xml or other raw data files as compressed attachments and that base PDF document, itself, can provide all the additional semantic information that would be needed in order to make use of the data — the metadata. If the raw data is in XML form, then a compressed XML Schema file (.xsd) can also be attached to the PDF document. So when using PDF, the points made above are addressed: filesize, necessary metadata to define the XML markup language used, and formats other than XML.

I have created a sample PDF envelope starting from this government dataset.  You might want to look it over. Note that both the XML dataset and the associated Schema file are attachments to the PDF that helps to define the XML markup language used for this file. I took the general introduction from the government web page and made up a brief description for each of the XML elements found in the file. Make sure to use a PDF reader that can display the attachment annotations and that can extract the attachments. Adobe Reader can do that.

Another important benefit that just works out when using PDF envelopes, is that the digital signature technology available for PDF files also can cover the attachments automatically since they are an official part of the PDF file. Government agencies can send digitally certified PDF files containing data files and their customers can authenticate that the PDF, and all the attachments, came from that agency and have not been tampered with. See my previous blog about Authenticated PDF Documents.

Other ways to use PDF attachments for government information delivery

So we described how we can use PDF to provide a complete package for raw data downloading.  But we can also augment richly formatted PDF information files with the raw data files that were used to create the PDFs.  For example, if a document contains, pie charts, bar charts or tabular information, the raw data that was used to create that formatted information can be attached to the PDF file. An annotation can be placed on the chart or table that allows the appropriate attachment to be extracted.  This is covered a bit more in my previous blog.

A third use for PDF attachments is to create hybrid files.  These are PDF files that attach the original editable source document (e.g., .odf or .docx file) that was used to create the PDF. In a sense, this make PDF editable, something that people ask for. Quoting from the OpenOffice.org website: "A hybrid PDF/ODF file is a PDF file that contains an embedded ODF source
file. Hybrid PDF/ODF files will be opened in OpenOffice.org as an ODF
file without any layout changes. Users without this extension can open
the PDF part of the hybrid file with their PDF viewer." Adobe Acrobat’s Microsoft Office tools can also create PDF files with the Office file that created them as an attachment.

So hammer away, you PDF enthusiasts.

Jim King (jking@adobe.com)

Today we are only going to discuss certification digital signatures. The idea of a certification signature is to make sure that the document is authentic and has been unaltered since it was signed by the authenticating party. Don’t you want to know that the PDF statement you receive from your bank did, indeed, originate with your bank and hasn’t been tampered with since they created it?  Of course, and that is what certification signatures are all about.

One other important thing to note is that any file attachments within the PDF document can also be covered by the same certification as the PDF file itself since those attachments are part of the PDF contents. This offers a very cool way to deliver information using file types other than PDF, by enveloping them in a PDF file that has been certified. And as explained in my blog about attachments the attachments may be compressed when attached to the PDF file making them smaller for transmission. This is especially cool for the delivery of certified XML data files. The enveloping PDF can also have pages describing the attachments and how to process them.

Example walkthrough

It is instructive to use a real example to show you about authenticated (certified) documents. The US Government Printing Office (GPO) delivers some certified PDFs from their website. This is a quote from the GPO home page:

"The U.S Government Printing Office (GPO) provides publishing &
dissemination services for the official & authentic government
publications to Congress, Federal agencies, Federal depository
libraries, & the American public."

The GPO certifies documents using PDF digital certification signatures and then puts those documents onto its website. Here is one you can look at and which I will use as an example.

If you open it in Adobe Reader either within your browser or in Adobe Reader directly you should see this:

Other PDF readers should also display some kind of verification information when a certified file is opened. If not, get a better reader.

Using Adobe Reader, the blue band at the top will display when a document has been certified with a digital signature as this one has. The blue band is normally not there. What the band displays is
“Certified by Superintendent of Documents <pki support@gpo.gov>, United States Government Printing Office, certificate issued by GeoTrust CA for Adobe.”
The blue band also displays an award-type ribbon on the left end to indicate that the document has been validated, just now when opened by Adobe Reader, and found to be authentic according to the digital signature that the GPO applied. 

Special Note:  The phrase "certificate issued by GeoTrust CS for Adobe" means that the GPO obtained a signing certificate from the certificate authority (CA) GeoTrust. The "for Adobe" has alarmed some people, but it just means that the certificate references the Adobe root certificate that comes with Adobe’s products. Adobe and GeoTrust, as well as many other CA’s, have a written contract that allows those vendors to reference the Adobe root (requires a secret key from Adobe) in exchange for guarantees on the level of integrity they enforce when issuing certificates. The bottom line for someone seeing this blue band with that reference means that the signing agency had to meet strict requirements to get that certificate and that the document authenticated successfully. This does not mean that Adobe has any lock on such certifications. It is OK for some other vendor to put the Adobe root certificate into their root store and also be able to validate these documents. Anyone can purchase such a certificate from many different vendors. This page gives a lot of information about certifying documents.

Another Note:  In July 2010 I gave an introductory talk about the public key infrastructure (PKI) and digital signatures at the American Law Librarians Conference. You may want to consult that for more details about this signing technology.

Double Checking
As shown below, using Adobe Reader, a panel will open on the left when you click on the blue band at the top right end where it says “signature panel”. Within that panel you can open the various topics to see the status of the integrity of the file and of the certifying digital signature. In this case, below, you see that the check showed that the file hasn’t been changed and the signature checks out OK to be that one really belonging to the GPO because it cryptographically chains up to the Adobe root.

The workflow

The workflow that I have referred to that GPO uses with these PDF files is one where they put the certified PDF file on their website and the user opens it in a web browser or downloads it onto their disk for viewing. So the validation of the certification happens on the user’s machine using Adobe Reader, Acrobat Standard, Acrobat Professional or any vendor’s reader as the document is opened. Some readers may not do this checking automatically or at all. If you want to have the document authenticity checked, make sure your reader does that, get a reader that does or use the Adobe Reader.

Here are the details of how those PDF documents are read and check for authenticity instantaneously as you open them on your machine.
A customer of the GPO downloads such a certified PDF file. She opens it in Adobe Reader either within a browser or not. The steps that the Adobe Reader takes before showing the blue banner are: it computes the cryptographic hash of the PDF file as downloaded onto her disk. The encrypted digital signature inside the PDF file is decrypted using the signer’s public key (which is also provided inside the PDF file). The decryption can only be successfully accomplished if it had been encrypted by the private key corresponding to the signers public key (the one belonging to the GPO). The cryptographic hash that was put into the digital signature by the GPO is compared with the one we just computed. If they match then the document hasn’t been changed since the GPO signed it. If they don’t match then some tampering has happened. This is reported to the person opening the file in the blue band at the top.

Further, the certificate for the signer is checked to see if it “chains” successfully up to the Adobe root (or some other root that has been agreed to or that is commonly available). This chaining operation also uses the public/private key pairs and encryption to verify that each certificate was indeed the one issued by the CA that it says it was issued by. If all this works, then we know that whomever is identified as the signer that certified this document is indeed who they say they are according to the CA’s. The success or failure of this is also reported to the person opening the PDF file. In some cases the software opening the file may attempt to go onto the Internet to check with CA’s to make sure that the certificates are still valid and have not been revoked.

In this example, the GPO obtained a certificate from the GeoTrust Certificate Authority (CA) which chains to the Adobe trusted root.

I took that file and modified one byte in it. Now when I open it I get this in the blue banner indicating that it has been tampered with (is invalid).

Making certified documents

Here is how those PDF documents are made and put onto the GPO website:
At the GPO, the files are signed with a certificate owned by the GPO and issued by a Certificate Authority (CA) who issues them against the Adobe root. If you have Adobe Acrobat Professional and a certificate you, too, can perform this certification step. Just go to the menu item Advanced->Sign & Certify and pick either Certify with visual signature or Certify without visual signature. This example was certified without a visual signature. You need to have obtained a digital certificate that contains your public/private keys and identifying information in order to do this. These are obtained from CA’s like GeoTrust as shown here. Adobe, through its LiveCycle server product also provides services that can certify documents as they are created on a server.

The Adobe software, whether Acrobat Professional or LiveCycle server, saves the file to disk, then computes the cryptographic hash over the bytes found on the disk with the exception of the bytes where the digital signature will eventually be placed. It then encrypts that hash using the certifiers private key (GPO private key) and incorporates the result into a “digital signature” block as defined by various standards organizations. It then inserts this block into the hole that was left in the PDF file on disk to receive it. The digital signature contains the signer’s certificate that includes her public key and the identity of the CA that issued the certificate. Any other certificates that will be needed to confirm the validity of the signer’s certificate are also included in the digital signature or in other places in the PDF file.

This signed/certified document is then placed onto the agencies website (e.g., GPO).

Here is a version of the GPO story.

Jim King (jking@adobe.com)

One of the great features of PDF is its ability to carry attached files, just as e-mail messages can carry attached files. Any kind of file, and any number of files, can be sucked into a PDF file. These are held internal to PDF as “stream” objects, one of the basic 8 object types from which all PDF content is built (numbers, arrays, strings, true, false, names, dictionaries and streams). Streams start with a dictionary object but then carry along an arbitrarily long sequence of arbitrary 8-bit bytes. Stream objects meet the generic description for disk files quite well.

Various compression/decompression methods can be used on streams, so even though including attachments in PDFs can tend to make the PDF files large, it is mitigated considerably by using one of PDF’s 8 compression methods. Flate compression, the technology also used in ZIP and PNG, is most commonly used for PDF attachments since it is lossless (you get exactly the same bytes you started with from a compression/decompression cycle), it gives good compression on arbitrary byte sequences and compression and decompression are both fast. Examples which I have, go from 1/3 reduction in size to 1/200 reduction.

So what might PDF attachments be used for? One use is for “hybrid files”. The office products from ODF while making a PDF file can attach the original source ODF document to it. They have made their software recognize these special PDF files, which they call hybrid files, and when asked to open them, they actually open the attachment. In this way a hybrid file is both a final form PDF file and an editable office document. Adobe’s Acrobat products can attach Microsoft Office documents to the PDF files produced by them automatically, also. The user has to extract the Office document from the PDF, themselves, in order to then edit it with the Office product.

Another very cool way to use attachments is as if the PDF was a file folder. In fact, the ISO standard defines a way to include an index for those file folders so that when opened, a panel alongside the base PDF display can show a list, much like an e-mail list, of all the attached files, providing their name, data of creation, author, or whatever else the creator of the PDF wanted to include in the index (see 1.a and 2.a below). These are called PDF collections.

Choices for using attachments

There are basically two ways, each with two ways, to manage attached files in PDF.

1. Attachments to a base document.

1.a. The names and sizes of all documents attached can be pulled out of the PDF file and display in a special viewer panel. From this panel attachments can be extracted, additional files may be attached, or existing ones deleted.

Here is a screen shot of a sample of what you will find in the base attachments panel of Adobe Reader and Adobe Acrobat. The contents and display of this panel is fixed by the application and just includes the basic information normally available about a file.

1.b. In addition to being an attachment as in (1.a), an attached file can be represented by an annotation on a page. Various icons annotations can be used to represent the attachment like a thumb tack, chart icon, or paper clip. When the icon is double clicked upon, the attachment can be extracted or deleted. Of course, these are normal attachments that show up in the list provided in 1.a above. I like this choice because, for example, you can attach the raw data that was used to produce a pie-chart and have a clickable annotation right alongside the chart to extract the data. You can do that for all 20 pie-charts in the document. In that way, if someone wants to play with the numbers in a spreadsheet or make their own chart of the different type, they can. This is a screenshot of what that can look like with an annotation popup window showing:

            

2. Attachments with additional indexing and navigation information

2.a.  As noted earlier, the ISO PDF standard provides for the author of the PDF file to include a table of terms and values in order to be able to display a specialized index of all the attachments. PDF files set up this way are called PDF Collections. The mortgage industry has found this to be a very useful way to bring together, into one PDF file, all the various documents that represent a complete mortgage deal.  And they can be indexed, dated and named to help in getting to the right sub-document efficiently. Very similar to the base panel of (1.a), these allow the document author to use any index terms and values. In this example a “Birthday” field has been added to the basic indexing data.

2.b.  Based upon Collections (2.a), an author can also include a graphically rich interactive “navigator” that allows the creative juices to flow for showing various aspects of the attached Collection files. These are called Portfolios. This is a new feature that makes use of Flash embedded within the PDF file and called a Navigator. It has access to the indexing information described in 2.a and it can also get thumbnails of the first pages, etc. In Adobe Acrobat there are a set of pre-made Navigator files that present various effects like a carousel of sub-files, a linear fly-by of them, etc. Adobe currently supports this in Acrobat 9.0 and greater via an Adobe extension to ISO 32000-1.

PDF attachments can be a very effective way to have one file that contains a wealth of related content.

Jim King (jking@adobe.com)

I think PDF forms represents a very powerful and significant tool. Increasingly we want both humans and computers to read documents. However, the requirements for easy reading for each is considerably different.

For a long time the primary use of computers has been "data processing". Business data processing has worked with structured data consisting primarily of numbers and text strings whose meaning and properties are well defined and known a priori. Much of the data semantics is build into the data processing software.  In the last decade or two, distribution and sharing of information among humans has moved in to share the primary spot.

Humans, when given numbers and strings, also need a context in which to understand their meaning and significance.

When creating a PDF form, a designer makes a very explicit decision of which information is needed by humans and which is needed by our data processing software. People do document processing and for the most part computers do data processing.

Those places where fill-able blanks occur in a form define the data that is being collected or displayed.  The background or "artwork" of the form turns the raw data into a document that provides the context in which the human can understand the data.

Here is a diagram showing how an artwork presentation layer and a data layer come together to make a document from which both the human and the computer can obtain exactly what they need.

PDF forms maintains this separation of layers and the data layer can be imported or exported into and out of the form artwork layer. The humans see the composed document and the computer can process the data only, with the traditional data processing software. Either the computer or the human can supply the contents for the data layer for presenting or gathering the data.

So, I think forms offer a very clever way for computers and humans to both see that part of a document most suitable and necessary for them to process.

Jim King (mailto:jking@adobe.com)

I think most technical people share a problem that I have: we have extreme difficulty in expressing ourselves in one simple sentence.  I have this problem when responding to questions/issues about PDF. For example, I have a hard time responding to this inaccurate statement in a short sound bite:

"PDF is great because it is not editable and freezes the content."

Technically that statement is totally inaccurate but there are related statements that are true.  For example:

"PDF is great because it not only captures my content but allows me to chose and lock down the look and feel for my content."   or

"PDF is great because I can apply a document signature to the file after I create it and then people can detect if it has been tampered with between me and them."

And here is one that I encountered from my financial advisor: "I had always sent my customers paper spreadsheets in the mail because I didn’t want them to have my spreadsheet electronic files that have my intellectual content as far as the calculations and macros. Once I could make PDF files from my spreadsheets I can send them electronically and not worry."  Am I to conclude that his primary value to me was in his spreadsheets?

Editablility and Resuse

But to get at the issue of re-use and editability versus frozen content, I have to use quite a few sentences, in fact, the remainder of this blog and the following blog.

The first issue we have to get straight is whether something is a function of the PDF file format or of the software that processes it. If people are concerned about the PDF file format then they need to join the ISO committee that is now managing PDF as the ISO 32000 standard. Many of my previous blogs record the process of moving the ownership of PDF from Adobe to ISO which was completed in January 2008.

But most of the reuse issues are a property of the software not the PDF file format. So if someone doesn’t like the behavior of their current software they might consider looking for other software and/or convincing someone to provide software with the needed function. 

But just for example, there are degrees of resuse that have been incorporated into Adobe’s Acrobat viewer including the following:

•  Copy/Paste.  If the author permits it, I can copy content from a PDF and paste it into other files. Adobe has spent a great deal of time and effort to make this work as well as it does, especially given the complexity of dealing with text.  Please see my previous blog entry about text in PDF.

• Export. Acrobat supports exporting PDF content into various formats including .rtf, .doc, .html, .eps, .png, ,jpeg, .xml, .jpeg2000, .tiff, .xls, .ps, .txt.  I was almost alarmed when I opened Acrobat to obtain an accurate list and found so many format supported.  And there are choices and setting for many of these. I assure you that this represents a great investment by Adobe to provide this support for reuse of PDF content. Many of these export functions are imperfect but do provide a strong basic ability to reuse content.

•  Hybrid Files. One can make a "hybrid" PDF document that includes the author’s original source file as an attachment. This is supported as an automated feature by Open Office tools as well as the Acrobat tools that create PDF files form Microsoft Office products. This provides a final form PDF document with the editability of the original source that the author used to create the PDF in the first place.

•  Forms. A more sophisticated kind of hybrid file is supported by PDF fill-in forms. This is so cool that I am going to make the discussion about it a separate Part 2 to this entry. (I wonder if the reason I think this is so cool is because I defined the properties for the Acrobat forms prototype in 1993. Na!  It is just cool!)

If an author wants to inhibit the reuse of the content in their documents they can set properties within the PDF file to prohibit it. For some authors the content of their documents represent their intellectual property and they want to protect it.

So, if things don’t work to our liking it may be an authors decision or the software designers decision, but seldom should we hold this as a PDF deficiency. PDF is a cool tool.

Jim King (mailto:jking@adobe.com)

Adobe tried very hard to allow new PDF files with new features to be viewable in older viewers.  Likewise, we tried to design our viewers so that when they were the “old viewer” they would do the best that they could to display a file even if it had new features.  And perhaps most important of all, we almost never removed any feature from PDF so that any file could be viewed by today’s viewers even if the file was made 16 years ago.

This paper was derived from an internal Adobe technical note written by me and a task force of employees who studied the whole issue of versions and compatibility in 2006. I have acknowledged those other employees at the end of the paper. Thanks to them.

That paper is, of course, a PDF file and is attached here:
Compatibility_090819 (56K)

Enjoy!  (Oh, and I almost feel like I should apologize for not making more regular posts to this blog. I have just been very distracted with several things. Hopefully I can find more time in the future.)

Jim King  <mailto:jking@adobe.com>

Background

The European Union (EU) is going through a very interesting and exciting time of trying to bring together many hitherto independent countries into, well, a European Union. There is particular interest in government-to-constituent electronic communications and in conducting business transactions across the EU electronically. So the EU is pushing the electronic envelope in many ways that by the nature of their activity is very “standards” oriented. This is in contrast to, say the US where various government agencies and various business are approaching these things less from a standards view and more from a view of what works and what is available. At least these are my observations, for what they are worth. This is not to say that the US is not interested in standards but just that the EU seems to be nearly consumed by a standards consciousness.

Electronic signatures (e-signatures) play an important role in assuring trustworthy and legally sound communication between governments and those governed and for businesses conducting electronically assisted transactions with other businesses and with customers also on a sound and legal basis.

The Directive 1999/93/EC of the European Parliament and of the Council, dated December 13, 1999, about a Community Framework for Electronic Signatures, is commonly referenced as the test(s) that electronic signature technology has to pass to be used in various legally binding manners within the EU. In fact, all EU countries have agreed to accept “Qualified Electronic Signatures” (QES) on a par with plain old ink-on-paper signatures and other electronic signatures including “Advanced Electronic Signatures” (AES) “cannot be denied legal effectiveness or admissibility as evidence”.

PDF digital signatures can be AES and QES

One reason I bring all this up is that PDF digital signatures can be QES according to the EU definitions, provided that the certificates used are “Qualified Certificates” (QC). PDF digital signatures can also be AES.

A lot of this has to do with establishing a hierarchy of trusted “Certification Service Providers” (CSPs). In practice these are Internet servers that deliver certificates that establish an association between people and their public keys.  I want to make the most important point that since PDF digital signatures are based upon the same PKI (Public Key Infrastructure) standards that these CSPs use then PDF can provide AES and QES. PDF is very suitable for conducting business in the EU and now with PDF an international standard (ISO 32000-1) the EU owns PDF just as much as anyone else. Certainly, Adobe no longer owns PDF.

Meeting requirements

Here is a key quote from the 1999/93/EC Directive (Article 2.2):

“advanced electronic signature” means an electronic signature
which meets the following requirements:

(a) it is uniquely linked to the signatory;
(b) it is capable of identifying the signatory;
(c) it is created using means that the signatory can maintain
under his sole control; and
(d) it is linked to the data to which it relates in such a
manner that any subsequent change of the data is
detectable;

All of these properties can be satisfied by a PDF digital signature. The standardized PKI Certificates can satisfy the (a) through (c) and (d) is satisfied by typical PKI signing technology using message digests. More background about PKI and PDF signatures is provided in my previous blogs. The QES add to these requirements additional ones about the quality of the certificates used and the CSPs and is spelled out in Annex I and II of the Directives. PDF can use these qualified certificates and hence can support QES also.

Watch the wording

PDF digital signatures have many optional choices and exactly which ones are used for any given signature depends upon the software used and in some cases on the signers choices. For example, which signer certificate and who issued it are the signers choice. I have tried to word my claims carefully by saying “PDF digital signatures can be” QES and AES since it is possible to chose options that will not satisfy the EU requirements.

ETSI/ESI and Electronic Signatures

European Telecommunications Standards Institute (ETSI) is recognized by the European Commission as a European Standards Organization. Its Electronic Signatures and Infrastructure Technical Committee (ESI) has established standards as its title suggests, in particular CAdES (CMS Advanced Electronic Signatures:TS 101 733 ) and XAdES (XML Advanced Electronic Signatures:TS 101 903 ). These standards were carefully crafted to follow the European Commission Directives and have become relatively synonymous with those directives.

PDF digital signatures and CAdES share the same infrastructure. They both use the Cryptographic Message Syntax (CMS) including particularly PKCS#7. PDF also allows the use of PKCS#1 and other schemes so this is a point where we must say that PDF supports PKCS#7 but not exclusively. We note that the European Directive is not so specific that it spells out the use of these technologies but CAdES and PDF have made these implementation choices.

There are some very particular differences in the exact way that PDF uses PKCS#7 and the way that CAdES uses it. Since some people associate CAdES synonymously with the European Directive they conclude that these differences make PDF not comply with the directives. The fact is that CAdES and the European Directive are not the same thing and although CAdES is an outstanding standard that follows the directives it is not, nor will it be, the only technology that follows the directives. The difference between PDF digital signature and those of CAdES are very minor and the European Directives does not give enough technical detail to distinguish between them.

ETSI/ESI and ISO 32000-1

ETSI has recently established a Task Force (TF) within ESI to establish standards common between CAdES, XAdES and PDF digital signatures as specified in ISO 32000-1. This TF is in the process of making sure that these technologies come together to everyone’s satisfaction and they will make special efforts to make sure there is no doubt that they follow the European Directives as they evolve. In particular, they will spell out and standardize which choices for PDF (ISO 32000-1), assure that the signatures are AES or QES. They also plan to work with the ISO working group on any changes for the future digital signature technology in PDF (anticipated as ISO 32000-2).

This is good news for both ETSI/ESI and the ISO PDF working group.  It is especially good for users who want to use standard digital signatures.

I think I have one more blog article in me about digital signatures, so stay tuned for a few more details about PDF digital signatures and how they work.

Jim King (contact: jking@adobe.com)

If you have topics related to PDF that you would like me to publish blog articles about, just send me a note.

Again, happy new year and to best to you in this upcoming, probably financially difficult, year.

Jim King  (contact: jking@adobe.com)

If I receive a digitally signed PDF file what should I expect?  First I would like to know that the file is unchanged since it was signed. A central notion is that there are encryption techniques, essentially all of them, where if the encrypted file is changed it cannot be successfully decrypted. The decryption produces some garbled garbage instead of meaningful results. So the person who is digitally signing the document encrypts it with their private key. Then if I can successfully decrypt it, I know it has been unchanged since they encrypted it.

So we need to know that the file is unchanged, but also since when and who says so. At least, in theory, I know exactly who that signer is because the certificate that has her public key was created by an authority (CA) that I can trust or one that is trusted by one that I can trust or … .  Her signature is also time stamped so I know when the file was signed. So the “since when” and “who says so” is handled by the use of PKI certificates and the encryption that takes place when the PDF file is signed. At the time of the signing the document that is to remain unchanged is established.

An interesting side note: there is no way to stop someone from changing a file, but we do have the means to detect if they did.

Message digests (cryptographic hash functions)

Encrypting or decrypting a large file is computationally intensive and can take a noticeable amount of time even on today’s powerful computers. So in cases where we would use encryption as a technique, not to hide the contents, but to assure that a file has not changed, another method has been developed based upon what are called “message digests” (MDs) or “hashes”.

What if we could just number all documents or files that were ever made. Then I could just tell you the document number and you would know what document I meant. Very crudely, this is the idea of a hash or MD. If we truly numbered all documents the size of the numbers would be huge, so nothing would be gained even if we could do this.

The idea of a hash or MD is that the number of real documents is extremely tiny compared to the number of all possible documents so people have developed algorithms that will derive a small sized number for any given document and the algorithms are mathematically justified to 1) rarely, if ever, produce the same number for two different documents 2) not be invertible, (nearly impossible to generate the document from the number) and 3) make it nearly impossible to make a second document, different from the first that has the same number. These phrases have rarely’s and nearly’s because this work is based upon statistically rare events not happening or extremely complicated and time consuming computations that are, today, impractical.

OK, so what is the deal here. It is faster to compute an MD over the document than to encrypt the document, and the resulting digest is limited to say, 32 bytes of data so compared to the actual document, it is very small. I send you both the document and an MD I computed over its bytes. Then you compute the MD again on your copy of the file, using the exact same algorithm that I used and you compare your MD to the one I provided. If they match, you are nearly assured that your copy is the same as mine. Again, computing the MD is faster that encryption/decryption, and besides that, the document can remain unencrypted and hence readable without any special computation. Something useful if I am not so concerned about the chance that it might have been altered but just want to view it.

But there is one glitch in what I have said so far. What if someone intercepted the document and its MD in transit and changed the document, recomputed the correct MD to match the changed document and forwarded it to you. It would check out OK.  So in addition to what was said above, we encrypt the MD with the signer’s private key so that it cannot be altered without detection. We use the same document signing technology discussed earlier but we only apply it to the computed MD and not to the document. We let the MD serve as a document surrogate or “digest” for these purposes.  As long as we trust and believe the inventors of these hashing algorithms, the chances of making a mistake are so rare they will not happen to us in our lifetime (or some strong statement like that).

So to summarize what happens when a PDF document is signed: 1) the signing software computes an MD using the bytes of the document file and encrypts it with the signer’s private key and 2) the signer’s public key is made available in a signed certificate (package), the certificate having been issued and signed by a CA which may be authenticated by a chain of CAs via their certificates. 

When the signature for a document is checked: 1) the MD of this copy of the document is computed, 2) the signer’s public key is obtained from the certificate and if needed the certificate is decrypted and examined to make sure the identity of the signer is as expected, 3) the encrypted MD sent with the file is decrypted using the signer’s public key from step 2 and compared with the MD computed in step 1.  If all this checks out then the document is an identical copy to the one signed and the signer is who they say they are.

One last glitch for the basic story. A PDF document is a single file and we want to keep that property.  But we have the MD which is a hash over the whole PDF file which would itself then contain the MD. That is a circular problem that is impossible to solve directly: computing something that is based on the results of the computation. So the complete PDF file with all the signature information is saved onto the disk but with a reserved “hole” where the package containing the MD will be placed. Then the MD is computed over all the bytes on the disk, except for the hole. Then the MD package is written into the hole. Of course, when the signature is checked, the hole is again ignored when the MD is computed.  A little funky but it works, and we can have a signed PDF document that maintains its property of being a single self-contained file.

And, of course, all of this is documented in the public ISO 32000-1 standard.

Next time I will go over some of the things specific to the European Union with respect to electronic and digital signatures.

Jim King (contact: jking@adobe.com)