PDS, XML

Manifold's PDS dataport imports data from collections of files that are organized using NASA's Planetary Data System (PDS) specification.  PDS is a very old specification, which can be challenging for modern GIS users to navigate.  Recent versions of PDS are introducing more modern approaches, such as the use of XML instead of a legacy ODL (Object Description Language), but many legacy quirks remain.

 

NASA's PDS4 version of PDS uses XML files to describe the different files and data within a file set organized using the PDS specification.  PDS file sets are often used to provide raster data in the form of images and tabular data in the form of tables.    The raster data in PDS images may be photographic image data or other raster data, such as terrain elevation data, measures of gravity, temperatures or other data.

 

Manifold provides two ways to import or to link tables images or tables from PDS file sets: we can use Manifold's built-in PDS dataport or we can use the PDS capability of GDAL.   This topic discusses the built-in Manifold PDS dataport.  See the GDAL / OGR topic for information on using GDAL.  Using either method provides import/link capability for images or tables that are in PDS file sets.

File Types and Extensions in PDS

PDS file sets often have many folders and files. The basic organization is straightforward, although the nomenclature used by NASA in PDS is often confusing to modern GIS users.  For example, in GIS a text file that describes the contents and provenance of data files is called a metadata file,  while PDS calls such files labels, using a word that in GIS usually means a text caption that appears in a map.   Mentally substituting the word metadata for label when working with PDS files or documentation may help avoid confusion.  

 

A PDS file set, often referred to as an archive, contains one or more data products, such as images and tables.  These are organized into folders and subfolders and are accompanied by catalog files and label files, both of which are plain text files that describe the organization of the archive or describe the contents of the data product files.   Typical files found in a PDS file set:

 

Images - Files with names ending in .IMG are in PDS image archive format, simple binary arrays of 16-bit signed integers, without any compression.  Each .IMG file is accompanied by a human-readable text file ending in .LBL that describes the image file.  .IMG files may be accompanied by a similarly-named .HDR file, a plain text, human-readable file which provides ENVI header information.   PDS .IMG files accompanied by an ENVI .HDR file can be directly imported by the Manifold IMG dataport as ENVI .IMG format.   More recent PDS data sets may use .RAW file extensions for PDS image archive format, with required image-reading metadata provided in an accompanying .XML file.

 

PDS archives may also include images in .JP2 (JPEG 2000) format, or .JPG (standard JPEG) formats.  Such files are provided for browsing convenience and are not the actual data, which is only provided in PDS .IMG format to avoid any risk of data loss through lossy compression that is allowed in JPEG formats.

 

Tables - Files with names ending in .TAB (for table) contain data for tables in plain text form where each row contains field values for one record, with the field values usually separated by one or more space characters.  Despite the use of a .tab extension for the file, these are not tab-separated values, and the first lines do not provide the names of fields.  .TAB files may be accompanied by a .LBL file that describes the field names and schema, or with more recent PDS archives may be accompanied by an .XML file that describes the fields names and schema.

 

Metadata - PDS provides human-readable, plain text metadata using the extensions .CAT for catalog files that describe the files and folder organization of a PDS file set, .LBL for label files that describe the contents of images, tables or other data files, and .XML in more modern PDS file sets to describe the organization and contents of other PDS files.  Metadata may also be provided in the form of plain text .TXT files, which usually contain plain English commentary.

Importing or Linking Images or Tables

The Manifold PDS dataport currently imports tables and images within PDS file sets that are provided with an .XML file.   Choosing the XML file will import the images and tables it describes.    Depending on how the PDS file set is organized, choosing a single XML file may result in the import of just one image or table, or it might result in the import of many images and tables.

 

To import from PDS formats:

 

  1. Choose File-Import from the main menu.

  2. In the Import dialog browse to the folder containing data of interest.

  3. Double-click the file ending in .xml for the PDS data of interest.

  4. Data referenced by the XML file, such as images and tables, will be created.

 

Older PDS file sets often do not include an XML file.   In many cases, the formats used for images and tables in such older PDS file sets can be imported into Manifold using direct import of files as follows:

 

Images - .IMG files containing images are often in ENVI .IMG format accompanied by an ENVI .HDR header file and can be directly imported. using Manifold's IMG dataport. Choose File - Import, navigate to the file and then double-click on the .IMG file to import it.   PDS .IMG files not accompanied by an ENVI .HDR usually can be imported with Manifold's RAW image importer, using the Scan Raw Binary File tool with information provided by the accompanying .LBL file.

 

Tables - PDS .TAB files can be imported as tables by first linking using the Manifold CSV dataport, unchecking the First line contains field names box, and choosing (space) as the List delimiter character.   That usually will result in a linked table with some extra, blank, columns.  Copy the table from the linked data source and paste into the main part of the project, and then the table structure can be cleaned up with Edit - Schema to delete blank columns, to name columns with useful names and so on.  See a detailed example in the Example: Create Terrain Elevation Raster from a NASA PDS Table topic.

 

Metadata - Metadata files with .LBL, .CAT, or .TXT extensions can be imported as comments by choosing File - Import, navigating to the file in the Import dialog and clicking the file to load it into the File name box, choosing TXT Files (*.txt) in the type box, and then pressing Import.

 

See the Create Terrain Elevation from a NASA PDS Table video for examples of the above.

 

Important: PDS imports may be slow.  Many large components may result when choosing a single .XML file.

Example: Import an Image

We have downloaded a sample PDS data set from NASA.  We choose File - Import and then navigate to the folder containing data of interest.  

 

 

We double-click on the .xml file in that folder.  

 

 

Using the information in the .xml file, the dataport automatically imports and names the image from the .raw data file.

 

 

This particular PDS data set contains a single grayscale image that shows part of the deflated landing system for the Mars Pathfinder 2 lander, with a deflated bag seen on the Martian surface.

Example: Import a Table

PDS data sets will often include tables.

 

 

We import a PDS .xml that we know contains a table.

 

 

Manifold automatically uses the XML file to find and read the table file, creating a table in the project.

 

 

PDS tables are imported as read-only tables, since they usually are in simple formats that do not contain indices.   The imported table will open, as seen above, with gray background color to indicate it is read-only.

 

We can make the table read/write by clicking Edit-Schema and pressing the Add Identity button in the Schema dialog.  That will add an identity field (the usual mfd_id field) and an index on that identity field, making the table read/write.

 

Most PDS data sets are accompanied by .LBL files, which provide metadata information.  

 

 

For example, drilling into the PDS table set for the table above we find a PDS3_ATM_CHAR_TABLE_ODL.LBL file that gives information on the table and the fields it contains, as seen above opened in Notepad.  The .LBL files are plain text files.

 

Notes

Inventory data - The Manifold PDS dataport reads inventory data.

 

Non-Intel floating point - The Manifold PDS dataport reads floating point values with non-Intel byte order.

 

PDS simplified - This topic provides an informal, simplified introduction to PDS.  For the full story, drill down into NASA documentation using a search engine, to find links like the Data Standards page, or the NASA eBook, NASA's Planetary Data System, written in 1992 and available for free on Google Books and the Google Play store.

 

PDS ambiguities in coordinate systems - Due to ambiguities in the PDS specification, images may import with a coordinate system (georeferencing) that might be slightly off or might be wildly wrong, and may require manual tinkering to set correct coordinate system information.  

 

0 to 360 longitudes - Tables in PDS usually contain longitude data as numbers from 0 to 360 degrees, instead of the usual GIS convention from 0 to +/- 180 degrees.   0 to 360 longitudes are easily converted to +/- 180 coordinates using a simple expression.  See the Example: Create Terrain Elevation Raster from a NASA PDS Table topic for an example.

 

Videos

Create Terrain Elevation from a NASA PDS Table - Do in six minutes what can take hours in other GIS packages! We import terrain elevation raster data from a NASA PDS archive and style the images with an elevation palette and hillshading. One of the images was wrongly georegistered by NASA, so we import the original LiDAR point data from a table in the PDS archive. We transform the imported table from 0 to 360 degree longitudes into +/-180 longitudes, construct a geometry field, and then we create a drawing from the table that shows the LiDAR point data in the table. Next, we use Kriging to create a terrain elevation raster from the LiDAR point drawing, and then we style that and hillshade it so it can be exactly overlaid on a Bing satellite layer to enhance the satellite photography with terrain relief.  See how Manifold's wonderfully efficient user interface and blisteringly fast parallel performance makes GIS work fast and easy!

 

The video above shows direct import of .img and .tab files from an older PDS archive that does not utilize XML files.

 

See Also

Images

 

Tables

 

File - Import

 

File - Link

 

Tools - Scan Raw Binary / Text File

 

Example: Link NLCD using Scan Raw Binary File - Use the Scan Raw Binary File tool to scan and to prepare a configuration file, which we use to link an NLCD raw binary file providing land cover data for Delaware as a raster image.   We use a standard palette to color the land cover data and then we assign a projection to the newly imported image so it can be used as a correctly georegistered layer in maps.

 

Example: Convert a 0 to 360 Degree Projection - We often encounter data, both images and drawings, using latitude and longitude degrees that appears to be in Latitude / Longitude projection but which has longitude values from 0 degrees to 360 degrees and latitude values from 0 degrees to 180 degrees, instead of the usual arrangement of -180 degrees to 180 degrees for longitude centered on the Prime Meridian, and -90 degrees to 90 degrees for latitude centered on the Equator.  This example shows how to utilize such data by assigning the correct projection.

 

Example: Create Terrain Elevation Raster from a NASA PDS Table - A long, multistep example showing many powerful features of Manifold in various steps.  We import terrain elevation raster data from a NASA PDS archive and style the images with an elevation palette and hillshading.  We notice one of the images was wrongly georegistered by NASA, so we import the original LiDAR point data from a table in the PDS archive.  We transform the imported table from 0 to 360 degree longitudes into +/-180 longitudes, construct a geometry field, and then we create a drawing from the table that shows the LiDAR point data in the table.  Next, we use Kriging to create a terrain elevation raster from the LiDAR point drawing, and then we style that and hillshade it so it can be exactly overlaid on a Bing satellite layer to enhance the satellite photography with enhanced terrain relief.