PBF format is OpenStreetMap (OSM) native Protocol Binary Format (PBF), a vector format that provides total, native access to every detail of OSM data using OSM's recommended format. Manifold also reads OpenStreetMap OSM and O5M formats.
Important:
PBF is a complicated format internally: imports are slow. Manifold's
PBF dataport is steadily improving to handle larger and larger PBF files
and to import faster. The PBF importer currently can import
the entire planet OSM PBF (all OSM data for the entire Earth, a terabyte
project). A whole planet import takes slightly less than a day on
typical desktop systems with conventional hard disk drives, or about 12
hours with faster systems.
Reading PBF splits PBF data tags into multiple tables based on the tag name: a table for tags starting with addr, a table for tags starting with building, a table for tags starting with source, and a table for all other tags. Splitting a single PBF data set into multiple tables allows importing whole-world PBFs covering the entire planet without running into the current Manifold limit of 2 billion records per table. When the 2 billion record per table limit is removed in upcoming builds, the PBF dataport will, be altered to read all tabs into a single table.
When a PBF file is imported, we get a main drawing and table plus accessory tags tables, as noted above The main table in a PBF import will have an mfd_id_x index on the mfd_id field and also a Geom_x index on the geometry, Geom field. That allows the main drawing to be visualized and it enables selections in the main drawing and table.
Other tables will not have any indices, since they may or may not be of interest to users, and adding an index to very large PBF files can take a few minutes. The main candidate for adding an index in tag tables, as well as the main table, is the id field in those tables, which is used to associate data in in tags tables with objects in the main table. See a step by step example of adding an index to the id field in the SQL Example: Extract Airport Runways from an OpenStreetMap PBF topic.
In the following example, we will select objects from the main drawing and table, which already has an index on the mfd_id field, so we do not need to add any additional indexes.
To import from PBF format:
Choose File-Import from the main menu.
In the Import dialog browse to the folder containing data of interest.
Double-click the file ending in .pbf for the data of interest.
Tables and a drawing and other components, such as labels, will be created.
The drawing file above is the main component of interest, providing a mix of points, lines and areas using default formatting.
The illustration above shows a view zoomed into the drawing, near downtown Boston. We typically will want to create separate drawings for the points, lines and areas from the drawing. This is easy to do by using the Select pane to select all areas, copy them, paste into a new table and then create a drawing from that table, and so on for lines and points.
With the focus on the open drawing window, using the Select pane choose the Geom field, and then double-click on the Search template to launch it.
In the Search template, in the pull down menus for parameter boxes, choose type as the Use option, with equal (=) as the Condition and area as the Value. We want to select all geometry where the type of the geometry is areas. For Action, we leave the default replace selection choice.
Press the Preview button to see a preview.
We do not have to do a preview before applying the selection, but it is often a good idea to check our work with a preview before proceeding.
Right away, the template shows in blue preview color the areas that will be selected. A blue preview caption bar appears at the top of the window giving the name of the template being previewed. Previews are always drawn on top of all other layers and objects.
If we would like to see objects under the preview, we can right-click the caption bar and set the opacity of the preview to less than 100%, so objects from below can be seen, or, as in the illustration above we can right-click the caption bar and choose Left to get a split screen effect, with the preview shown only on the left side of the window. We can drag the vertical blue splitter line left and right to show less or more of the preview.
If we like what we see, we press the Select button in the Select pane.
That selects the areas, which right away are shown in red selection color. We choose Edit - Copy to copy the selected areas to the Clipboard.
We click on the Project pane and choose Edit - Paste. That pastes the copied areas from the clipboard as a new table, called boston_massachusetts 2. We change the name of the table to boston_massachusetts areas, a more useful name. We Right-click onto the new table and in the context menu choose Create - New Drawing.
In the New Drawing dialog we leave default settings as is, but we must specify the coordinate system by clicking the coordinate system picker button.
In the resulting menu we choose Latitude / Longitude and then we press Create Drawing.
A new drawing appears in our project. We can repeat the procedure above to copy the lines (using the Lines template in the Select pane), to paste them as a table and then to create a new drawing using that lines table. We can then again repeat the procedure to copy points (using the Points template in the Select pane) from the original drawing and to make a new points drawing.
In the display above, we have created a new Map, we have created a Bing Streets imageserver data source and dropped that as a layer into the map, and then we have dragged and dropped the new areas drawing we created into the map. It appears using default formatting. We have zoomed into the map to view the downtown region of Boston.
We can color the fill color for areas using the ColorBrewer CB Paired palette, coloring stroke color for areas using the same palette, but adjusted to be darker. We use the OSM id field for a formatting field. That has no significance other than as a number to assign varied colors to different areas.
The result shows a more interesting display.
We can also drop the lines drawing into map, coloring the lines using the id field and the ColorBrewer CB Pastel B palette with a halo in black that is one point thicker than the line. The result shows the wonderful detail of OSM PBF data.
Not for the Faint of Heart - PBF is the native format of OpenStreetMap, and requires significant learning to utilize to full advantage. Many users prefer to work with extractions from OSM in shapefile form.
Why Latitude / Longitude? - When we created the areas drawing, how did we know in the New Drawing dialog to change the coordinate system to Latitude / Longitude? When we had the focus on the original boston_massachusetts Drawing, from which we copied the areas, we could have taken a look at the Component tab of the Info pane to see that the drawing used Latitude / Longitude. Any geometry pasted from that drawing therefore also must be in Latitude / Longitude.
Example: Create a Map Showing OSM Use by Country - A start-to-finish real life example of map creation that combines various Manifold capabilities, including use of Edit - Join. Copying a table of numbers from a web site, we create a map that is thematically colored to show usage of OpenStreetMap by country in proportion to the population of that country.
Example: Create a Drawing from a Geocoded Table - A partner example to Example: Create a Geocoded Table from a Drawing A geocoded table has records with a latitude and longitude for each record. This example starts with a table containing a list of cities with a latitude and longitude field for the location of each city. We create a geom from the latitude and longitude fields using a template in the Transform pane and then we create a drawing that shows the cities as points. This example shows all the infrastructure steps involved.
SQL Example: Extract Airport Runways from an OpenStreetMap PBF - We write a simple SQL query using INNER JOIN to extract runway lines from an OpenStreetMap PBF of Cyprus, and to save those lines to a new drawing and table.
Example: Connect to an OSM Vector Server - We connect to an OSM Server that provides a vector layer containing points and lines in the OpenStreetMap database. We then show how to scrape (copy) data from the OpenStreetMap server into local storage. We extract building footprints from the local copy.
Manifold Future - Easy to Use Large OSM Data - An out-of-date video shot for Manifold Future that is still useful: it shows how to use the Edit Query button in the Select dialog to create a query that does the selection of areas, lines or points as desired. We can then edit the query to create a table in a single step by using SELECT ... INTO.