Example: Create a New Data Source from a Web Server

Many web servers are image servers, providing image.  tiles for a geographic view to browsers or to applications.  Typical image servers are Bing, Google Maps, Yandex and many others. Image servers can provide street maps, overhead satellite imagery, combinations of streets and satellite imagery and other data as well.  Web servers using other technologies, such as WMS, TMS and others, can also be used by Manifold, allowing us to pull layers from many thousands of other web served sources.

 

 This topic uses a connection to a web server that is operated by a third party.   Web servers operated by third parties may change their access policies, use different URLs, go offline, or be very slow.  If you have any connection problems with the server in this topic, review the Notes at the end of the Web Servers topic for tips.

 

A short cut:  We often use image server data sources as the base layer for a map.   The fastest way to do that is to simply create a new map, because the New Map dialog allows us to pick any image server from our Favorite Data Sources list as a base layer for the map.   The New Map dialog will automatically create an image server data source for us, if necessary, for the image server we choose as our base layer.  See the Maps topic.    

 

This topic shows how to create a new data source using the New Data Source dialog.   That allows us to create a web served data source from any of many thousands of choices, and not just from image servers in the Favorites list.   See File - Create - New Data Source for instructions on connecting to many different types of web servers, including custom web server connection strings.  

 

Many image servers are listed by default as Favorites, to make them one-click choices.  But we can get a seeming infinity of other servers by using the full New Data Source dialog.   In this example we will use the full dialog to connect to Microsoft's Bing server to generate a hybrid dynamic satellite image that is overlaid with text labels.

 

Launch Manifold and choose File - Create - New Data Source.   The dropdown menu provides a list of favorites to choose from as well as a More... option.  

 

 

Choose More... to launch the New Data Source dialog.

 

 

In the dialog we enter a descriptive name, Bing Hybrid, and for the Type we choose Web Server: imageserver.   That will automatically load the supplementary box just below the Type box with a list of all the various image servers we have installed on this machine, many of which are built into Manifold.   

 

We scroll through the list (a typically long list on this particular machine) and choose Bing Maps Hybrid.  That will load the Source box with the connection string this image server utilizes.   We press Create Data Source.

 

To fit into this documentation, illustrations show a small Manifold desktop, with only a few panes, docked to the right side.  In real life we use a much larger Manifold desktop, and more panes would be turned on, with panes docked to the left or to the right, or undocked, as we prefer.   Right-click a pane's tab to change where it is docked.  Manifold will remember that new arrangement for our next session.

 

 

 

 

A new data source appears in our project.   We can expand it and double click onto the image to open it and see it displays a typical Bing satellite image that has had labels added.  Microsoft refers to the satellite image with text labels and, zoomed in closer, streets, as a hybrid presentation, since it contains elements of a typical Bing Streets as well as Bing Satellite displays.

 

 

 

 

We can zoom into the image by right-clicking and dragging a zoom box.

 

 

 

 

As we zoom into the image Manifold automatically fetches more tiles from the Bing server.   The Bing server automatically provides those tiles and also adjusts the content as we zoom further in.   Like many image servers, the Bing server has been set up to provide more detail as views zoom further into the scene.

 

Using right-click and drag we can zoom further into the image and more details will appear.  We will zoom far into Italy into the region around Rome.

 

 

 

 

As we zoom into the region around Rome, Bing will begin showing roads, such as the ring highway around Rome, overlaid on the satellite view.

 

 

 

 

In the illustration above, we have zoomed far enough into Rome to see the bend of the Tiber river with the green star-shaped park of the Castel Sant'Angelo above the river.   The park is captioned Parco Adriano because what is known as the fortress of Castel Sant'Angelo is really Hadian's mausoleum, positioned on the opposite side of the Tiber not far from the now-ruined mausoleum of Augustus.

 

If we look very closely, we can see a faint white line extending horizontally from the lower one-fourth of the view, turning upward and then following along the outside of the ellipse of the plaza in front of St. Peter's, then continuing north, turning left in a diagonal and leaving the view by the s in the Bing Maps. tab.  That is the boundary of the Vatican state at the far left of the image.   

 

 

 

 

We can pan the display a bit to see the round circle of the mausoleum of Augustus, just to the right and above the gray bridge near the middle of the view.  The mausoleum of Augustus was built on the edge of the Campo Marzio, the Field of Mars, in early Roman times a field for practising military maneuvers.

 

 

 

 

Zooming in we get a better view of Augustus' Mausoleum, with the white building housing the Ara Pacis, the "Alter of Peace," across the street to the left.   The Ara Pacis is one of the most magnificent remnants of ancient Rome and is illustrated in the notes below.

 

Web server data sources in Manifold are highly useful because they can provide georeferenced imagery from commercial web servers like Bing.  Such commercial services are often far faster than WMS or similar servers run by government organizations or by private groups.   

 

Web server data sources can also connect to various "open" mapping servers and can be pre-configured by their authors for specially desired features, such as requesting formatting from the server or requesting specialized content of interest, such as maps that show public transportation or terrain elevation of interest to people undertaking journeys on bicycles.

Variations

Web servers run by various organizations often will utilize data from yet other organizations or may provide their own data using the same protocol.   For example, the opentopomap.org group based in Germany provides topographic maps using OSM (Open Street Map) data and using OSM protocols.   Therefore, the same image server module built into Manifold that works for OSM also works for opentopomap if we change the URL used for the source to point to the opentopomap server instead of the OSM server.

 

We right-click into the Project pane and choose New Data Source.  

 

 

We choose More... to launch the New Data Source dialog.

 

 

In the New Data Source dialog for the Type we choose Web Server: imageserver and choose the OpenStreet Maps Base image server.   We change the default Name to Opentopomap so we know this data source is not OSM.   Next, we will edit the Source string to replace the OSM URL with the correct opentopomap.org URL.

 

 

We use a URL of http://tile.opentopomap.org/{zoom}/{x}/{y}.png    Press Create Data Source to create the new data source.

 

 

 

 

The new data source appears in the Project pane under the name we gave it.   

 

When we open the data source we see that the names used inside are still those assigned by the imageserver module we used, for OpenStreet Maps.  We double-click open the image, and then we Shift-click the name tab to undock the image window.

 

 

Pressing Zoom to Fit we see a typical image server display. Depending on how loaded the opentopomap server it may take a few seconds to fill in the display.  Success!

 

 

Zooming in closer, into the regions surrounding the Alps, we see that our small hack did indeed work: providing the opentopomap.org URL to the OSM image server correctly generates opentopomap displays.  The choice of colors from this server is a bit garish for our tastes, but hey, it's free!

 

Because image servers are dynamic, easy to add to a project and require zero effort to style the maps and images displayed,  they can be a great way to provide background layers in maps.   

Notes

Credits - The use of the opentopomap.org URL with the OSM imageserver was first suggested by James Kelly in the georeference.org forum for Manifold users.   Thank you, James!

 

Internet changes - Providers of image servers come and go and often change their policies from free to paid, set limits on their services, change their URLs or protocols or otherwise alter what once was offered.  The URLs above, for example, for Bing and for opentopomap.org worked at the time this topic was written.   As with all things having to do with an ever-changing web, they might require adjustment in the future.

 

Collections of Web Servers - Check the Product Downloads page for pre-packaged project files that Manifold publishes in Release 9 / Viewer .map format which contain collections of dozens of popular web servers data sources.

 

Custom Setting for Web Servers:  Some web servers, notably TMS servers, use embedded command tokens.  Use Manifold's custom setting to connect to those.   See the File - Create - New Data Source topic for how to do that.

Augustus and the Ara Pacis

The image server illustration showing Rome in the topic above captures one of the most fascinating and least-visited monuments in Rome.

 

After almost sixty years of internal warfare, Gaius Julius Caesar had defeated all rivals to become Dictator for Life in 45 BC, ruling the entire Roman Empire until his assassination a year later in 44 BC.   His young great-nephew and adopted son, Gaius Octavius, born 63 BC, was one of the contenders for power in the chaos and civil wars that followed the assassination of Julius Caesar.  In 30 BC at the age of only thirty three, after slaughtering rivals and former partners, Octavius became sole ruler of the entire Roman Empire.   He was declared Augustus, "the August," by the Senate in 27 BC.  Augustus Caesar established "the Principate," the rule of Rome by emperors, ruling for forty years.   

 

 

He is seen above, approximately aged thirty six, in a bust on display at the Capitoline Museum in Rome, a fine example of the over 150 known busts having survived.  As emperor, Augustus preferred a highly idealized style in his busts, so earlier busts like the one above, which are not yet so idealized, provide a better sense of his true appearance.  The long years of peace under Augustus, an effective and personally restrained ruler, cemented the power of Rome.  Late in life, Augustus boasted he had found Rome a city of bricks and had left it a city of marble.

 

 

The Ara Pacis Augustae, the "Altar of Augustan Peace" and almost universally known as the Ara Pacis, is a large, open-air, marble altar surrounded by marble walls covered with continuous friezes of relief sculpture.   It was commissioned by the Senate in 13 BC to celebrate Augustus' return to Rome from lengthy and successful military campaigns in Gaul and Spain. Pieces of the altar were discovered in the late sixteenth century, it was not until the 1930's that the Italian government mounted a massive effort to excavate the entire altar from underneath existing, modern buildings.  The altar was then reassembled where it now stands, approximately a kilometer from the original location.

 

 

Originally brightly colored, as was most sculpture in antiquity, the altar presents religious processions and elaborate decorations.   It is noted for the numerous sculptures of historical personalities who participated in the establishment of the Principate, converting Rome from a republic into a monarchy ruled by an absolute emperor.  Although it is one of the best preserved large monuments from Rome, many tourists miss seeing it when visiting Rome.

 

 

Above is the Ara Pacis frieze showing figures who played key roles in establishing the rule of emperors in Rome.   

 

At left in the frieze, with his head hooded by a fold of his toga and powerful right forearm holding a scroll, stands the tall Marcus Agrippa, a commoner who became the close friend, son-in-law and right-hand man of Augustus Caesar.   Agippa became a successful general, an administrator with powers of a co-emperor, and a master builder of aqueducts, baths, temples and other public buildings.  Agrippa built the Maison Carrée in Nîmes, France, as shown in the Notes to the Example: Create Maps topic.

 

Agrippa wrote on geography and created one of the earliest charts of the Empire.  He established the length of his own foot as the Roman Foot, an official unit of measure, in turn defining the Roman mile as 5000 units the length of Agrippa's foot.  Agrippa is shown with his head covered, since by the time the Ara Pacis was completed he had died, relatively young at age 51, from disease after hard campaigning in winter weather.

 

The identity of the child holding Agrippa's robe is a matter of speculation.  In front in the center of the frieze is a woman who is most likely Julia, Augustus' licentious daughter by his first marriage, or possibly Livia, his scheming and overbearing second wife of fifty one years.   The figure in the middle at right is Tiberius, Augustus' stepson, who succeeded Augustus as the next emperor.

 

After the early death of her first husband, Julia was married to Agrippa, twenty five years her senior, in a marriage arranged to bind Agrippa within Augustus' immediate family.  She bore five children with Agrippa, including the mother of the future emperor Caligula, the grandmother-in-law of the emperor Claudius and the great-grandmother of Nero.  

 

After Agrippa's early death, Julia was married to her step-brother Tiberius, who she betrayed in an endless series of affairs, ultimately being divorced and exiled by Augustus.  Despite his deep love for his daughter, Augustus could not allow such infidelity in his immediate family.  Julia died of starvation while in captivity after Tiberius became emperor, exacting his revenge on his former wife for her infidelity.

 

A talented general earlier in life, Tiberius became a gloomy, isolated, reclusive, and vengeful emperor, hating his domineering mother, Livia, who had outlived Augustus by many years.   He abandoned Rome to live in secluded depravity on the island of Capri for over ten years at the end of his reign.   Tiberius was succeeded as emperor by the infamous Caligula, the grandson of Julia and Agrippa.    

 

See Also

File - Create - New Data Source - an absolutely key topic for connecting to many thousands of different web servers.

 

Web Servers

 

ArcGIS REST Servers

 

CSV Servers

 

Custom Servers

 

GeoJSON Servers

 

Image Servers

 

JSON Servers

 

OSM Servers

 

TMS Servers

 

WFS Servers

 

WMS Servers

 

WMTS Servers

 

PBF .pbf, OSM, O5M

 

Example: Spectacular Images and Data from Web Servers - A must see topic providing a gallery of views illustrating how Manifold can use web servers such as imageservers and other free resources to provide a seemingly endless selection of spectacular background maps, satellite images and GIS data with nearly zero effort.

 

Example: An Imageserver Tutorial - An extensive tutorial showing step by step how to add new data sources that are imageservers, how to show them as layers in a map, how to create a new drawing that matches the projection of the map and how to trace over what is seen in an imageserver layer to create an area object in the drawing.

 

Example: Vector Layers from an ArcGIS REST Feature Server - Visit an ESRI web site, copy a URL, and then use that URL to connect to an ArcGIS REST web server that shows petroleum fields in Kansas, getting the data as a vector drawing layer. Style the layer as if it were local.  ESRI refers to ArcGIS REST servers that provide vector data as feature servers.  

 

Example: Connect to a WFS Server for State Government Data - Gathering our courage, we connect to a WFS server that provides 1200 vector layers, run by the state of Massachusetts.   We open a layer showing airports and then scrape the vector data into our own local storage.  

 

Example: Connect to a WMS Server for National Map Layers - Visit the National Map services web page, copy a URL for a shaded relief layer from USGS, and then use Style to enhance that shaded relief data for combination with other layers and really spectacular effects.

 

Example: Connect to a Custom Server for Cadastral Data - We connect to a custom image server that provides cadastral information originally from the French national cartographic agency, IGN.  We create a map and use the Style pane to re-style the web served image on the fly into a more usable form.

 

Example: Connect to a Custom OpenRailwayMap Server - We connect to a custom server that provides an OpenRailwayMap view of railroads worldwide, showing railway, tram, and subway infrastructure based on OpenStreetMap data.  Our first try at creating a data source does not work.  After consulting the Log Window we try again with a slight adjustment and our second try works.

 

Example: Connect to an OSM Vector Server - We connect to an OSM Server that provides a vector layer containing points and lines in the OpenStreetMap database.  We then show how to scrape (copy) data from the OpenStreetMap server into local storage.  We extract building footprints from the local copy.

 

Example: Raster Layers from an ArcGIS REST Image Server - Visit an ESRI web site, copy a URL, and then use that URL to connect to an ArcGIS REST web server that provide a raster layer showing a mosaic of aerial photographs near Portland, Oregon.

 

Example: Create a New Data Source from a MAP File - Create a new data source from an existing Manifold .map project file.   This is the classic way to nest projects, one calling another, to create libraries of data and projects.   Access to nested projects has effectively zero performance loss and the links within projects take up effectively zero space so we can create huge constellations of data at our fingertips.

 

Example: Create a Data Source within an Existing Data Source - When a data source is writable, for example, if the data source is a Manifold .map file, we can work within that data source as if it were at the top level of our project.   For example, we can create a new data source that is nested within the existing data source.   This example shows how.

 

Example: Create and Use New Data Source using an MDB Database - This example Illustrates the step-by-step creation of a new data source using an .mdb file database, followed by use of SQL.  Although now deprecated in favor of the more current Access Database Engine formats, .mdb files are ubiquitous in the Microsoft world, one of the more popular file formats in which file databases are encountered.  

 

Example: Link GPKG and Save Style - A companion topic to the GPKG topic.   How to link a GPKG, open a drawing, Style it and then save so the styling is retained within the GPKG file.

 

Example: Trace an Area in a Map over an Image Background - In a map with a drawing layer above an image layer (served dynamically by an imageserver), create an area object in the drawing by tracing over the outlines of something seen in the image layer below

 

Example: Style Applied to an Image Server Image - Because the Style dialog simply changes the way an image is displayed and not the data, it can operate on read-only data served by various web servers such as WMS REST servers.    In this example we look at every detail of creating a data source using a WMS REST  server and then manipulating the appearance of the display with Style.  We will connect to a WMS server that provides LiDAR data in various forms, including as terrain elevation.