Click the web server illustrations in this topic to launch a full-sized image in a new browser window.
Manifold has the ability to bring data into a project from a seemingly endless range of different types of data sources, including almost any file format or database management system.
In addition, Manifold also has the ability to reach out through Internet to fetch spatial data, both vector drawings and raster images, from web servers that provide such data using a variety of well-known protocols.
Such web servers typically provide data dynamically to fill in whatever view we are looking at in a window that shows content from the web server. As we pan or zoom in or out in the view Manifold automatically tells the web server what view we are looking at and the web server responds by sending the data needed to fill in the view. Manifold begins by fetching tiles closer to the center of the requested area whenever possible, to produce a usable picture faster. We normally do not need to wait for the entire screen to fill before zooming in to what we can already see.
This topic discusses web servers and image servers. Image servers are a very popular form of web servers which use their own protocols and have become so famous, such as Google Maps, that they are worth supporting with pre-configured modules. Image servers are often referred to as imageservers. The illustrations in this topic show examples of different web servers and image servers.
Choose File - Create - New Data Source, and then choose More... if one of the Favorites in the initial list is not the desired choice.
Use the New Data Source dialog to configure a new image server or web server.
Launch Manifold.
Choose File - Create - New Data Source, or use the Project pane context menu.
Choose one of the favorites in the list, or press More... to launch the New Data Source dialog.
In the Type box choose Web Server: imageserver.
In the box just below the Type box choose the desired image server, for example, Bing Maps Street Map.
Press OK.
The new data source appears in the project. Click on the + icon to expand it.
Double-click on the image within to open it and see what the image server provides.
See File - Create - New Data Source for instructions on connecting to many different types of web servers, including custom web server connection strings.
For examples, see the Example: An Imageserver Tutorial topic or the Example: Create a New Data Source from a Manifold Image Server topic.
Manifold comes pre-loaded with several image servers in the Favorites
for data sources, typically Bing streets, Bing satellite, Google streets,
Google satellite and OpenStreetMap streets. To use one of those
we can choose File - Create - New Data
Source. and simply click on the image server desired from the list
of favorites that appears.
Web servers normally send either raster data such as images or they can send vector data that represent objects in drawings. When they serve raster data such data normally is sent in the form of tiles, rectangular groupings of pixels that are 128 x 128 pixels or 256 x 256 pixels or some other convenient size. Sending tiles through the web is much more efficient either than sending individual pixels or computing on the fly and sending a screen's worth of pixels. For that reason such web servers are often called tile servers.
Launch Manifold.
Choose File - Create - New Data Source, or use the Project pane context menu.
Choose one of the favorites in the list, or press More... to launch the New Data Source dialog.
In the Type box choose the desired web server technology, for example, Web Server: wms.
In the Source box enter the connection string / URL required for the desired server.
Press OK.
The new data source appears in the project. Click on the + icon to expand it.
Double-click on the image within to open it and see what the we server provides.
See File - Create - New Data Source for instructions on connecting to many different types of web servers, including custom web server connection strings.
For an example, see the Example: Style Applied to an Image Server Image topic.
Besides the obvious use of pixels and imagery in the form of tiles within web servers that provide imagery such as overhead satellite photos, as, for example, Google or Bing Satellite view do, web servers will often also use imagery to provide a visual representation of fundamentally vector data as well.
For example, street maps that show roads are maintained on the servers as drawings, that is, vector data. To show those street maps on the web, the server creates an image, like a screen shot, of the vector data complete with all formatting, labels and other accessory graphics arts wonderfulness that helps create an appealing map. The server then sends that imagery in the form of tiles out over the web.
Sending a visual representation of vector data out over the web in the form of imagery, that is, screenshots of the data, is a good thing for people who just want to look at the map. They can look at the map using ordinary web browsers or simple software on their telephones, tablets or computers that need not do anything more complicated than put together images from tiles that consist of well-known, simple image formats like jpg or png.
But that is not a good thing for people who want access to the actual vector data so they can use it for GIS purposes. For example, if we want the actual vector line objects that make up the boundaries of real estate parcels so we can accurately calculate the areas inside an image where the boundary lines are just fuzzy swaths of different-colored pixels (as they are in an image) is not the data we need. For such purposes there are protocols, such as WFS, that can be used by web servers to provide the actual vector data served dynamically.
Image servers, often written in concatenated form as imageservers, are a subset of web servers that serve imagery in a form supported by dedicated modules specific to each such image server. Such image servers typically do not utilize an "open" protocol but use their own specific protocols and methods to do something similar, serving tiles dynamically on demand to fill in a desired view.
Many image server modules are already built into Manifold. Choosing the Web Server: imageserver type in the New Data Source dialog when creating a data source provides a choice of popular image servers like Bing, Virtual Earth, Wikimapia, Yahoo!, Google Maps, Yandex and others.
Web servers are operated by third parties,
not by Manifold. The operators of web servers have their own
rules for free access (if any) to their servers, or how much free access
is allowed before we must start paying to use servers. Web server
operators often change their rules without notice. The built-in
list of popular web servers in Manifold is a convenience, not a guarantee
that those web server operators will continue to offer free access. Manifold
updates the list fairly frequently, but web server operators can change
their rules overnight.
Just like we must know the URL of a web site to connect to it, we usually must know the protocol utilized and the URL to the web server which provides the data we would like to use.
Some types of web servers, for example well-known image servers such as Google maps or Bing maps, are built into Manifold in the sense that their default connection strings are pre-loaded into the dialogs that invoke them. Adding dynamic imagery from such servers is as simple as choosing the desired image server from a list in a dialog. Other types of web servers, for example those utilizing wms protocol are more general and connect to a vast variety of different servers operated by various web sites, individuals and institutions. To use those we must note the connection string (a form of URL) required and enter it into the dialog.
Some web servers are free to use without any restrictions at all. Some utilize access controls such as the need to enter a special key or to incorporate login credentials as part of the connection string. Manifold dialogs assume the user is capable of visiting the desired web site to learn about any restrictions or special credentials required to access a given server.
Some web servers, often those which use WFS or similar to provide vector data, can be astonishingly slow. If no data appears when using such servers, wait for a minute or two and check the Log Window to see what is going on.
See File - Create - New Data Source for instructions on connecting to many different types of web servers, including custom web server connection strings.
The New Data Source dialog provides the option to Cache data when using a web server, including when using an image server.
Beginners should not check the
Saved cached data between sessions
box. That causes all tiles that have been fetched to be saved
into the project file, which can cause project files to become very large,
hundreds of megabytes, if we have browsed many locations with the image
server layer turned on.
Using cache by checking the Cache data box (checked by default) can provide faster performance with slower web servers because data that is reused, say, because we return to a previously-viewed scene, need not be re-fetched. If we are no longer connected to Internet or if the remote server goes down, the data source can still show what tiles have been downloaded into cache. Checking the Cache data box will save tiles that have been fetched during our current Manifold session into virtual memory. When we close the project or exit Manifold those saved tiles will be discarded.
Checking the Saved cached data between sessions box (not checked by default) tells the system to save all tiles that have been fetched into the project itself, stored in cache tables within the Cache folder in the System Data hierarchy. When we save the project into a .map file, those tiles will be saved within the .map project. If the web server has more recent data than the last time tiles were downloaded and saved, the new tiles will automatically be downloaded. Storing fetched tiles can be convenient, because whenever we use that web server in the project any view that we have already seen will be fetched instantly. If we want to use the web server, we can do so even if we no longer have an Internet connection, so long as we visit only those views (pan and zoom) that have already been seen.
If cached data saved by using the Saved cached data between sessions box is no longer desired, individual cache tables or even the entire Cache folder can be deleted to reduce the size of the project file. When the data source created with the Saved cached data between sessions box option enabled is opened again, the Cache folder and cache table for that source will be re-created so the cache can grow again.
The downside to saving cached tiles in the project, by using the Saved cached data between sessions box, is that cache tables can accumulate hundreds of megabytes of data within the project file. Therefore, beginners should not check the Saved cached data between sessions box, since it is easy to forget that caching data and then browsing can grow apparently small projects to large size. As a practical matter, most web servers used by most users tend to be image servers provided by large companies such as Microsoft Bing and Google, which provide tiles so rapidly that there is no need to use save cached data between sessions to achieve a reasonably speedy presentation.
See the discussion and illustrations in the File - Create - New Data Source topic.
Problems connecting - Check the Log Window to see what is going on behind the scenes if an attempted connection does not work. The problem is usually a wrong connection string, failure to provide required credentials such as an key string that authorizes access, wrong choice of protocol (the server uses WMS and the user picks something else), an incredibly slow server, a server that is offline or a server that is wrongly configured and which is not correctly using the protocol it claims to use.
Visit the Manifold community forum and talk out difficulties with other users. Make sure to post full information on what you are doing, the connection URL you used, all details of how you tried to connect (including all settings in the data source dialog), what happened, and what the Log Window reported. If other users cannot help you, spending a tech support incident will produce an authoritative analysis of the issue.
The very last thing to consider is the possibility that a bug in Manifold is causing the problem. Bugs happen, of course, but leaping to the conclusion as a first step that a bug is the problem is almost always a mistake.
Try the URL in a browser - Checking the URL by launching it in a browser can reveal many problems with the URL or with the web server. If a URL does not work in a Manifold web server dataport, try exactly the same URL in a browser. If a browser cannot connect to the URL, the Manifold web server dataport will not be able to connect to it either. If a browser cannot connect to that URL, that indicates the problem is the URL or the web server. For example, the web server might be offline. Or, for example, If the browser connects to a page other than the actual web service endpoint, such as, to a web page that lists various options for web servers, that shows the URL is not a URL for a web server but a URL to some other sort of web page.
Connection problems are often caused by incorrect URLs. There might be a typographical error in the URL or the URL might not be an endpoint to a functioning server but instead a URL to some other web page. The server responding to the URL may have geographic restrictions (surprisingly common) that does not respond to connections from IP addresses that are thought to be in a canceled country. Trying the URL in a browser will fail in such cases. Web servers may also have other restrictions, such as only allowing connections from white listed IP addresses, from paying clients, or from those clients that use a special security scheme.
URLs change frequently - Web server sites often change URLs. If a URL no longer exists for a data set of interest, use a search engine like Google with terms like the subject of the server, "ArcGIS REST" and so on to see if a newer URL is now available.
Use a correct URL - It is important to use a URL that does, indeed, connect to a functioning ArcGIS REST Server.
Web servers are often slow or mis-configured - Some web servers to which we might connect using WMS, WMTS, WFS or other connection standards can be incredibly unresponsive or, despite allowing a connection, will not allow data to be fetched. If pressing the Test button results in a successful connection, or if layers available are displayed when we expand the data source, but then when we try to display an advertised layer nothing happens, we can open the Log Window to see what is going on.
Here are a few lines of log window information for a happy session, connecting to a WMS server in New Zealand run by the ever-efficient LINZ organization:
Web request: (root)::[Data Source] (GetCapabilities) (0.578 sec, 13.0 KB)
Web request: (root)::[Data Source]::[layer-767] (GetMap) (1.255 sec, 47.9 KB)
Render: [Data Source]::[layer-767 Image] (1.561 sec @1.0)
In contrast, a few lines of log window information for an unhappy session, attempting to connect to the WMTS server run by the French cartographic service IGN but getting no image for a cadastral parcels layer:
Web request: (root)::[IGN]::[CADASTRALPARCELS.PARCELS_bdparcellaire_b] (GetTile) (0.070 sec, 0 B)
*** (root)::[IGN]::[CADASTRALPARCELS.PARCELS_bdparcellaire_b] (GetTile) The remote server returned an error: (403) Forbidden.
Render: [IGN]::[CADASTRALPARCELS.PARCELS_bdparcellaire_b Image] (0.315 sec @1.0)
From the above we see an empty cadastral parcels image was rendered because the remote server refused to provide tiles in response to the GetTile request, returning a 403 error Forbidden. Finding out why a web server refuses service can involve some detective work.
Keep in mind that just because a web server offers tiles via various standards like WMTS or whatever does not mean that it will be fast or even reasonably tolerable. Some web servers are painfully, incredibly slow and will take many seconds or even minutes per tile. When images do not appear or fill in with painful slowness a look at the Log Window can show the problem is not our Internet connection or Manifold, but just incredibly slow response from the web server.
Some WMS servers are erroneously configured so the server mislabels image tiles as text/xml. A WMS compliant client will reject such mislabeled tiles, but because webmasters often fail to test their sites using WMS compliant clients such mislabeling errors are frequent. To handle such errors, Manifold includes a simple defensive measure: When Manifold reads WMS tiles that a server labels as non-image data, the system will distrust the server and try common image formats. If the tiles are images the system will disregard the server's claim such tiles are text and instead will process them as images. Second-guessing such server errors allows Manifold to interoperate with wrongly-configured servers, albeit at the cost of some processing overhead when correctly working servers return text/xml data.
Some webmasters may insist their web servers are correctly configured even when explicit errors are pointed out. That can happen even in the case of national cartographic services as reported, for example, in threads in the georeference forum.
Web servers often require API keys for access - The server might require special credentials, such as a special key for access, a key we are trying to use may be out of date, or a login and password may be required for some resources. Google servers, for example, often require an API key for access, even if that access is provided for free. Many web server operators provide "developer keys" for reasonable, free use. See the web sites of the various server operators to read about their latest requirements.
Web servers may allow some amount of use for free before they start charging for service. Some "open" servers are only open until the server decides we have used it too much, after which it denies further access.
User-unfriendly server configurations - Correctly configured web servers may sometimes make user unfriendly configuration choices, for example, only starting to show data when the view is zoomed in to some specific area. That is a perfectly legal configuration, but it may catch users by surprise if they open a web served layer and it is blank, because they have not yet zoomed sufficiently far into a particular region. If a web server layer is blank, add it to a map with a "known good" background layer, such as Bing Streets, and then zoom into the area of interest that is likely to be covered. The Geosciences Australia web server, for example, often serves blank layers until one zooms into Australia, and at times far into Australia.
Must Have Internet - It seems obvious that to connect to a web server we must have a live, functioning Internet connection. What might not be obvious is that sometimes we may have only a partial Internet connection, without realizing we do not have a full Internet connection. A classic example is using a browser, such as Opera, which may ignore failed IPv6 DNS resolution but work OK using IPv4 DNS resolution. Without IPv6 DNS resolution we can browse Internet using Opera but we might not be able to connect to Microsoft, Google, or some other web servers.
Manifold's web server connections, just like the latest Google Chrome and Microsoft Edge browsers, require full IPv6 DNS resolution. That normally happens automatically using whatever IPv6 DNS servers are assigned by our Internet service provider, but at times IPv4 DNS continues to work OK while the IPv6 DNS servers assigned by our ISP do not work. If we are using Opera, and not Edge or some other browser that uses IPv6 DNS resolution, we might not notice that our ISP-provided IPv6 DNS resolution is not functioning. We might only notice the problem when trying to use an image server or other web server within Manifold and it does not work.
If we cannot get a display from a web server, we can check the Manifold Log Window. If the log reports that a connection cannot be made, in Windows network dialogs for our Internet connection we can try setting the IPv6 DNS servers to some "known good" DNS servers. A good choice is using Google's public DNS service, using the following addresses for primary and backup DNS server addresses:
2001:4860:4860::8888
2001:4860:4860::8844
Collections of Web Servers - Check the Product Downloads page for pre-packaged project files in Release 9 / Viewer .map format that Manifold publishes which contain collections of dozens of popular web servers data sources.
Embedded command tokens: Some web servers, notably TMS servers, use embedded command tokens. Use Manifold's custom setting to connect to those. See the File - Create - New Data Source topic for how to do that.
Imageserver or Image Server? - Manifold uses two styles for writing the term, with and without a space character. The two styles as synonyms without any special meaning attached to one or the other. For historical reasons going back to prior Manifold releases the term imageserver tends to be used in programmatic contexts. For discussions in ordinary text and spoken language the term image server tends to be used.
Old versions of WMS - Connecting to a WMS server always uses XY axis ordering based on the assumption that servers using old versions (pre-1.1.1) of the WMS standard ignore axis order for the coordinate system.
Stealth WMS server errors - The WMS dataport detects server errors that are not flagged as errors by the server and reports them in the log window. For example, some WMS servers announce the use of a specific format for tiles but then fail to use that format or will send no tile, failing to announce (as the standard requires) an error in sending no tile. Manifold will log such server errors, which can be very useful to debug server errors or when informing webmasters their server is misconfigured.
Handling misconfigured servers - The WMS dataport performs additional filtering of layer coordinate systems to better work with servers that are misconfigured or fail to conform to the WMS standard. For example some WMS servers report bounding box data for a layer in a coordinate system that the server does not support, but then will advise in layer information the use of a different coordinate system that is supported. Manifold patiently tries various possibilities to work around such server errors, to enable more use of misconfigured servers.
Additional Tips - Make sure to review the topic for the web server you are using in the Web Servers book within the Formats and Data Sources book of the user manual.
File - Create - New Data Source - an absolutely key topic for connecting to many thousands of different web servers.
Favorite Data Sources - View and edit the Favorites dialog for data sources, showing various data sources that may be added to the project with a single click from a list of favorites.
Example: Spectacular Images and Data from Web Servers - A must see topic providing a gallery of views illustrating how Manifold can use web servers such as image servers and other free resources to provide a seemingly endless selection of spectacular background maps, satellite images and GIS data with nearly zero effort.
Example: An Imageserver Tutorial - An extensive tutorial showing step by step how to add new data sources that are image servers, how to show them as layers in a map, how to create a new drawing that matches the projection of the map and how to trace over what is seen in an image server layer to create an area object in the drawing.
Example: Create a New Data Source from a Manifold Image Server - Manifold image server modules are snippets of code which use the Manifold Image Server Interface (ISI) to automatically fetch image tiles from popular image servers like Virtual Earth, Wikimapia, Yahoo!, Google Maps, Yandex and many others. Image servers can provide street maps, overhead satellite imagery, combinations of streets and satellite imagery and other data as well. Using Manifold Image Servers is one of the most popular Manifold features.
Example: Vector Layers from an ArcGIS REST Feature Server - Visit an ESRI web site, copy a URL, and then use that URL to connect to an ArcGIS REST web server that shows petroleum fields in Kansas, getting the data as a vector drawing layer. Style the layer as if it were local. ESRI refers to ArcGIS REST servers that provide vector data as feature servers.
Example: Connect to a WFS Server for State Government Data - Gathering our courage, we connect to a WFS server that provides 1200 vector layers, run by the state of Massachusetts. We open a layer showing airports and then scrape the vector data into our own local storage.
Example: Connect to a WMS Server for National Map Layers - Visit the National Map services web page, copy a URL for a shaded relief layer from USGS, and then use Style to enhance that shaded relief data for combination with other layers and really spectacular effects.
Example: Connect to a Custom Server for Cadastral Data - We connect to a custom image server that provides cadastral information originally from the French national cartographic agency, IGN. We create a map and use the Style pane to re-style the web served image on the fly into a more usable form.
Example: Connect to a Custom OpenRailwayMap Server - We connect to a custom server that provides an OpenRailwayMap view of railroads worldwide, showing railway, tram, and subway infrastructure based on OpenStreetMap data. Our first try at creating a data source does not work. After consulting the Log Window we try again with a slight adjustment and our second try works.
Example: Connect to an OSM Vector Server - We connect to an OSM Server that provides a vector layer containing points and lines in the OpenStreetMap database. We then show how to scrape (copy) data from the OpenStreetMap server into local storage. We extract building footprints from the local copy.
Example: Create a New Data Source from a MAP File - Create a new data source from an existing Manifold .map project file. This is the classic way to nest projects, one calling another, to create libraries of data and projects. Access to nested projects has effectively zero performance loss and the links within projects take up effectively zero space so we can create huge constellations of data at our fingertips.
Example: Create a New Data Source from a Web Server - Many web servers are image servers, providing image. tiles for a geographic view to browsers or to applications. Typical image servers are Bing, Google Maps, Yandex and many others. Image servers can provide street maps, overhead satellite imagery, combinations of streets and satellite imagery and other data as well. This topic shows how to create a new data source that is an image server. Using image servers is one of the most popular Manifold features.
Example: Create a Data Source within an Existing Data Source - When a data source is writable, for example, if the data source is a Manifold .map file, we can work within that data source as if it were at the top level of our project. For example, we can create a new data source that is nested within the existing data source. This example shows how.
Example: Trace an Area in a Map over an Image Background - In a map with a drawing layer above an image layer (served dynamically by an image server), create an area object in the drawing by tracing over the outlines of something seen in the image layer below
Example: Style Applied to an Image Server Image - Because the Style dialog simply changes the way an image is displayed and not the data, it can operate on read-only data served by various web servers such as WMS REST servers. In this example we look at every detail of creating a data source using a WMS REST server and then manipulating the appearance of the display with Style. We will connect to a WMS server that provides LiDAR data in various forms, including as terrain elevation.