File - Create - New Data Source

The New Data Source dialog allows us to create a data source within a project.    The data source can be created within the existing .map project or within another, nested data source that already exists within the project.  

 

Important: New Data Sources are linked, not Imported, into the project.  See the Importing and Linking topic.

 

A new data source links data into our project without importing the data.    The data in that new data source remains stored within whatever is the originating source.  See comments on linking data into a project in the Getting Started topic.  

 

Creating a new data source is a more sophisticated way of linking data.   It uses a more comprehensive dialog than a simple File - Link and so is the way we link data from databases, where optional settings such as login or password for the database might be required.

 

 

Choose File - Create - New Data Source, and then choose More... if one of the data sources in the Favorites list is not desired.

 

 

Use the New Data Source dialog to configure a new data source.

 

 

To link data as a new data source:

 

  1. In the main menu choose File - Create - New Data Source, and then More... if one of the Favorites is not desired.

  2. In the New Data Source dialog provide a Name for the new data source if the default is not desired.

  3. Choose the Type of data source from the long list available.

  4. Choose the sub-type of data source when those are available.

  5. The Source box will be automatically loaded for some data sources, for example, connection URLs to Bing or Google servers.   For other sources, use the [...] browse button to navigate to the file or data source connection as required.

  6. Set option boxes that appear as desired, for example, checking Open as read-only if the data source should be kept read-only.

  7. Press Create Data Source.

 

See the File - Create - New Data Source topic for details.

 

For an example of creating a new data source see the Getting Started topic as well as the many examples recommended at the end of this topic.

 

A new data source can be created from data that is resident in hundreds of different type of data sources.  The New Data Source dialog organizes data sources as four different Types, depending on what we specify to access the data source, to make them easier to find in a long list of available types:

 

 

 

 

 

 Web servers are operated by third parties.  Published URLs might not be functional, or the owner of the web server may place limitations on use of the web server, for example, requiring a login and a password, or requiring an API key, a code that authorizes access.  Google, for example, may allow some amount of free use of their image servers, but then may cut off further use until an API key is procured from Google and used.  See the discussion in the Web Servers section below.

 

Connections to the above are provided within Manifold by dataports, which are modules of code within Manifold that provide connection services to different data sources.   Dataports are always at least read-only and will often also provide read/write capability.

 

There may be more than one way to connect to a data source.   For example, we can connect to Microsoft Access MDB file databases directly from a Manifold session or we can connect using the generic OLE DB connection.  Normally we would connect directly using the MDB File Database option but if the MDB file has a password for access we can connect using the OLE DB option, which allows specifying a password.

 

Note that we can create a new data source that links a script into the project.  This is a handy way of including within projects scripts that might reside outside of a project, be used by other programs, edited or updated by other users and so on.   

 

We can also create a new data source, using File: txt that links text files into a project to create comments components.  These are useful when documenting information about other data, sharing "to do" lists and other team-oriented comments and other text documents that might be edited by other users using other programs or created automatically by other processes.

Databases

 

For important notes when connecting to DBMS data sources, see the DBMS Data Sources - Notes topic.

 

Name

Name for the new data source, "Data Source" by default.  Specify a more  memorable name as desired.  If we forget the origin of a data source we can hover the mouse over the data source and a tool tip will provide connection information.

Type

Choose from the list of different types of Database: data sources that may be connected.

 

A generic connection technology such as ADONET will enable a secondary box below the Type box when Database: adonet is selected as the Type.   That secondary box will be loaded with a list of connection options made available by any DLLs within the PATH supporting ADONET connections.  If no such DLLS exist the list will be empty.

Source

A connection string or path to a file or other resource to open.   The source string can be manually entered or it can be created using dialogs launched by the [...] browse button or it may be automatically populated for some types of data sources.

 Browse button

Click to populate the Source box.  When choosing a database Type that consists of a generic connection technology such as ODBC or OLEDB the [...] Browse button will launch the Windows dialog for that connection technology, for example, launching the Windows Data Link Properties dialog for OLEDB or the Windows Select Data Source dialog for ODBC.

Open as read-only

Open the data source read-only.  Not checked by default so that if the dataport for the data source supports a read/write connection by default the data source will be both readable and writable.   Some types of data sources, such as enterprise class DBMS servers like Oracle, will allow multiple users to connect read/write to a data source.  Other types of data sources, such as most file formats, will lock out other users if we connect to them read/write.  Therefore, if we would like to allow other users to be able to simultaneously connect to this data source and we do not need to write to it, we can check this box.  We can also check the read-only box if we would like to protect against accidental changes to the data source.

Use vector data as either XY or YX according to coordinate system

This option appears for those data sources where coordinate systems can be in either ordering.   Standards such as EPSG for specifying coordinate systems have codes which sometimes specify that data occurs in XY form, that is longitude, latitude coordinate ordering, and sometimes as YX form, that is, latitude, longitude coordinate order.   In theory, this box should always be checked.  It is not checked by default because, unfortunately, so many popular packages disregard EPSG standards that most spatial data encoded with EPSG codes which require flipped coordinate ordering is not flipped.  See the That YX Thing essay topic.

Create Data Source

Create the new data source in the project pane and close the dialog.

Edit Query

Launch the Command Window loaded with a query that creates the data source using the given settings.  A great way to learn how to use SQL to create data sources.

Cancel

 Exit the dialog without doing anything.

Database Login Dialog

When a Database: data source is the Type, the [...] button next to the Source box launches the Database Login dialog that allows specifying a server and connection characteristics.

 

 

Server

The connection string for the database server.  This is simply an IP address with optional port for Manifold Server.

Use login and password

Use security credentials specified in the Login and Password boxes, which are enabled when this option is selected.  If not checked, no login and no password is used, with Windows integrated security credentials used if relevant.

Database

Specific database to use in this server.  Enabled for database servers such as PostgreSQL, SQL Server, and Manifold Server that allow specifying a database within the database server.

Test

Press to try a connection to the server using the specified settings.

Specifying ports when connecting to Database data sources

When connecting to a DBMS we can specify the port to use either in the connection string or within the name of the server after a colon.   For example, when connecting to a PostgreSQL server we can write for the name of server:

 

localhost:5433

 

The dialog will know we are connecting to PostgreSQL, a database that allows explicit specification of a port to use for the connection, and it will split out the port number from the name automatically.   Alternatively we could explicitly specify the port as part of the connection string, in the form:

 

{ "Source": "host=localhost port=5433 user=postgres password=xxxx dbname=xxxx" }

Files

 

Name

Name for the new data source, "Data Source" by default.  Specify a more  memorable name as desired.  If we forget the origin of a data source we can hover the mouse over the data source and a tool tip will provide connection information.

Type

Choose from the list of different types of File: data sources that may be connected.  .map project files are of type File: manifold.

Source

A connection string or path to the file or other resource to open.   The source string can be manually entered or it can be created using dialogs launched by the [...] browse button.

 Browse button

Click to populate the Source box.  Browse the Windows file system to locate and open the desired file using a Select File dialog modeled on the standard Windows File - Open dialog.  The type of file will be pre-loaded into the dialog depending on what was selected in the Type box

Open as read-only

Open the data source read-only.  Not checked by default so that if the dataport for the data source supports a read/write connection by default the data source will be both readable and writable.   Most file formats, will lock out other users if we connect to them read/write.  Therefore, if we would like to allow other users to be able to simultaneously connect to this data source and we do not need to write to it, we can check this box.  We can also check the read-only box if we would like to protect against accidental changes to the data source.

Fault-tolerant save

Appears when File: manifold is selected in the Type box.   When checked (the default) the system will create a .SAVEDATA accessory file and will use fault tolerant saves if a .map file is connected as a new data source.   See comments at the end of the Getting Started topic and in the Accessory Files Created topic.

Cache data

Create and utilize an accessory .MAPCACHE file for improved performance. Also provides greater flexibility with read-only data sources.  Checked by default.  This option appears with most file data sources and has two options, both checked by default, one to cache only data that is expensive to dynamically compute and the other to save cached data between sessions.  Uncheck the latter two options to reduce the size of project files, but it is a good idea to leave the Cache data box checked.  Caching data also allows flexibility with Style when linking data from read-only files.  For example, some TIFF images linked from read-only .tif files might not display if Save cache is not checked.

Save cached data between sessions

Appears as a sub-choice for some data sources for which the Cache data option appears.   Save the cached data for the next time this project is opened, either within the .MAPCACHE file or within the .map project itself within a Cache subfolder within the System Data hierarchy.

Cache only data that is expensive to compute dynamically

Appears as a sub-choice for some data sources for which the Cache data option appears.   If there is significant overhead in computing data to be displayed, cache the results.  For example, some file formats may require complex calculations to extract data from the format.  It is quicker to do that once and save the result for re-use than to repeatedly perform the same computation/extraction.

Create Data Source

Create the new data source in the project pane and close the dialog.

Edit Query

Launch the Command Window loaded with a query that creates the data source using the given settings.  A great way to learn how to use SQL to create data sources.

Cancel

 Exit the dialog without doing anything.

 

Folders

 

Name

Name for the new data source, "Data Source" by default.  Specify a more  memorable name as desired.  If we forget the origin of a data source we can hover the mouse over the data source and a tool tip will provide connection information.

Type

Currently supports only the Folder: library type, to support Release 8 image libraries.

Source

The path to the image library folder, along with a template for the file name pattern that includes [X] and [Y] specifiers.   For example, [X]-[Y].* will match image files using names like [01]-[01].png, [01]-[02].png, [02]-[01].png, and so on.

 Browse button

Click to populate the Source box.  Often used to launch the File - Open dialog to open a file format or file database format.

Open as read-only

Open the data source read-only.  Not checked by default so that if the dataport for the data source supports a read/write connection by default the data source will be both readable and writable.   Most file formats, will lock out other users if we connect to them read/write.  Therefore, if we would like to allow other users to be able to simultaneously connect to this data source and we do not need to write to it, we can check this box.  We can also check the read-only box if we would like to protect against accidental changes to the data source.

Cache data

Create and utilize an accessory .MAPCACHE file for improved performance. Also provides greater flexibility with read-only data sources. Checked by default.  This option appears with most file data sources and has two options, both checked by default, one to cache only data that is expensive to dynamically compute and the other to save cached data between sessions.   Uncheck the latter two options to reduce the size of project files, but it is a good idea to leave the Cache data box checked.

Save cached data between sessions

Appears as a sub-choice for some data sources for which the Cache data option appears.   Save the cached data for the next time this project is opened, either within the .MAPCACHE file or within the .map project itself within a Cache subfolder within the System Data hierarchy.

Cache only data that is expensive to compute dynamically

Appears as a sub-choice for some data sources for which the Cache data option appears.   If there is significant overhead in computing data to be displayed, cache the results.  For example, some file formats may require complex calculations to extract data from the format.  It is quicker to do that once and save the result for re-use than to repeatedly perform the same computation/extraction.

Recurse into subfolders

If checked, recursively scan folders below the location named in Source to find more image files.   If not checked, the dataport will use only image files found in the Source folder.

Reverse Y

The X axis is always numbered in increasing order from left to right.  By default, Y axis ordering is numbered in increasing order from bottom left upwards.   Checking this box specifies Y axis numbering in increasing order from upper left downwards.

Cut margins

The number of pixels to clip from left, right, top and bottom margins. Used when image tiles have borders that should be removed.

Create Data Source

Create the new data source in the project pane and close the dialog.

Edit Query

Launch the Command Window loaded with a query that creates the data source using the given settings.  A great way to learn how to use SQL to create data sources.

Cancel

 Exit the dialog without doing anything.

 

Web Servers

Web servers are operated by third parties, not by Manifold.  They vary wildly in their terms of service, their quality, and even in whether they are running or not.   URLs and connection strings used in the Source box can change from day to day, and may be different than the defaults used by Manifold.  Web servers are frequently misconfigured, even by national cartographic services (as reported, for example, in this thread in the georeference forum).   Web servers might be so painfully slow it seems they are not working

 

Correctly configured web servers may sometimes make user unfriendly choices as well, for example, only starting to show data when the view is zoomed in to some specific area.  If a web server layer is blank, add it to a map with a "known good" background layer, such as Bing Streets, and then zoom into the area of interest that is likely to be covered.  The Geosciences Australia web server, for example, serves blank layers until one zooms into Australia, and at times far into Australia.

 

 

Name

Name for the new data source, "Data Source" by default.  Specify a more  memorable name as desired.  If we forget the origin of a data source we can hover the mouse over the data source and a tool tip will provide connection information.

Type

Choose from the list of different types of Web Server: data sources that may be connected.  

 

Choosing Web Server: geocodeserver in the Type box will enable a secondary box below the Type box that will be loaded with a list of built-in geocoding servers, as of this writing: Bing Geocoder, Google Geocoder, Here Geocoder, Here Landmark Geocoder, Here Reverse Geocoder, MapBox Geocoder, MapQuest Geocoder, OpenCage Geocoder, and Yandex Geocoder.

 

Choosing Web Server: imageserver in the Type box will enable a secondary box below the Type box that will be loaded with a list of built-in image server, as of this writing: Bing Maps Hybrid, Bing Maps Satellite, Bing Maps Street Map, Google Maps Satellite, Google Maps Street Map, Google Maps Street Map (Transparent), Google Maps Terrain, Google Maps v2 Hybrid, Google Maps v2 Satellite, Google Maps v2 Street Map, Google Maps v2 Terrain, Here Maps Base, Here Maps Hybrid, Here Maps Satellite, Here Maps Terrain, MapBox Hybrid, MapBox Satellite, MapBox Street Map, MapQuest Hybrid, MapQuest Satellite, MapQuest Street Map, NearMap Street Map, NearMap Street Map (Direct),  OpenStreeMaps Base, OpenStreeMaps Migurski's Terrain, OpenStreeMaps OpenCycleMap, OpenStreeMaps OpenCycleMap Transport, WikiMapia Street Map, WikiMapia Street Map (Transparent), Yandex Maps Satellite, Yandex Street Map, Yandex Maps v2 Hybrid, Yandex  Maps v2 Satellite, Yandex Maps v2 Street Map, Yandex Maps v2 Terrain, Custom.

 

Choosing a built-in geocode server or image server from the list will automatically load the connection string into the Source box.

Source

A connection string or path to the file or other resource to open.   The source string can be manually entered, or it will automatically be populated for Web Server: geocodeserver and Web Server: imageserver types.  The connection string can also be entered using the Web Login dialog launched by the [...] browse button.

 Browse button

Click to launch the Web Login dialog, to allow use of a login and password plus use of a proxy server if desired.   The Web Login dialog is also handy for providing a Test button that can be used to test the connection.

Open as read-only

Open the data source read-only.  Usually has no effect with web servers since they are read-only in any event.

Cache data

Cache data downloaded from the server while the project is open.  Provides better interactive performance and greater flexibility with read-only data sources.

Save cached data between sessions

Save cached data for the next time this project is opened.  Cached data is saved within  a Cache sub-folder in the System Data hierarchy.   Caution: checking this box can result in very large .map files when the image results of browsing web servers are all saved.  However, having such data cached in the .map is handy for offline browsing of the project.

API key

Provide a key that authorizes use of an API when connecting to a proprietary data source that requires such a key.

Create Data Source

Create the new data source in the project pane and close the dialog.

Edit Query

Launch the Command Window loaded with a query that creates the data source using the given settings.  A great way to learn how to use SQL to create data sources.

Cancel

 Exit the dialog without doing anything.

 

Web Login Dialog

When a Web Server: data source is the Type, the [...] button next to the Source box launches the Web Login dialog that allows specifying a server and connection characteristics.

 

 

Server

The connection string for the server.  This may be a simple URL or a very lengthy URL/connection string that embeds parameters such as keys that grant access or other parameters

Use login and password

Check this box for servers that require logging in with a login name and a password, providing the required name and password in the Login and Password boxes.

Use proxy server

Check this box when connecting through a proxy server.   The Proxy, Login, and Password boxes allow specifying the connection string to the proxy server as well as the login name and password required to use the proxy server.

User agent

Identifies what application (Manifold) is asking for a connection.  Some web servers want to know what client software is connecting, for compatibility or for business reasons.  The default string optimizes compatibility (Mozilla is very generic) while also identifying Manifold Release 9 as the client.  Users can adjust the string as necessary to comply with any special server requirements.

API key

Provide a key that authorizes use of an API when connecting to a proprietary data source that requires such a key.

Application key

A secondary application key or authentication code for those servers, such as here.com (also known as wego.here.com) that may require it.

Timeout

Specify a time in milliseconds to wait for connecting to the specified server.  Use 0 for the default timeout or specify whatever is the desired time to wait before giving up on the server.

Test

Press the Test button to try the connection using the specify parameters.   If successful, a Connection established information dialog will pop open.

Server Connection String

Server connection strings can contain much specialized information, such as required keys.  For example, a typical string to connect to a WMS server that provides topographic maps from New Zealands's LINZ / National Topographic office might be:

 

https://data.linz.govt.nz/services;key=ca62cf6a884a4f0cbfe5/wms/layer-767?service=WMS&version=1.1.1&request=GetCapabilities

 

We enter that into the Server box:

 

 

(The above connection string has been edited so the key shown above is not accurate:  anyone can get their own key by registering on the LINZ website.)

Custom Web Servers

When choosing a Type of Webserver: imageserver, we can choose the Custom subtype to utilize a very wide range of web servers that provide image tiles using connection strings that contain some more-or-less typically encountered, but far from standard, control sequences.    Control sequences are tokens within curly { } brackets that are substituted during the action of the server with values that control the result.    To learn how a specific server uses control sequences, as well as learning other required information such as the tile size used and the zoom range supported, we must consult whatever documentation the server operator has provided.  

 

 

Name

Name for the new data source, "Data Source" by default.  Specify a more  memorable name as desired.  If we forget the origin of a data source we can hover the mouse over the data source and a tool tip will provide connection information.

Type

Choose Web Server: imageserver and Custom in the secondary box below.

Source

A URL connection string that contains control sequences used by the server.  

 Browse button

Click to help populate the Source box.  Clicking the [...] Browse button will launch the Web Login dialog, to allow use of a login and password plus use of a proxy server if desired.   The Web Login dialog is also handy for providing a Test button that can be used to test the connection.

Open as read-only

Open the data source read-only.  Usually has no effect with web servers since they are read-only in any event.

Cache data

Cache image tiles downloaded from the server while the project is open,  saved within the .map project itself in a Cache subfolder in the System Data hierarchy. Also provides greater flexibility with read-only data sources.

Save cached data between sessions

Save the cached data for the next time this project is opened,  within the .map project itself in a Cache subfolder in the System Data hierarchy.  Caution: checking this box can result in very large .map files when the image results of browsing web servers are all saved.  However, having such data cached in the .map is handy for offline browsing of the project.

Tile size

Size of tiles issued by the server.  Always square, so 256 means tiles that are 256 pixels wide by 256 pixels high.  Some servers use 512 pixel tiles.   See the documentation for the server.

Zoom Range

0 is zoomed all the way out and 20 is zoomed all the way in to the closest zoom served by the server.  The closest zoom varies by server and must be entered correctly.   See the documentation for the server.

API key

Provide a key that authorizes use of an API when connecting to a proprietary data source that requires such a key.

Application key

A secondary application key or authentication code for those servers that require it.

Create Data Source

Create the new data source in the project pane and close the dialog.

Edit Query

Launch the Command Window loaded with a query that creates the data source using the given settings.  A great way to learn how to use SQL to create data sources.

Cancel

 Exit the dialog without doing anything.

Control Sequences

The Custom imageserver dataport understands the following control sequences, as typically used:

 

{x}, {y}

X and Y coordinates of a tile

{zoom}

Zoom level of a tile.  Zoom level is a numeric value similar to an image level but going in the reverse direction: an image level of 0 is the most detailed level, while a zoom level of 0 is the least detailed level supported by the server.   Some URL connection strings specified for servers use the control sequence token {z}.  Replace {z} with {zoom} in the URL to construct a URL to use as the  Source string.

{key}

API key, if entered by the user.

{keyApp}

Application key, if entered by the user.

{lon}, {lat}

Longitude and latitude of the center of the requested tile.

{switch:a,b,c}

Server selector for load balancing: Choices after the semicolon are available alternatives.

Examples

Dozens of examples using the Custom dataport have been published in the JOSM sources.map project that is available on the Release 9 Downloads page on the Manifold website.  Most of the web server data sources in that project are created from strings published in the JOSM sources page at https://josm.openstreetmap.de/wiki/Maps page, from which URL strings may be copied for use in the Source box of the Custom web server data source creation dialog, along with other parameters such as maximum zoom.  

 

Following are some examples of entries for various servers from the JOSM sources page.   These show selected tags and not all information for what sometimes are very long entries.   See the JOSM sources page for complete info.

 

OpenStreetMap using French cartographic style:

 

        <name>OpenStreetMap (French Style)</name>

        <type>tms</type>

        <url>http://{switch:a,b,c}.tile.openstreetmap.fr/osmfr/{zoom}/{x}/{y}.png</url>

        <max-zoom>20</max-zoom>

 

OpenStreetMap using German cartographic style:

 

        <name>OpenStreetMap (German Style)</name>

        <type>tms</type>

        <url>http://{switch:a,b,c,d}.tile.openstreetmap.de/tiles/osmde/{zoom}/{x}/{y}.png</url>

        <max-zoom>18</max-zoom>

 

In both the above cases we can copy the string between the <url> and </url> tags to use as our Source string.  In the case of the German style version we must change the Zoom range value in the New Data Source dialog from [0, 20] to [0, 18]

 

OpenTopoMap

 

        <name>OpenTopoMap</name>

        <type>tms</type>

        <url>http://tile.opentopomap.org/{zoom}/{x}/{y}.png</url>

        <min-zoom>3</min-zoom>

        <max-zoom>17</max-zoom>

 

We can copy the string between the <url> and </url> tags to use as our Source string.  We must change the Zoom range value in the New Data Source dialog from [0, 20] to [3, 17]

 

OpenCycleMap

 

        <name>OpenCycleMap</name>

        <type>tms</type>

        <url>https://{switch:a,b,c}.tile.thunderforest.com/cycle/{zoom}/{x}/{y}.png?apikey=7bd5ed2197cf4da29fa26de0ba6530cc</url>

        <max-zoom>22</max-zoom>

 

An example using an embedded API key, apparently issued by the originator for use by people creating features for OpenStreetMap.  We can copy the string between the <url> and </url> tags to use as our Source string.  We must change the Zoom range value in the New Data Source dialog from [0, 20] to [0, 22]

 

Some of the web servers in the JOSM sources.map project are licensed only for creating features in OpenStreetMap while others are completely unrestricted.  See the comments in the project for links that give complete info.

Cache Data Option

When creating new data sources for a web server data source we can check or uncheck the Cache data box (checked by default)  in the New Data Source dialog.   This option merits attention when creating data sources from which we might draw hundreds of megabytes of imagery or other data.  The option also provides greater flexibility with read-only data sources, for example,  it allows us to launch Assign Initial Coordinate System in the Info pane to specify the correct initial projection for linked data.

 

 

Creating an imageserver data source with the Cache data option checked will cache all tiles received from the server while that project is open.   If in addition the Save cached data between sessions box is checked the tiles will be saved into the project file within the Project's System Data hierarchy in a sub-folder called Cache.

 

Web connections are often fast in many parts of the world but even so fetching tiles from a web server is not as fast as fetching them from local storage on local disc.   By caching tiles that we have seen as we pan and zoom within the images served by the web server we can provide faster performance, very much faster in the case of slower Internet connections, if we view a scene more than once.   

 

For example, if we view a scene and then zoom in to part of it, zoom back out and then zoom into another part of it, when we zoom back to views we have already seen if the tiles for that view are cached the system does not need to fetch them from the server over the web but instead it can fetch them much more quickly from cached storage.   If we save the project and then open it later, the cached tiles will still be in the project so any views, such as a zoom to fit followed by zooms into areas of interest, are likely to be viewable using tiles already in cache.

 

Another benefit of caching tiles is that the system does not have to re-fetch tiles when the same scene is viewed in the future.  When using tile servers operated by small organizations that can be a polite way to minimize the load we place on the organization's resources.

 

Improved rendering performance by using cache is wonderful, of course, but it comes at the cost of storing tiles within cache tables that are saved within the project if the Save cached data between sessions box is checked.   As we zoom into views and pan to different views the number of tiles we have seen can grow to where the data stored in cache can be hundreds of megabytes.

 

If we have plenty of free space on disk, terabytes, we might not care about that at all.  But if space is limited or if we want to send a project to someone else without also sending all of the cached tiles we should delete the cache.

 

A compromise is to check the Cache data box but to leave the Save cached data between sessions box unchecked.   In that case tiles will be cached for the duration of our session so if we zoom back and forth and pan between the same views we will get snappy performance without having to fetch and re-fetch the same tiles, but at the same time when we save the project and move on to another project the .map project file will not contain the possibly many megabyte of cached tiles.

Cache Data for Flexibility

When working with read-only data sources, such as a .tif image file that has been set read only by Windows, checking the Cache Data box, whether or not we check the Save cached data between sessions box, will also allow Style controls and (depending on the data source) some other features, such as Selection, to work that otherwise may not be available.   Linking a read only image file without leaving the Cache Data box checked by default may even prevent that image from being displayed.

Using Cache for Offline Viewing of Web Servers

If we check the Save cached data between sessions box when using a web server,  all tiles that are served by the web server we are using will be saved within the .map project in the Cache folder within the System Data hierarchy.   When we save the .map project, all of those tiles will be saved within the .map file that is created.    If we spend a lot of time browsing, the cache can easily grow to hundreds of megabytes, since every time we pan and zoom within the display to a different view we might download another megabyte or so of tiles.  

 

The advantage of a saved cache is that if we re-visit the same view the system does not need to re-fetch those tiles.   If a tile for a given view is already in cache, even if we do not have an Internet connection, Manifold can still show it to us.  Instead of fetching it from the server through Internet, it simply pulls the tile from the saved cache.   We can exploit that effect to "load up" a project with many views from a web server, visiting locations and panning and zooming around in areas of interest, to fill up the cache with tiles from the various views we might want to see later on.  When we save the project it will be loaded with many tiles in cache.

 

If we later want to open and use that project in a location where we have no Internet, we can use the project as before, viewing tiles from cache.   For example, suppose we will be visiting an archaeological site in a location where there is no Internet and we would like to use Google Satellite imagery to guide us along dirt roads along our proposed track.   In our home or office where we have fast Internet, we can create a project that has a Google Satellite imageserver layer, along with layers showing approximate roads, the location of our archaeological site and so on.  

 

We spend some time browsing that project, panning and zooming the map in areas of interest.  For example, if we plan on using a particular access route we might zoom into that route and then pan the view along the route, moving the view from an access road through poorly marked terrain to our site.  Around our site we can zoom to various zoom levels and move the display around, capturing different views in cache.   We then save the project.

 

Out in the field where there is no Internet we can open the project and so long as we pan and zoom the map within the regions that we previously visited, with views and zoom levels like those we used to "load up" the project with cached tiles, the system will show us those tiles again.  It will seem as if we have live Internet, only faster, since everything will appear loaded from cache without any lag time at all.

Deleting Cached Tiles

To delete cached data for web servers that is created when the Save cached data between sessions box is checked we drill down into the System Data hierarchy and then delete the tables storing the tiles.

 

 

Click onto the + icon next to the System Data folder to open it.

 

 

Within the System Data folder we click open the Cache folder.   We see there are two tables there, one table for the street image tiles served by the Bing streets imageserver and one table for the satellite image tile served by the Bing satellite imageserver.

 

 

With the Cache folder highlighted we click the Delete button or simply press the Delete key on the keyboard.

 

 

The Cache folder and the two tables it contains are also deleted.   We can delete tables which cache tiles at any time without worries of de-synchronizing what is displayed.  The system adjusts automatically.  

Caches within External Data Sources

The preceding discussion assumes we have created a web server data source at the top level of the .map project.   In that case, if the Save cached data between sessions option has been turned on, the Cache folder will be created within the System Data folder for the .map project, and all tiles or other cached data will be stored within the .map project in that Cache folder.   

 

However, we might choose to create a web server data source within another data source, for example, within a SQL Server data source, or within a Manifold .map project linked into our current project.  In that case, if the Save cached data between sessions option has been turned on, the Cache folder will be created within the System Data folder within the parent data source, for example, within the SQL Server database, with the tiles or other cached data being saved within a table in the SQL Server database.  That is a strategy we might use if we want the data source (and the cached tiles created) to be used by other users connecting to the same database.    

 

To what extent we can cache data within a parent, external database when the Save cached data between sessions option has been checked depends on the capabilities of that external database, as follows:

 

 

 

 

Creating New Data Sources within Existing Data Sources

New data sources can be created either within the current .map project or within another data sources that is nested within the map project if that nested data source is writable.   If we have moved focus in the Project pane into another data source nested within our project then the Data Source box will be automatically populated with the name of that data source if it is writable.

 

For example, suppose we have a project that includes two data sources, one called France and the other called Germany, and both of those data sources were created from Manifold .map format files.   We can create anything within those external data sources that we could create in the current map project.  

 

We can right-click on the France data source so that has the focus and choose New Data Source.

 

In the dropdown menu we then choose More... to launch the New Data Source dialog.

 

 

The New Data Source dialog will open with an advisory at the top that we are creating the new data source within another data source called France, which is a manifold data sources.      We specify a useful name, choose a Type of file: manifold, browse over to the .map we want to use and then click the Create Data Source button.

 

 

That creates a new data source within the France data source.  If we expand it, we see that it contains all of the many items within the .map project file we used as a data source.  It even contains yet another nested data source, in this case a Virtual Earth hybrid image from a Virtual Earth web server.

Example: Create a New Data Source in a Folder

To see how the Path box works in the New Data Source dialog we can create a new data source within a folder in our project.

 

 

 

 

Suppose we have a new project and within the project we have created a folder called Schengen Zone Countries.    

 

We Right-click that folder and then in the context menus choose Create - New Data Source.

 

In the dropdown menu we then choose More...

 

 

We enter a descriptive name, France, into the Name box and then use the [...] browse button to choose a .map file we have previously created that we wish to use as the data source.   We press Create Data Source.

 

 

 

 

A new data source is created within the Schengen Zone Countries folder.     If we drill down into the newly created France data source and open one of the drawings we can see that we can reach into the France project through the data source.   Performance is identical, with zero loss, to having the drawing in our local project and not in a different project accessed as a data source.

 

 

 

 

After we save the project under some convenient name, whenever we open that project it will have within it the data source that we created which in turn links to the France project.   We will use a similar project in the Example: Create a Data Source within an Existing Data Source topic.

 

Portals to Other Worlds

When we add a data source to a project we create, in effect, a portal to a different world.   When we open the data source to show the hierarchy within, the Project pane is giving us a view into the different world of that data source, where the rules that apply are the rules of that world.    For example, if we create a read-only data source, then we can open the hierarchy within that data source and view what is in it, but we cannot add components within that world and we cannot change components within that world.

 

 

Consider the above illustration, a section of a screenshot shown in the Example: Create a Drawing from Read-Only Geocoded CSV File topic.   The portion of the hierarchy shown within the read-only data source is read-only.  We cannot create any new components in that portion of the Project pane because that portion resides in the different world of a read-only data source that links in a read-only .csv file.  We can open the Dots table and read data from it and the table can participate in queries by providing data, but we cannot change anything in that table.  For example, we cannot change the schema of the table by adding fields or indexes to it and we cannot edit values within records in that table.  

 

The region bordered in green and marked Read / Write is part of the world of the local project.   We can do whatever we like in that region, such as right-clicking there and using Create - New Drawing to create a new drawing in the project.

 

 

Consider the above illustration, a section of a screenshot shown in the Example: Create a Data Source within an Existing Data Source topic.  It shows worlds within worlds.  The hierarchy within the Europe data source is a world within the .map file called Europe.map that was linked in as a data source into our Project.   The hierarchy our Project pane shows is a window into the world of that other .map file.    It shows that within the Europe map file's world there are yet three more worlds, a linked in data source called England and two more linked in data sources called France and Germany, all of them linked in from other .map files.  

 

One of the worlds, that is, data sources, has been expanded so we can see a window into that world as well, the world of the Germany.map file.    In the above illustration all of the linked in worlds are themselves Manifold Release 9 .map files that have been linked in without any read-only restrictions, so we can work with them with full Manifold capabilities.    The rules which apply within those worlds are the same as the rules which apply in the local project, since Manifold is seamless in that way.  

 

However, if the world of, say, the Germany data source was that of a database such as SQL Server or PostgreSQL, then whatever rules which such databases apply within their own projects, for example, what data types are available,  would apply within that hierarchy.

 

Notes

'Database' vs 'data source' - In Manifold documentation and build notes, these two terms are frequently more or less the same thing. Data source is a collective name for files,  databases, or web servers which provide data.   When we start Manifold and create a new .map file, we set that .map file to be an unnamed root data source, 'root' because it is a root of a potential data source tree. When we link a SHP file, we do so by creating a new data source component, that is, a new data source.   That data source behaves like a database in that it exposes data through tables, and it has system tables with reserved names which describe its contents. The object representing an opened data source for a script is usually called Database, and in other ways data sources are treated as databases within Manifold.

 

Passwords - Some data sources to which we might want to connect may use passwords.   Normally such data sources are databases such as Oracle or SQL Server and the dialogs to connect to them using the Database: option as the Type will allow specifying a password.    In some cases file databases such as Microsoft Access MDB might also require a password.   To specify a password we should connect using the Database: oledb option, which utilizes a dialog that allows specifying a password, and not  using the File Database: mdb option which is a simpler option that does not allow specifying a password.

 

Tech Tip:  Some web servers to which we might connect using WMS, WMTS, WFS or other connection standards can be incredibly unresponsive or, despite allowing a connection will not allow data to be fetched.   If pressing the Test button results in a successful connection, or if layers available are displayed when we expand the data source, but then when we try to display an advertised layer nothing happens, we can open the Log Window to see what is going on.

 

Here are a few lines of log window information for a happy session, connecting to a WMS server in New Zealand run by the ever-efficient LINZ:

 

Web request: (root)::[Data Source] (GetCapabilities) (0.578 sec, 13.0 KB)

Web request: (root)::[Data Source]::[layer-767] (GetMap) (1.255 sec, 47.9 KB)

Render: [Data Source]::[layer-767 Image] (1.561 sec @1.0)

 

In contrast, a few lines of log window information for an unhappy session, attempting to connect to the WMTS server run by the French cartographic service IGN but getting no image for a cadastral parcels layer:

 

Web request: (root)::[IGN]::[CADASTRALPARCELS.PARCELS_bdparcellaire_b] (GetTile) (0.070 sec, 0 B)

*** (root)::[IGN]::[CADASTRALPARCELS.PARCELS_bdparcellaire_b] (GetTile) The remote server returned an error: (403) Forbidden.

Render: [IGN]::[CADASTRALPARCELS.PARCELS_bdparcellaire_b Image] (0.315 sec @1.0)

 

From the above we see an empty cadastral parcels image was rendered because the remote server refused to provide tiles in response to the GetTile request, returning a 403 error Forbidden.    Finding out why a web server refuses service can involve some detective work.  

 

The server might require a special key for access, a key we are trying to use may be out of date, a login and password may be required for some resources, the connection string may have been copied and pasted inaccurately or, of course, in the case of IGN the server may be refusing to work to show solidarity with striking  railroad workers or air traffic controllers.

 

Keep in mind that just because a web server offers tiles via various standards like WMTS or whatever does not mean that it will be fast or even reasonably tolerable.  Some web servers are painfully, incredibly slow and will take many seconds or even minutes per tile.   When images do not appear or fill in with painful slowness a look at the Log Window can show the problem is not our Internet connection or Manifold, but just incredibly slow response from the web server.

 

Failed Connections - Connecting to some data sources, such as slow servers hosted on machines that are far away via Internet, can take some time, possibly many seconds, to try to connect.  Such connections can involve many layers in the Windows and networking protocols that are used.  If for any reason a connection fails (server not running, etc.) a failure to connect can take even longer as the various Windows and networking layers decide they cannot reach the desired data source.   If a failed connection takes 5 or 10 seconds to surface from the failed attempt, it does not make sense to keep trying to connect every 1/10th of a second apart.   To prevent such failed attempts from cumulating into absurdly long waits, Manifold will record failed attempts and will then automatically throttle subsequent connection attempts by delaying subsequent connection attempts.  The delay starts at 1 second and gradually increases to 15 seconds.   When failed attempts generate error messages in the Log window, repeated failed attempts will have the same error message omitted to avoid loading up the log with repeated text.

 

Three Letter Extensions - Most file names in Windows end in what is called a three letter extension, which is usually three letters at the end of the file name following a dot . character.    The three letter extension is one way Windows at times (but not always) keeps track of what a file is supposed to be.    Unfortunately, by default Windows hides the three letter extensions of files and instead tries to associate files with icons of whatever program is normally associated with that file.  This is confusing when working with the many file formats that Manifold and similar products utilize.  

 

Therefore, please turn off the hiding of extensions by Windows.  A typical way to do so in most versions of Windows would be from Windows Explorer, choose Tools - Folder options, press the View tab and then in the Advanced Settings pane ensure that the Hide extensions for known file types is unchecked.  Press the Apply to Folders and then press OK.  You will then be able to see extensions such as .map and others.  See also the Essay on three letter extensions and why the default hiding of them by Windows is such a bad thing.

 

See Also

That YX Thing - An essay discussing how it is that so much spatial data claiming to use EPSG standards fails to honor the standard, and thus why Manifold must provide a check box to deal with such errors.

 

Web Servers

 

Custom Servers

 

Favorite Data Sources - View and edit the Favorites dialog for data sources, showing various data sources that may be added to the project with a single click of favorites.

 

Example: Spectacular Images and Data from Web Servers - A must see topic providing a gallery of views illustrating how Manifold can use web servers such as imageservers and other free resources to provide a seemingly endless selection of spectacular background maps, satellite images and GIS data with nearly zero effort.

 

Example: An Imageserver Tutorial - An extensive tutorial showing step by step how to add new data sources that are imageservers, how to show them as layers in a map, how to create a new drawing that matches the projection of the map and how to trace over what is seen in an imageserver layer to create an area object in the drawing.

 

Example: Create a New Data Source from a Manifold Image Server - Manifold image server modules are snippets of code which use the Manifold Image Server Interface (ISI) to automatically fetch image tiles from popular image servers like Virtual Earth, Wikimapia, Yahoo!, Google Maps, Yandex and many others. Image servers can provide street maps, overhead satellite imagery, combinations of streets and satellite imagery and other data as well.  Using Manifold Image Servers is one of the most popular Manifold features.

 

Example: Create a New Data Source from a MAP File - Create a new data source from an existing Manifold .map project file.   This is the classic way to nest projects, one calling another, to create libraries of data and projects.   Access to nested projects has effectively zero performance loss and the links within projects take up effectively zero space so we can create huge constellations of data at our fingertips.

 

Example: Create a Data Source within an Existing Data Source - When a data source is writable, for example, if the data source is a Manifold .map file, we can work within that data source as if it were at the top level of our project.   For example, we can create a new data source that is nested within the existing data source.   This example shows how.

 

Example: Create and Use New Data Source using an MDB Database - This example Illustrates the step-by-step creation of a new data source using an .mdb file database, followed by use of SQL.  Although now deprecated in favor of the more current Access Database Engine formats, .mdb files are ubiquitous in the Microsoft world, one of the more popular file formats in which file databases are encountered.  

 

Example: Link GPKG and Save Style - A companion topic to the GPKG topic.   How to link a GPKG, open a drawing, Style it and then save so the styling is retained within the GPKG file.

 

Example: Trace an Area in a Map over an Image Background - In a map with a drawing layer above an image layer (served dynamically by an imageserver), create an area object in the drawing by tracing over the outlines of something seen in the image layer below

 

Example: Style Applied to an Image Server Image - Because the Style dialog simply changes the way an image is displayed and not the data, it can operate on read-only data served by various web servers such as WMS REST servers.    In this example we look at every detail of creating a data source using a WMS REST  server and then manipulating the appearance of the display with Style.  We will connect to a WMS server that provides LiDAR data in various forms, including as terrain elevation.