File - Create - New Data Source

The New Data Source dialog allows us to create a data source within a project.    The data source can be created within the existing .map project or within another, nested data source that already exists within the project.  tech_angus_sm.png

 

Important: New Data Sources are linked, not Imported, into the project.

 

A new data source links data into our project without importing the data.    The data in that new data source remains stored within whatever is the originating source.  See comments on linking data into a project in the Getting Started topic.  

 

For an example of creating a new data source see the Getting Started topic as well as the many examples recommended at the end of this topic.

 

A new data source can be created from data that is resident in hundreds of different type of data sources.  The New Data Source dialog organizes data sources as four different Types, depending on what we specify to access the data source, to make them easier to find in a long list of available types:

 

 

 

 

 

Connections to the above are provided within Manifold by dataports, which are modules of code within Manifold that provide connection services to different data sources.   Dataports are always at least read-only and will often also provide read/write capability.

 

There may be more than one way to connect to a data source.   For example, we can connect to Microsoft Access MDB file databases directly from a 32-bit Manifold session or we can connect using the generic OLE DB connection.  Normally we would connect directly using the MDB File Database option but if the MDB file has a password for access we can connect using the OLE DB option, which allows specifying a password.

 

Note that we can create a new data source that links a script into the project.  This is a handy way of including within projects scripts that might reside outside of a project, be shared by other programs, edited or updated by other users and so on.   

 

We can also create a new data source that links text files into a project to create comments components.  These are useful when documenting information about other data, sharing "to do" lists and other team-oriented comments and other text documents that might be edited by other users using other programs or created automatically by other processes.

 

New Data Source Dialog Controls

il_newdatasource_oracle.png

 

The New Data Source dialog will automatically provide controls as needed when different types of data sources are selected:

 

Name - By default, the name Data Source is used to name whatever new data source we create.   We can change this to a more memorable name as we like.  If we forget the origin of a data source we can hover the mouse over the data source and a tool tip will provide connection information.

 

Type - Choose from the list of different types of data sources that may be connected.  .map project files are of type File: manifold.     The box below the Type box provides subcategories for types such as Web Server: imageserver that have more than one possible subtype.

 

Source - A connection string or path to a file or other resource to open.   The source string can be manually entered or it can be created using dialogs launched by the [...] browse button or it may be automatically populated for some types of data sources.

 

[...] Browse button - Click to populate the Source box.  Often used to launch the File - Open dialog to open a file format or file database format.

 

Open as read-only - Open the data source read-only.  Not checked by default so that if the dataport for the data source supports a read/write connection by default the data source will be both readable and writable.   Some types of data sources, such as enterprise class DBMS servers like Oracle, will allow multiple users to connect read/write to a data source.  Other types of data sources, such as most file formats, will lock out other users if we connect to them read/write.  Therefore, if we would like to allow other users to be able to simultaneously connect to this data source and we do not need to write to it, we can check this box.  We can also check the read-only box if we would like to protect against accidental changes to the data source.

 

Fault-tolerant save -  Appears when File: manifold is selected in the Type box.   When checked (the default) the system will create a .SAVEDATA accessory file and will use fault tolerant saves if a .map file is connected as a new data source.   See comments at the end of the Getting Started topic and in the Accessory Files Created topic.

 

Use vector data as either XY or YX according to coordinate system - This option appears for those data sources where coordinate systems can be in either ordering.   Standards such as EPSG for specifying coordinate systems have codes which sometimes specify that data occurs in XY form, that is longitude, latitude coordinate ordering, and sometimes as YX form, that is, latitude, longitude coordinate order.   

In theory, this box should always be checked.  It is not checked by default because, unfortunately, so many popular packages disregard EPSG standards that most spatial data encoded with EPSG codes which require flipped coordinate ordering is not flipped.  See the That YX Thing essay topic.

 

Cache data - Create and utilize an accessory .MAPCACHE file for improved performance. Checked by default.  This option appears with most file data sources and has two options, both checked by default, one to cache only data that is expensive to dynamically compute and the other to save cached data between sessions.   It only makes sense to uncheck either option if we are desperately low on free disk space.

 

Save cached data between sessions - Appears as a sub-choice for some data sources for which the Cache data option appears.   Save the cached data for the next time this project is opened, either within the .MAPCACHE file or within the .map project itself within a Cache subfolder within the System Data hierarchy.

 

Cache only data that is expensive to compute dynamically - Appears as a sub-choice for some data sources for which the Cache data option appears.   If there is significant overhead in computing data to be displayed, cache the results.  For example, some file formats may require complex calculations to extract data from the format.  It is quicker to do that once and save the result for re-use than to repeatedly perform the same computation/extraction.

 

Cache data also appears as a choice when creating a New Data Source for web servers, in which checking the box and also checking the Save cached data between sessions box creates a Cache folder within the System Data folder hierarchy in which cached tiles are saved.

 

Creating New Data Sources within Existing Data Sources

New data sources can be created either within the current .map project or within another data sources that is nested within the map project if that nested data source is writable.   If we have moved focus in the Project pane into another data source nested within our project then the Data Source box will be automatically populated with the name of that data source if it is writable.

 

For example, suppose we have a project that includes two data sources, one called France and the other called Germany, and both of those data sources were created from Manifold .map format files.   We can create anything within those external data sources that we could create in the current map project.  

 

il_nested_newdatasource01_01.png

We can right-click on the France data source so that has the focus and choose New Data Source.

 

il_nested_newdatasource01_02.png

The New Data Source dialog will open with an advisory at the top that we are creating the new data source within another data source called France, which is a manifold data sources.      We specify a useful name, choose a Type of file: manifold, browse over to the .map we want to use and then click the Create Data Source button.

 

il_nested_newdatasource01_03.png

 

That creates a new data source within the France data source.  If we expand it, we see that it contains all of the many items within the .map project file we used as a data source.  It even contains yet another nested data source, in this case a Virtual Earth hybrid image from a Virtual Earth web server.

 

Accessory Dialogs for Browsing Sources

Depending on the Type of new data source we choose, the system may launch accessory dialogs when we press the [...] Browse button.

 

Files - Browse the Windows file system to locate and open the desired file using a Select File  dialog modeled on the standard Windows File - Open dialog.  The type of file will be pre-loaded into the dialog depending on what was selected in the Type box

 

Web Servers - Launches the Web Login dialog that allows specifying a server and connection characteristics.

 

dlg_web_login_create_nds.png

 

The above dialog provides the following controls:

 

Server - The connection string for the server.  This may be a simple URL or a very lengthy URL/connection string that embeds parameters such as keys that grant access or other parameters.   For example, a typical string to connect to a WMS server that provides topographic maps from New Zealands's LINZ / National Topographic office might be:

 

https://data.linz.govt.nz/services;key=ca62cf6a884a4f0cbfe5/wms/layer-767?service=WMS&version=1.1.1&request=GetCapabilities

 

We enter that into the Server box:

 

dlg_web_login_server_create_nds.png

 

Note that the above connection string has been edited so the key shown above is not accurate:  anyone can get their own key by registering on the LINZ website.

 

Use login and password - Check this box for servers that require logging in with a login name and a password, providing the required name and password in the Login and Password boxes.

 

Use proxy server - Check this box when connecting through a proxy server.   The Proxy, Login, and Password boxes allow specifying the connection string to the proxy server as well as the login name and password required to use the proxy server.

 

API key - Provide a key that authorizes use of an API when connecting to a proprietary data source that requires such a key.

 

Application or Application Key - A secondary key or authentication code for those servers, such as here.com (also known as wego.here.com) that may require it.

 

Timeout  - Specify a time in milliseconds to wait for connecting to the specified server.  Use 0 for the default timeout or specify whatever is the desired time to wait before giving up on the server.

 

Test - Press the Test button to try the connection using the specify parameters.   If successful, a Connection established information dialog will pop open.

 

 

Web Servers using Manifold Image Servers - Choosing Web Server: imageserver in the Type box will enable a secondary box below the Type box that will be loaded with a list of Manifold image servers installed for use.  Choosing an image server from the list will automatically load the connection string into the Source box.   Clicking the [...] Browse button will launch the same Web Login dialog shown above, to allow use of a login and password plus use of a proxy server if desired.   The Web Login dialog is also handy for providing a Test button that can be used to test the connection.

 

Database Servers - The [...] button next to the Source box launches the Database Login dialog that allows specifying a server and connection characteristics.

 

dlg_database_login_create_nds.png

 

The above dialog provides the following controls:

 

Server - The connection string for the database server.  

 

Use integrated security - Use Windows integrated security credentials.

 

Use login and password -  Use security credentials specified in the Login and Password boxes, which are enabled when this option is selected.

 

Database - Specific database to use in this server.  Enabled for database servers such as PostgreSQL and SQL Server that allow specifying a database within the database server.

 

Test - Press to try a connection to the server using the specified settings.

 

Database Servers using Generic Connection Technologies - When choosing a database Type that consists of a generic connection technology such as ODBC or OLEDB the [...] Browse button will launch the Windows dialog for that connection technology, for example, launching the Windows Data Link Properties dialog for OLEDB or the Windows Select Data Source dialog for ODBC.

 

A generic connection technology such as ADONET will enable a secondary box below the Type box when Database: adonet is selected as the Type.   That secondary box will be loaded with a list of connection options made available by any DLLs within the PATH supporting ADONET connections.  If no such DLLS exist the list will be empty.

 

Cache Data Option

When creating new data sources for a web server data source we can check or uncheck the Cache data box (checked by default)  in the New Data Source dialog.   This option merits attention when creating data sources from which we might draw hundreds of megabytes of imagery or other data.

 

eg_imagesrv_tour01_48.png

 

Creating an imageserver data source with the Cache data option checked will cache all tiles received from the server.   If in addition the Save cached data between sessions box is checked the tiles will be saved within the Project's System Data hierarchy in a sub-folder called Cache.

 

Web connections are often fast in many parts of the world but even so fetching tiles from a web server is not as fast as fetching them from local storage on local disc.   By caching tiles that we have seen as we pan and zoom within the images served by the web server we can provide faster performance, very much faster in the case of slower Internet connections, if we view a scene more than once.   

 

For example, if we view a scene and then zoom in to part of it, zoom back out and then zoom into another part of it, when we zoom back to views we have already seen if the tiles for that view are cached the system does not need to fetch them from the server over the web but instead it can fetch them much more quickly from cached storage.   If we save the project and then open it later, the cached tiles will still be in the project so any views, such as a zoom to fit followed by zooms into areas of interest, are likely to be viewable using tiles already in cache.

 

Another benefit of caching tiles is that the system does not have to re-fetch tiles when the same scene is viewed in the future.  When using tile servers operated by small organizations that can be a polite way to minimize the load we place on the organization's resources.

 

Improved rendering performance by using cache is wonderful, of course, but it comes at the cost of storing tiles within cache tables that are saved within the project if the Save cached data between sessions box is checked.   As we zoom into views and pan to different views the number of tiles we have seen can grow to where the data stored in cache can be hundreds of megabytes.

 

If we have plenty of free space on disk, terabytes, we might not care about that at all.  But if space is limited or if we want to send a project to someone else without also sending all of the cached tiles we should delete the cache.

 

A compromise is to check the Cache data box but to uncheck the Save cached data between sessions box.   In that case tiles will be cached for the duration of our session so if we zoom back and forth and pan between the same views we will get snappy performance without having to fetch and re-fetch the same tiles, but at the same time when we save the project and move on to another project the .map project file will not contain the possibly many megabyte of cached tiles.

 

Deleting cached tiles - To delete cached data for web servers that is created when the Save cached data between sessions box is checked we drill down into the System Data hierarchy and then delete the tables storing the tiles.

 

eg_imagesrv_tour01_44.png

 

Click onto the + icon next to the System Data folder to open it.

 

eg_imagesrv_tour01_45.png

 

Within the System Data folder we click open the Cache folder.   We see there are two tables there, one table for the street image tiles served by the Bing streets imageserver and one table for the satellite image tile served by the Bing satellite imageserver.

 

eg_imagesrv_tour01_46.png

 

With the Cache folder highlighted we click the Delete button or simply press the Delete key on the keyboard.

 

eg_imagesrv_tour01_47.png

 

The Cache folder and the two tables it contains are also deleted.   We can delete tables which cache tiles at any time without worries of de-synchronizing what is displayed.  The system adjusts automatically.  

 

Example: Create a New Data Source in a Folder

To see how the Path box works in the New Data Source dialog we can create a new data source within a folder in our project.

 

eg_create_nds02_01.png

 

Suppose we have a project we call Europe and within the project we have created a folder called Schengen Zone Countries.    We highlight that folder and then choose File - Create - New Data Source

 

eg_create_nds02_02.png

 

We enter a descriptive name, France, into the Name box and then use the [...] browse button to choose a .map file we have previously created that we wish to use as the data source.   We press Create Data Source.

 

eg_create_nds02_03.png

 

A new data source is created within the Schengen Zone Countries folder.     If we drill down into the newly created France data source and open one of the drawings we can see that we can reach into the France project through the data source.   Performance is identical, with zero loss, to having the drawing in our local project and not in a different project accessed as a data source.

 

 

eg_create_nds02_03a.png

 

 

We will save the Europe project.  Whenever we open the Europe project it will have within it the data source that we created which in turn links to the France project.   We will use this Europe project in the Example: Create a Data Source within an Existing Data Source topic.

 

Notes

Passwords - Some data sources to which we might want to connect may use passwords.   Normally such data sources are databases such as Oracle or SQL Server and the dialogs to connect to them using the Database: option as the Type will allow specifying a password.    In some cases file databases such as Microsoft Access MDB might also require a password.   To specify a password we should connect using the Database: oledb option, which utilizes a dialog that allows specifying a password, and not  using the File Database: mdb option which is a simpler option that does not allow specifying a password.

 

tech_ravi_sm.png

Tech Tip:  Some web servers to which we might connect using WMS, WMTS, WFS or other connection standards can be incredibly unresponsive or, despite allowing a connection will not allow data to be fetched.   If pressing the Test button results in a successful connection, or if layers available are displayed when we expand the data source, but then when we try to display an advertised layer nothing happens, we can open the Log Window to see what is going on.

 

Here are a few lines of log window information for a happy session, connecting to a WMS server in New Zealand run by the ever-efficient LINZ:

 

Web request: (root)::[Data Source] (GetCapabilities) (0.578 sec, 13.0 KB)

Web request: (root)::[Data Source]::[layer-767] (GetMap) (1.255 sec, 47.9 KB)

Render: [Data Source]::[layer-767 Image] (1.561 sec @1.0)

 

In contrast, a few lines of log window information for an unhappy session, attempting to connect to the WMTS server run by the French cartographic service IGN but getting no image for a cadastral parcels layer:

 

Web request: (root)::[IGN]::[CADASTRALPARCELS.PARCELS_bdparcellaire_b] (GetTile) (0.070 sec, 0 B)

*** (root)::[IGN]::[CADASTRALPARCELS.PARCELS_bdparcellaire_b] (GetTile) The remote server returned an error: (403) Forbidden.

Render: [IGN]::[CADASTRALPARCELS.PARCELS_bdparcellaire_b Image] (0.315 sec @1.0)

 

From the above we see an empty cadastral parcels image was rendered because the remote server refused to provide tiles in response to the GetTile request, returning a 403 error Forbidden.    Finding out why a web server refuses service can involve some detective work.  

 

The server might require a special key for access, a key we are trying to use may be out of date, a login and password may be required for some resources, the connection string may have been copied and pasted inaccurately or, of course, in the case of IGN the server may be refusing to work to show solidarity with striking  railroad workers or air traffic controllers.

 

Keep in mind that just because a web server offers tiles via various standards like WMTS or whatever does not mean that it will be fast or even reasonably tolerable.  Some web servers are painfully, incredibly slow and will take many seconds or even minutes per tile.   When images do not appear or fill in with painful slowness a look at the Log Window can show the problem is not our Internet connection or Manifold, but just incredibly slow response from the web server.

 

Three Letter Extensions - Most file names in Windows end in what is called a three letter extension, which is usually three letters at the end of the file name following a dot . character.    The three letter extension is one way Windows at times (but not always) keeps track of what a file is supposed to be.    Unfortunately, by default Windows hides the three letter extensions of files and instead tries to associate files with icons of whatever program is normally associated with that file.  This is confusing when working with the many file formats that Manifold and similar products utilize.  

 

Therefore, please turn off the hiding of extensions by Windows.  A typical way to do so in most versions of Windows would be from Windows Explorer, choose Tools - Folder options, press the View tab and then in the Advanced Settings pane ensure that the Hide extensions for known file types is unchecked.  Press the Apply to Folders and then press OK.  You will then be able to see extensions such as .map and others.  See also the Essay on three letter extensions and why the default hiding of them by Windows is such a bad thing.

 

See Also

That YX Thing - An essay discussing how it is that so much spatial data claiming to use EPSG standards fails to honor the standard, and thus why Manifold must provide a check box to deal with such errors.

 

Web Servers

 

Example: Spectacular Images and Data from Web Servers - A must see topic providing a gallery of views illustrating how Manifold can use web servers such as imageservers and other free resources to provide a seemingly endless selection of spectacular background maps, satellite images and GIS data with nearly zero effort.

 

Example: An Imageserver Tutorial - An extensive tutorial showing step by step how to add new data sources that are imageservers, how to show them as layers in a map, how to create a new drawing that matches the projection of the map and how to trace over what is seen in an imageserver layer to create an area object in the drawing.

 

Example: Create a New Data Source from a Manifold Image Server - Manifold image server modules are snippets of code which use the Manifold Image Server Interface (ISI) to automatically fetch image tiles from popular image servers like Virtual Earth, Wikimapia, Yahoo!, Google Maps, Yandex and many others. Image servers can provide street maps, overhead satellite imagery, combinations of streets and satellite imagery and other data as well.  Using Manifold Image Servers is one of the most popular Manifold features.

 

Example: Create a New Data Source from a MAP File - Create a new data source from an existing Manifold .map project file.   This is the classic way to nest projects, one calling another, to create libraries of data and projects.   Access to nested projects has effectively zero performance loss and the links within projects take up effectively zero space so we can create huge constellations of data at our fingertips.

 

Example: Create a Data Source within an Existing Data Source - When a data source is writable, for example, if the data source is a Manifold .map file, we can work within that data source as if it were at the top level of our project.   For example, we can create a new data source that is nested within the existing data source.   This example shows how.

 

Example: Create and Use New Data Source using an MDB Database - This example Illustrates the step-by-step creation of a new data source using an .mdb file database, followed by use of SQL.  Although now deprecated in favor of the more current Access Database Engine formats, .mdb files are ubiquitous in the Microsoft world, one of the more popular file formats in which file databases are encountered.  

 

Example: Modify GPKG Geometry with SQL then Add Drawing - This topic provides a "Hello, World" example that shows a simple, but typical, task involving spatial data.  We will take a country-sized data set in GeoPackage (GPKG) format and change all areas in the data to the boundary lines for those areas and then save those boundary lines as a new table.  We add a spatial index to the table and create a new drawing to visualize the new table.

 

Example: Trace an Area in a Map over an Image Background - In a map with a drawing layer above an image layer (served dynamically by an imageserver), create an area object in the drawing by tracing over the outlines of something seen in the image layer below

 

Example: Style Applied to an Image Server Image - Because the Edit - Style dialog simply changes the way an image is displayed and not the data, it can operate on read-only data served by various web servers such as WMS REST servers.    In this example we look at every detail of creating a data source using a WMS REST  server and then manipulating the appearance of the display with Style.  We will connect to a WMS server that provides LiDAR data in various forms, including as terrain elevation.