Manifold project files, called projects, in .map format are the native "documents" for Manifold. The File - Open command will open an existing Manifold .map project file and the File - Save command will save a Manifold project in .map project file format. Manifold .map format is also referred to as MAP format, using all-capital letters.
Manifold stores everything it does in a project within a Manifold .map file. A .map project file is an incredibly fast, very rich, file database that can store data of all kinds. It can store vast amounts of data for large, complex entities like maps, images, drawings, tables, scripts, queries, and much more, and it can store links to other databases, like Oracle or PostgreSQL DBMS servers, and it can store connections to web data sources providing petabytes of data. A .map project file can have thousands of entities in it and itself be terabytes in size.
Manifold .map project format is an operational format: it is designed for fast, everyday work. Manifold .map format is built for maximum speed, and for nesting of projects with zero performance loss. .map project format is incredibly fast, convenient, and very robust, a perfect format for daily work.
Manifold also has another format, .mxb format, that is designed to store projects as compactly as possible for archival storage and for exchanging data. .mxb format is also referred to as MXB format, using all-capital letters. MXB is slower to write and slower to open than MAP and is not used for interactive work, but only for archival storage. When Manifold opens a .mxb file it will immediately decompress the .mxb file to create a .map file that it will use. MXB is a highly compressed and robust format that minimizes file size, perfect for archival storage and for minimizing the size of files for transmission. MXB is also useful in ensuring maximum data reliability. See the For Maximum Reliability topic.
Manifold .map files at the present time do not allow simultaneous read/write access by multiple Manifold sessions. If a .map file has been opened in one Manifold session it cannot be opened in another. We can, however, link a .map file into several different Manifold sessions if we create the data source for each with the read only check box checked.
For simultaneous, multiuser read-only sharing of spatial data, use Manifold Server. Manifold Server is a high performance, parallel, spatial database server that is built into Manifold Release 9 Universal edition. It allows many Manifold users to share data in a .map file, or in a constellation of nested .map files, that is published through Server.
If we require simultaneous read/write access to data by multiple Manifold sessions, such as by different people on different machines, we can store the data in a DBMS, such as PostgreSQL/PostGIS, Oracle, MySQL, SQL Server, DB2, or other DBMS.
A project is the collection of stuff we are working on in any given Manifold session. It is all the different things we have in our Manifold workspace in that session. The Project pane in the Manifold desktop will show the contents of the project. See the Project Pane topic.
File - Save will save the contents of a project into a Manifold .map project file. Stuff that has been imported into the project will be saved within the .map file. Stuff that has been linked into the project will remain outside Manifold in whatever file or server from which it was linked.
File - Open will open a previously-saved project file to get the project back. All data that was imported into the project will still be there. All linked data sources also will be there, including connection information, like paths and URLs, to the external files and servers they use. When a project has links to external files, if you move that project file to a different computer system do not forget also to copy the linked files to that computer system, to the same relative paths from the project file.
To fit into this documentation, illustrations show an artificially small Manifold desktop, with only a few panes, docked to the left or right side. In real life we use a much larger Manifold desktop, and all panes would be turned on, with some panes docked to the left and others docked to the right. Right-click a pane's tab to change where it is docked. Manifold will remember that new arrangement for our next session.
Manifold always launches with a project, a new blank project called New Project if not opened with an existing project file. An asterisk * appears next to the project name if the contents have been changed since the last save. The desktop opens with the same arrangement of panes that were used the last time Manifold was closed, with panes docked to the left or to the right side of the desktop. Turn on panes using the View - Panes command in the main menu.
A new project is empty but for a System Data folder containing tables Manifold uses to manage the project.
Clicking the + icon opens the System Data folder to show the special tables it contains, normally the mfd_meta and mfd_root tables.
Other tables, such as a mfd_meta_source table for some data sources, or mfd_collate and mfd_srid tables for DBMS data sources, might appear in the System Data folders of a data source when linking to various data sources.
Tables in System Data folders store operational information for the data source. We do not need to work with the System Data tables during routine Manifold use. However, more advanced users may want to use them to manipulate components. Since they are small, simple tables and they always are at hand we can use them to illustrate examples in this documentation.
In the illustration above we have used File
- Open to open a previously-saved project called Aus
Hydro.map. It all comes back, including a data source that is a Microsoft
Bing image server
that is ready to go. 
Tech tip: Within the Project pane most Windows moves we know, like Copy and Paste, will work. For example, we can Copy a table and Paste it within a folder. We can drag and drop a table into a folder to move it there. One modification to a standard Windows key: drag and drop in Windows is a move command and Ctrl-drag and drop is a copy command. In Manifold, drag and drop also is a move command but Shift-drag and drop is a copy command.
We can organize what is in our project using folders, like the System Data folder seen above, a good idea if our projects include many components. The Project pane itself includes useful features, like filtering, to help us zero in on the components we want in a project that may contain very many.
When we add stuff to our project that stuff will appear in the Project pane as various items, called components. Components are items such as tables, images, scripts, data sources and so on depending on what sort of data is added. The project pane will show all components in our project with an icon next to each to indicate the type of component.
Tables are the center of everything in Manifold. All data in Manifold is stored in tables in one form or another. That's good because it means we can get at anything with SQL. Nothing is hidden.
Other components such as images or drawings are visualizations of data stored within some table. Map components show other visual components, such as images, drawings and labels, as layers within a map window.
|
Tables - Where all data within Manifold is stored in one form or another. Opening a table window shows a table in classic row and column presentation. |
|
Drawings - A visual display of vector data stored within tables using a variety of geometry types. The same data can be seen in multiple different drawings using different styles, and the same drawing can be open in multiple windows with different selections and views in each. |
|
Data Sources - Data linked into a project from an external source, which could be a file, a file database, a database serve or a web server, covering virtually every database and file type encountered in database or GIS work. |
|
Images - A visual display of raster data stored within tables as tiles using a variety of different data types and channel combinations. All rasters are called images in Manifold even those which are terrain elevation or other non-photographic data. |
|
Labels - Text annotations automatically created from the fields of a drawing's table. |
|
Layouts - Layouts are compositions on virtual sheets of paper, allowing us to create documents showing maps or other components as PDF documents or hard copy printed sheets. |
|
Locations - Locations save a particular geographic location with an associated scale, thus saving a given view in a map or other window to which we can return. |
|
Maps - Consist of a stack of layers where each layer is a drawing, image or labels component. |
|
Queries - Written in SQL, queries manipulate data and projects, for example, creating new tables and other components, performing calculations, altering the structure of databases and tables, and extracting, editing and analyzing subsets of data. |
|
Scripts - Manifold includes built-in support for scripting in eleven different languages with six to eight always available and the remainder easy to install. Scripts provide custom capabilities and can automate virtually anything. |
|
Comments - Text saved and displayed in a comments window provides a simple but convenient way to save notes about a project or to save text as a scratchpad. |
|
Folders - We can organize our project by using folders. |
Components we see in the Project pane may be a link to data stored somewhere else or they may be a local component for which the data is stored within the project file. Local components are stored in the project's .map file. The project's .map file will grow in size as we add local components to a project. Linked components do not take up any space in the .map file that stores the project. Linked components are just pointers to the file or server somewhere else where the data is stored. .map files can get very big if we add large components to them.
A yellow database
cylinder marks a linked file or data source in the Project pane. The
yellow database cylinder tells us that everything within that hierarchy
is coming into the project from some source outside of the project. The
source will also have its own System
Data folder with information on the linked source. In
the above example the Products
table seen under the products
data source hierarchy is coming in from some data source outside
of the project. In contrast, the bronze
image and the bronze Tiles table
are both local components that are stored within
the project file.
A yellow database
cylinder with a lock marks a file
or data source that has been linked read-only.
The lock indicates that the contents may be viewed, but they
cannot be changed. Files or servers that have been linked read-only can be used simultaneously
by multiple different Manifold sessions. For example, if a
file is stored on a different machine and we link it into our project
read-only, a user
on some other machine could also link that same file into their projects
so long as they linked that file read-only.
Linking a data source into a project also creates a section of the project that is a portal to another world, where what we see in that data source is not in the project file but is a view into a different file or data source (like a database) which may or may not be writable. See the discussion in the Portals to Other Worlds section of the Project Pane topic.
When writing queries, we can refer to tables within a data source using the syntax [datasource]::[table]. For example, we could write a query that selects all records from the Products table within the products data source in the illustration above as:
SELECT * FROM [products]::[Products];
See the Example: Refer to a Table within a Data Source topic for more information on referring to tables within data sources.
When we use File - Save to save a project we write that project into a file database stored in Manifold .map file format. The .map file we create saves all the data for local components as well as saving links, Project pane folder hierarchies and other housekeeping information for the project. The next time we open the project file in Manifold we will get all of that back, with the Project Pane showing components and folders organized exactly like they were when we saved the project.
Manifold project files can save large amounts of data, over a terabyte, and provide performance second to none in database applications. That makes Manifold itself a fine choice for a file database system, using Manifold .map format project files for storage. Because a Manifold .map database can store very large tables, images, drawings, or other data, we can use Manifold for storage without being forced to use an external DBMS just because we want to work with larger data.
At the same time Manifold is exceptionally adept at working with and very fast at connecting to DBMS servers like Oracle, PostgreSQL, DB2 or SQL Server. Manifold itself is so fast that Manifold will take full advantage of the best performance those systems can provide. That makes Manifold a great console for working with multiple different DBMS databases at once, with big data stored locally in the project file, or with big data stored in external DBMS servers or both at the same time.
Manifold can also link other, previously created Manifold project files into a project with no loss of performance compared to importing the data into local storage within the project.
For example, the products link seen in the illustration above is a Manifold .map project file called products that was linked into the project. Manifold can then work with such linked-in projects with zero loss of performance no matter how big the linked-in project may be, just as if the contents of those linked-in projects were saved locally within the session's own project file. Working with linked project files is so fast that we would never guess they were linked files from performance alone if we did not see in the project file they were marked with a yellow database cylinder.
There are many ways to get data into project:
Open an existing Manifold .map project file that already has data in it.
Import data into the project from a file. See the Importing and Linking topic.
Link data into the project from a file. See the Importing and Linking topic.
Create - New Data Source, a more sophisticated form of linking, usually used with servers. See the Importing and Linking topic.
Copy and Paste data from other Manifold sessions.
Manually add data, such as by manually creating a table or a drawing and then entering records into the table or objects into the drawing.
Transform existing data to create new components.
Write queries, scripts and programs to create new data or to acquire it.
Use File - Open to open a Manifold .map project file or a Manifold .mxb archive file.
Newer .map files in Release 9 or later versions will open instantly, populating the Project pane with the contents of the .map file.
Files in .mxb archival format will open more slowly than .map due to the process of decompression and reconstituting the project from highly compacted, archival form. Using File - Save to save the project will thereafter save to .map format, Manifold's everyday, working project format. Use File - Export Project to write out an .mxb archive file, if desired.
Older .map files from before Release 9, such as for Manifold Release 8, will be loaded and automatically migrated into Release 9 project format when opened by Release 9 . Caution: saving such a project to the same .map file will replace the old Release 8 format with new Release 9 .map format which older Manifold products (Release 8) will not be able to read. Use File - Save As to save the modernized project to a new and different file name to avoid overwriting the old format.
Large .map files in older Manifold formats, before Release 9, can take a long time to be converted into Release 9 .map file format. Be patient in such cases as once the project is saved by Manifold as a Release 9 .map file it will thereafter open instantly.
If we launch more than one Manifold session we can open different projects with those sessions and then Copy and Paste between projects. Because Manifold projects open instantly they make very good storage containers for collections of data. Users will often have terabytes of folders on disk that are full of Manifold projects that contain their data holdings. The same .map format is used for all Release 9 applications, so Manifold System can open and work with .map projects created and saved by any Release 9 version.
We can use Manifold projects by linking them into our project or by popping the project open in a separate Manifold session and then copying and pasting from it to our current session. The best way to get data from one Manifold project into another is to create a data source for that other project within the session where we are working. That is generally faster than making copies using the Windows Clipboard.
Tech tip: When copying records from a table and pasting into another table, to add records to the destination table we should use the Ctrl-Shift-V command for a Paste Append. A paste append will automatically avoid collisions between existing and pasted values of the mfd_id key field used in tables that contain data for drawings and images because the pasted records will automatically be given new, unique mfd_id values. If we neglect to use Ctrl-Shift-V we might not be able to paste any records because mfd_id values must be unique: if the records we would like to paste have any mfd_id values that already exit in the destination table they will not be pasted into that table.
We can use the File - Create menu (or right-click into the Project pane and choose from the context menu) to create a new component such as a table or a drawing and then add data manually to that component, or add data programmatically, taking data from some process that generates data. The simplest way to add data manually is to create a table, specifying what fields we want in the table and to then proceed to add records one by one manually to the table. We are not going to manually add a billion records to a table but a few hundred are easy to add and some organizations over time manually will create tables with thousands of records.
We can also create a drawing and then manually create objects in that drawing. Manifold System provides exceptionally rich editing capabilities for drawings.
We can utilize a variety of automated means to create new components and to populate them with data. We can use SQL to create new tables and other components, either from existing ones or as a result of computations. We can use SQL or scripting or a combination to connect to external sources, for example, external sensors such as GPS devices or cameras which generate data, and to capture that data within our project.
Adding data from previously saved Manifold .map project files:
.mxb project archive files can be dragged and dropped into a project just like .map files.
There are three main ways of saving data from a project:
Save the project to a .map project file.
Export data from a component into a file.
Automatic dynamic connections to writable linked files or data sources.
See the Exporting topic for information on how to export and save data from all types of components: tables, images, drawings, maps, and other components.
A few words on each option:
An asterisk * appears next to the project name if the contents have been changed since the last time we have saved the project. Using File - Save to save a project will save all data stored in that project within the .map file created for the project and it will also save all components in the project including links for linked components and connection information for data sources. Until we save the project any edits made to local components are maintained in temporary Manifold storage. If we quit without saving the project those changes to local components will not take effect in the project file. By default, Manifold will prompt us to save changes before closing a project.
Note that data within linked components and data sources is always stored within the files that were linked or the data sources. When we save a project file that only saves pointers to where the data is actually stored.
Saving .map project files is reasonably fault tolerant. See the Notes section in this topic for more info.
MXB archival files are for archiving and exchanging data using the smallest possible project file size. They are small, but slower to create or to open than a .map project file. To export a project as an .mxb archive file, choose File - Export Project, choose MXB as the file type, provide the name desired and press Save. Exporting to .mxb files takes significantly longer than saving a .map project file, which usually is very fast.
Right-clicking onto a component in the project pane and choosing Export in the pop up menu will allow export to those file formats Manifold supports for that type of component. See comments in the How to Edit a Single File topic.
If we have connected to a data source such as an Oracle Server, PostgreSQL Server, ESRI geodatabase, GPKG, MySQL, SQLite or any other data source, even a web server based data source, we can harvest the entire contents of that data source and export into either a .map project file or an .mxb archival project file. Want to capture an entire ESRI geodatabase in a single, high speed .map project file? Easy: link the geodatabase into a project as a data source, open it to show the hierarchy within, and then right-click onto the data source, choose Export and then export as a .map or .mxb project file as preferred. In one step we can convert a messy set of many dozens of files into a single, modern .map project that opens instantly and features massively fast and reliable storage.
Manifold connects to linked files and to data sources in read/write mode if possible. As with most DBMS products, editing the data in a writable linked file or a writable data source (the default) will immediately change the saved data. As with most DBMS products, there is no "undo" for such edits. See the discussion in the Importing and Linking topic.
See also the Where Manifold Data is Stored topic.
Manifold .map format is fast - really fast. For larger files it is usually far faster for spatial data on almost any machine than any of the classic GIS or spatial data formats. In many cases for spatial work it will be faster than even enterprise-class DBMS storage, even when large amounts of data are stored.
Manifold .map format is so fast that when building libraries of data that might be used in future projects it pays to create those libraries as collections of .map files, importing different data sets of interest from various older formats into .map files. Doing that will allow instantaneous opens and will also allow use of "stock" data by linking those .map files into new projects with zero performance penalties. The one-time tedium of importing a big file into Manifold .map format will be long forgotten in the many times thereafter the big file opens instantly and the data springs immediately into use. Experience such speed once with a big file and you will not want to go back to slower formats.
Even better, linking a Manifold .map file into a project or connecting to that .map file as a data source provides a means of cascading the contents of projects with zero loss of performance. Projects can be linked one within the other to many levels without any performance loss at all. When we link other Manifold .map files into our project we can use the contents of those files as if they were in the local file with absolutely zero delay. That remarkable characteristic allows us to organize our data by using .map files to group data, either stored within a .map file or via links within a .map file, into logical and convenient collections.
For example, we might have one project that contains one or more data sources where a complex, difficult to remember connection string is used to connect to those data sources. We can give those data sources simple, easy to remember names. Whenever we need to use one of those data sources we can connect to the .map file that hosts the desired data source.
To take another example, we might have a collection of standardized data sets for many different countries kept as a library of .map files. We can link in whichever one we need for a given project and performance will be as fast as if the data were imported into that project's own .map file. See the discussion for Nested Data Sources in the Tables topic.
Links and data sources take up virtually no space within a project file (the data remains stored in the linked file or the data source) so we can have many project files with links and data sources in them for organizational convenience without duplicating the potentially vast amount of storage space to which they refer.
Other applications, for example GIS packages such as Manifold Release 8 or other GIS packages, can connect to Manifold .map projects via Manifold's ODBC driver to:
Access data within Manifold .map projects. Applications can read and write data in Manifold tables, drawings or images, depending on the capability of the application. Most GIS applications, for example, can link to drawings stored within Manifold .map projects.
Connect through Manifold to access data from other sources that are linked into a Manifold project. This makes it possible for applications which do not have the very wide-ranging capabilities of connecting to various data sources to connect through Manifold to those data sources. LibreOffice, for example, cannot connect directly to an ESRI GDB file geodatabase, but by connecting to a Manifold project that has an ESRI GDB data source in it LibreOffice can utilize Manifold as an intermediary to link that GDB data into LibreOffice as if GDB was a built-in capability of LibreOffice.
Third party applications can also use the Manifold ODBC driver to connect directly to any data source or file format that Manifold knows, so long as Manifold is installed on that machine. The Manifold ODBC driver automatically calls Manifold's internal capabilities to form a connection to the data source or file format.
See the examples listed in the Interoperability Examples topic for step by step instructions on how to connect to Manifold using ODBC.
Linking a file or a .map project into a Manifold project by default links the file or .map project both for reading and writing, and restricts the use of that file or .map project to only one Manifold session. If we link a file or a .map project read-only into a Manifold project, then other Manifold sessions can also use that same file or .map project so long as they also link that file or .map project into their projects read-only.
By adding a .map project to another project as a read-only data source, we can share a single .map project for simultaneous use by multiple users on different machines. That's very easy to do and it allows us to keep frequently used data in archived .map files.
The quickest way to link a file read-only into a Manifold project is to Drag and Ctrl-Shift-Drop it into the Project pane. Drag and Ctrl-Shift-Drop means to link the file read-only.
See the Example: Multiuser Sharing of .map Projects topic for a quick example.
Manifold's world view is that a project consists of databases that store entities of different types. Those entities are called components, typically taking their data from some associated table.
All types of components have metadata that is stored in a system table named mfd_meta. The names and types of all components are listed in a system table named mfd_root.
Data sources
are external files or databases. A database can store a reference
to another database. This is done by creating an entity of the type data source in the database that
refers to another database. Data sources can be nested
with no limit to nesting depth.
Data sources with a lock
icon have been linked read-only.
Read-only data sources show a database icon with a lock,
whether they have been opened or not. Their contents can be
viewed, but nothing in the data source can be changed.
The system supports many types of data sources, roughly classified as:
Databases, for example, Manifold Server, Oracle, PostgreSQL, MySQL, SQL Server, generic connections such as ODBC and so on.
File databases, for example, Access .mdb files, sqlite files and .dbf files.
Files , for example, .shp, rasterlite files, .e00, .bmp and .dgn files.
Web servers, for example, WMS or WMTS servers, Manifold image servers and OSM.
The Project - The .map project itself is a data source.
Databases and file databases differ from all other types of data sources in that they normally support storing multiple tables (albeit usually with various limitations) and can thus store many components at once. They are also normally updatable in that they allow the creation and deletion of components within them.
Databases and file databases can also use Manifold dataports, which are pluggable pieces of logic designed to interpret raw data contained in databases. For example, the Manifold dataport for ESRI ArcSDE data interprets raw blobs and metadata within ArcSDE for structured use within Manifold as drawings.
A dataport might allow one-way conversion (reading) or two-way conversion (reading and writing) with two-way conversion implemented in Manifold dataports whenever possible and sensible. All dataports available for a specific type of database are turned on by default with provisions to turn off some dataports if desired, for example, for performance reasons.
File databases and files differ from all other types of data sources in that they have a recognizable physical file somewhere within the file system, a characteristic that Manifold can exploit.
For example, Manifold can place additional files near such files to cache some data derived from the file for faster access. In such cases Manifold might create a cache file containing a spatial index for faster use of a file data source that contains geometry data but which does not in the native file format support spatial indexes. Manifold usually will provide options such as whether to allow creation of such additional cache files and so on.
Manifold can also cache data for data sources that are web servers, but that is done by creating tables in the parent data source, if the parent data source exists. Although a parent data source might not exist in purely programmatic use via the script API, a parent data source always exists otherwise because there is always a project and a .map file associated with that project. Even projects that have not yet been saved to a named .map file work on the basis of a temporary, unnamed and as yet unsaved .map file.
Probably the most frequent use of nested data sources is when a data source has been created in a given Manifold project and then the .map file for that Manifold project is used to create a data source in another Manifold project. Nested .map projects are a great way to serve a hierarchy that can contain hundreds of .map files through Manifold Server.
For example, we can create a data source in a project and then save that project as a .map file called myproject.map. The myproject.map project has one data source in it, that is, a data source nested one level down within the project. Suppose now in a different Manifold session we create a new project: we can create a new data source in that new project from the previously saved myproject.map file. When we expand that data source we will see within it the data source previously created within the myproject.map project. In our new project that data source is now two levels deep.
Nested data sources can be very convenient for organizing many projects and sub-groupings of data. Suppose we have a worldwide business with tables organized by regions and subregions in each country with databases for each. For example, in the Centre region of France we have databases for the various departments of Centre such as Cher, Eure-et-Loir, Indre and so on. (In France, sub-regional administrative divisions are called "departments," similar to how US states have "counties.")
To keep these organized we use .map files with nested data sources. At the lowest level for each department or subregion of a country we have a .map file that contains tables for that department, for example, a .map files that hold tables for Cher, one that holds tables for Eure-et-Loir and so on. To group those for convenient access for regional projects we have map files for each region such as a .map file for Centre that has data sources created in it from the various departmental .map files for that region. For national projects we have a .map file for France that has data sources created in it from the various regional .map files, such as Centre, Normandie and so on. At the continental level we have a .map file for Europe that has data sources created in it for the various country .map files such as for France, Germany, Italy and so on.
We can open a worldwide project that has data sources created in it from or the European, North American and other continental level .map projects and then drill down to the tables all the way within the lowest subregional level. All of the different data sources will appear nested within each other in the Project pane's hierarchy. In the illustration below we've drilled all the way down through nested data sources to open the Customers table for the Eure-et-Loir department.
To help us keep track of the origin of a component, component windows (such as a table window) will note the data source of the component listing nested names in order separated by a < character. In the example above the Customers table is within the Eure-et-Loir data source that is in turn within the Centre data source that is within the France data source within the Europe data source.
There are many useful aspects to nested data sources but one of the most convenient is that we can reach out through any of them that are read/write to work with, including changing, their contents as if they were at the top level of the project.
For example, as in the illustration above we could slow-click on the Categories table to rename it if we wanted, just as if it were a table hosted in the top level project and not something four or five levels deep in one project file and data source nested within another.
Nested data sources require no storage (their data is stored in whatever origin file or server to which the data source refers) and they may be used with zero loss of performance, just as if they were part of the parent project at the top level. When we import a large image into a Manifold project it will open instantly. If that same large image appears in some data source nested five levels deep within Manifold projects it will still open instantly when we double-click it.
We would like to keep track of archaeological sites in Germany, France, Italy, Turkey and Bulgaria. We have Manifold projects for each of those countries saved to .map files called germany.map, france.map, italy.map and so on. Each of them has many large tables, drawings, images and maps.
Our new research topic involves trends in the French city of Strasbourg adjacent to the border with Germany. For that work we will want data from our projects in Germany and in France. We create a project and save it as strasbourg.map.
We can easily use our stockpile of German and French data within that new project by using the File - Create - New Data Source dialog to create new data sources called Germany and France which use the germany.map and the france.map project files as the source of their data. Doing so will create two new data sources in the Project pane, each marked with a yellow database cylinder and a + icon button we can press to expand the data source. If we click the + icon button next to the new France data source to expand it within the Project pane hierarchy we will see the contents of that project.
Suppose now we would like to use an image server to create images of the region around Strasbourg. To keep all of our data for France together in the same archive we would like to create that image server within the database that the france.map project file represents. That's easy to do. With the focus within the France data source hierarchy (just clicking the + button to expand that data source has already moved the focus into that data source's hierarchy) we create a new data source using a Manifold image server, for example, a Virtual Earth (VE) Hybrid image that shows streets overlaid on satellite photos.
When we do that we'll see files for an image server data source created in the usual way, all within the France parent data source hierarchy. We can open the VE image server's image and see the visual display of satellite imagery overlaid with roads and other vector data provided by VE. If we browse through the hierarchy of the France data source we will also see the .Tiles table Manifold created to cache tiles fetched from the VE server over the web that were used to create the image we see. The full name of that cache table will be something ungainly, but self-documenting, such as "imageserver_VirtualEarthHybridImage.Tiles." The tile cache table was created within the France data source because that is the parent data source of the newly created image server data source nested within.
To recall where we stand in the hierarchy of nested data sources, we are working with a project saved in a file called strasbourg.map. That project in turn has two data sources in it called France and Germany which seamlessly link into the project the Manifold databases and all their components that live within the france.map and germany.map project files we had created during earlier work.
Within the France data source we have created yet a third data source, this one coming in from a web server in the form of a Manifold image server connection to hybrid imagery served by Virtual Earth. When we look at the images served up by VE, panning and zooming into them to see our area of interest in Strasbourg, France, we are looking in to the third level of nested databases: the VE image server data source nested within the France project data source that is nested within our current Strasbourg project.
There are two things that are very cool about this situation. First and foremost at the high end of the "cool" scale is that we encounter absolutely zero performance loss from nesting databases / data sources in this way. The entire contents of the france.map project are seamlessly available to us within our strasbourg project just as gobsmacking fast as if we had opened the france.map project within its own Manifold session and were not using it within the strasbourg project as a nested data source. We could nest data sources ten levels deep and still have zero performance loss.
The second very useful characteristic of nested data sources is that changes propagate as we would like and would expect. If we make any changes to a nested data source, for example, adding an image server data source nested within, that change updates the parent data source as we would expect.
If we had nested an image server data source within a SQL Server data source the addition would take effect immediately, since any edits within SQL Server take effect immediately, as anyone who works regularly with SQL Server well expects.
In the case of a data source that is a .map project file, when we added the VE image server data source as a nested data source we would notice that a small asterisk * symbol appeared in the corner of the yellow database cylinder icon next to the France data source name. That asterisk reminds us the project needs to be saved for the changes to be saved to the .map file used for that data source.
When we next save our strasbourg.map project the .map files used within any nested data sources will also be saved along with any changes we've made, such as adding a nested VE image server data source within their database.
Suppose we save our strasbourg.map file and exit Manifold. We launch Manifold again and we open the france.map project. We'll see that the project now has a new data source in the project, the VE image server data source. It also has the ".Tiles" cache file still full of tiles from our previous browsing of the VE image server display. Because the france.map project was the parent data source when we added the VE image server data source in our earlier session that nested data source and any accessory files that appeared with its use all were created in the france.map project.
Suppose we close that project and now we create a new, empty project that we save under the name alsace.map (Strasbourg is the capital of the Alsace region of France). We use File - Create - New Data Source to create a data source that uses the france.map project as its database. When we open that data source we will see the nested VE image server data source still there, ready for use.
Such effects make it easy to create repositories of related data and to organize additions to data so they go into the right repository. If we are going to do work in the region around Strasbourg it could be useful to have our data sets for Germany immediately at hand as well as the data sets for France that we expect to use in our work with Strasbourg. But we also may as well ensure that various additions involving French data for Strasbourg get created within the French compilation so that they can be used in any other projects that link in the French repository.
Consider an example that shows data source nesting on a machine that has Microsoft SQL Server available for use. We open Manifold and create a new project. We use File - Create - New Data Source to create a new data source for a SQL Server database which we will call SQL Server. The new data source appears in the project pane with a yellow DBMS cylinder icon and a + sign indicating that data source can be expanded, that is opened.
In the Project pane we click on the + icon to open the SQL Server data source. Next, we create a new data source for a Manifold image server, for example, the Virtual Earth Street Map image server, allowing it to cache data whenever it wants to. A new data source for the image server appears, under the SQL Server data source hierarchy since that was opened and had the focus when we created the data source for the image serer.
We open the image server street map image and see it download tile data from the web to form the expected image of a street map. If we look closely we note that it has also created a new table in the SQL Server data source with a .Tiles extension to the name as a table to hold the tiles it has downloaded. The .Tiles table was created in the SQL Server data source because that is the parent data source to the new image server data source we created using the VE image server.
Suppose we now exit Manifold. All the tile data and the created data source are still in the SQL Server database we used. Suppose we launch Manifold again and create a new project. Within that new project we use File - Create - New Data Source to create a new data source that once again uses the same SQL Server database we used earlier. When we click on the + icon to expand that newly created SQL Server data source we will see the data source for the Manifold image server that we created in our earlier Manifold session and also we will see the .Tiles extension table it created with tile data it had cached.
This example takes advantage of the sophisticated capabilities of SQL Server. It won't work if we try it on a data source that we created using an .mdb file database because the limitations of .mdb for the names of tables will prevent Manifold from creating tables using the default naming scheme used for image servers. It won't work with sqlite files either since those cannot host the indexes Manifold would like to create. But it will work with other sophisticated DBMS environments like PostgreSQL or Oracle.
Release 9 .map format is a different, much improved .map format that is not backward compatible with Release 8, the prior generation of Manifold GIS products. Release 9 can read old format Release 8 .map files, but Release 8 cannot read new format Release 9 .map files.
When Release 9 opens a Release 8 .map file the contents of the project are automatically converted, as much as possible, into their Release 9 equivalents. Linking a Release 8 .map file or creating a new data source based on a Release 8 .map file will also cause that .map file to be converted into Release 9 .map file format.
Make a backup copy of any pre-Release
9 .map files you wish to
retain in the old format before you open them or use them in a Release
9 project.
If we open a Release 8 .map file in Release 9, if we try to save that project Manifold will automatically increment the name so we save it to a different name, not overwriting the original Release 8 file. This helps preserve the original Release 8 file for use with Release 8. If we insist on writing it to the same name we can do that, but in that case the new .map file will be a Release 9 .map and not readable by Release 8.
To exchange data between Release 9 and Release 8 consider the following:
To work simultaneously with the same data using both Release 9 and Release 8, use a multiuser database to store the data. Both Release 9 and Release 8 can connect to many different multiuser databases to read/write data.
If access does not need to be simultaneous:
To utilize data from Release 8 within Release 9, launch Release 9 and open the Release 8 .map file. From Release 9, save that project using a different name for the .map file since a .map file saved by Release 9 is not readable by Release 8.
To read/write data from either Release 9 or Release 8, and to maintain the data in Release 9 storage, keep the data in a Release 9 .map file and connect to it from Release 8 using Release 8's built in ODBC driver for connecting to Release 9. The most recent Release 8 updates include an ODBC driver for Release 9.
To read/write data from either Release 9 or Release 8, and to maintain the data in Release 8 storage, keep the data in a Release 8 .map file and connect to it from Release 9 using Release 9's built in ODBC driver for connecting to Release 8.
To interchange data using a third party format, export data from either Release 9 or Release 8 to a format such as shapefiles that both Release 9 and Release 8 can import.
Opening a Release 8 .map file in Release 9 will automatically migrate images with two nuances:
Migrated images will no longer have a Level field, which is not used in 9.
Compressed images within the Release 8 .map project will be migrated into a new ECW / JP2 image file created in the same folder as the opened Release 8 .map project, and automatically linked into the 9 project. The new file will be named by concatenating the Release 8 project name and an iterated number starting with 1. If we subsequently open the same Release 8 .map project in Release 9, the new ECW / JP2 image file will be used automatically, to avoid having to re-migrate the compressed image again.
.map format is designed to be as fast as possible for the GIS work most people do most of the time. A key part of that is enabling very fast saves after changes, even if the project is a very big project. Being able to do fast saves is critically important to encourage safe work habits that will protect the revisions we make in case of hardware or power failures.
However, to accomplish such fast saves Manifold will not completely re-optimize internal structures in some cases, specifically, after deletions of very large data. Such optimizations take time, and when we want to have instantaneous saves we do not want to waste time on optimizations that are rarely needed.
Consider an example: suppose we start with a .map project that is over 20 gigabytes in size, that contains vector layers that each have millions of records and hundreds of fields, totalling to over 100 million various data values in all the records. For example, that might be a project that contains parcels and attributes for real estate parcels for several states in the US.
Suppose we are only interested in parcels for one town, so we select those parcels and then press Ctrl-I to invert the selection, and then we delete all other parcels. Suppose that ends up deleting most of the parcel records, so the project now will contain only a few thousand parcels instead of millions. However, when we save the .map project it still will be a large file, possibly around 800 MB or so in size. That is smaller than the original 20 GB, but it is not hundreds of times smaller like it could be. Why is that?
The reason is that when very many records and fields are deleted from very big tables, most of the space occupied by their data is reclaimed, but some of the data used in infrastructure that supports fast reads and fast saves is not. The bigger the original table is, the more space will be retained after very large deletes. Tables will reserve a few bytes per original value in the storage structure, so even after deleting more than 98% of the original table that storage space will still be retained, resulting in a larger .map file than could be stored given optimization.
To create the smallest possible .map files after very large deletions, save the project to .mxb format, which contains only the bare minimum of structure required to archive data. Next, delete the .map file and then open the .mxb format project file, which automatically will re-build the .map project and file, and which will rebuild all the internal structures required for speed, but now optimized for the much smaller amount of data in play. The new .map file will be smaller, possibly very much smaller, than the .map file that was exported to .mxb.
Note that such compactions are not every day operations since usually we either work with about the same project sizes or we do work that increases the size of projects, which automatically grows the size of internal storage space. It is a fairly unusual approach to GIS workflow to start with a 20 GB data set when you only need 10 MB of it, and then to delete almost all of the starting data set to just get the data of interest. That Manifold makes such workflow possible and easy is pretty cool, but that is still fairly rare workflow.
We only need to consider manual compaction in cases where we do dramatic reductions in the size of very big projects, and where it also is important to us to reduce the size of the .map file to the minimum possible. .map files that are bigger than they could be will still work fine: Manifold is so fast that most users will not notice the much larger file size, as it will still load, open, and save instantly. But if we can make an 800 MB .map file smaller than 10 MB, that is worth a round trip through .mxb format. If you delete almost all of a project that started off being a few gigabytes or more in size, consider doing a quick compaction to reduce the size of the resulting .map project file.
In most cases, we will run out of machine resources before hitting any limitations of Manifold. See the Limitations topic for a current list of limitations.
Importing bigger data - Importing bigger data into a project can be slow, possibly very slow for large files, but that only has to be done once. After that everything goes really fast. Once we save the imported data as a Manifold .map project it will usually be in far faster form than the original format. Save a 100GB image or a big GIS drawing as a Manifold project and Manifold thereafter will open it instantly with performance most other environments cannot match.
Fault Tolerant .map Saves - The process Manifold uses for saving a .map file is reasonably fault-tolerant, taking measures to greatly reduce the chances of .map file corruption as follows:
When saving to a new .map file using File - Save As, Manifold will not make any changes to the original .map file.
When saving to an existing .map file using File - Save, Manifold will first collect changes into a separate, accessory .SAVEDATA file created in the same folder as the .map file. Only after changes are saved to the .SAVEDATA file will Manifold proceed to write the same changes to the .map file. If any failure occurs while collecting changes in .SAVEDATA the .map file stays intact and uncorrupted.
If the collection of changes in .SAVEDATA works OK but there is a failure when writing changes to the .map file, Manifold will repeat the save operation when opening the .map file. As a result the .map file will either receive none of the changes or eventually will receive all of them.
Saving data keeps internal memory structures stable so that if the save fails, for example, due to not enough disk space, the combination of data in the partially-saved image and in memory remains coherent with no losses. The user can then take a few moments to throw off panic, restore a calm demeanor, free some disk space or find a different location for saving with more space, and repeat the save.
Fault tolerance through stable internal memory structures applies to all failed-save scenarios where Manifold is still running and has not itself been nuked by a catastrophic failure, such as a power outage with no battery backup. For example, it applies to saving to a new file with File - Save As and also to saving with File - Save to an existing file with or without use of .SAVEDATA.
Using .SAVEDATA has a performance cost, albeit a cost which is not overly large due to two factors: first, the time to save is proportional to the amount of changes and, second, much of the data between the two save stages, first into .SAVEDATA and next into the .map file, will end up being cached by the operating system. But since the cost of using .SAVEDATA is non-zero Manifold provides a connection string option to turn off the use of .SAVEDATA for a particular .map file.
In addition, when connecting to .map files as a data source we can turn off the use of .SAVEDATA by unchecking the option for Fault-tolerant save in the New Data Source dialog.
Important: By default .map file data sources use .SAVEDATA. If .SAVEDATA is not used and the system crashes during a save operation, literally between the writes, the resulting .map file will no longer load. Given that the universe is designed by Fate to ensure both the main power and battery backup together will fail exactly in the middle of a critical write, it is generally a bad idea to tempt Fate by turning off the use of .SAVEDATA. Fate always wins.
Open a .map file from a command prompt - For users who want to use a batch file to launch Manifold or Viewer with a particular .map project file, Manifold supports the usual Windows way of opening a file as a command line argument for the opening program: Running manifold.exe myproject.map will launch Manifold and open myproject.map in Manifold. Use double quotes around the file name if there are any special characters or spaces in the .map file name. For example, we would write manifold.exe "-this is my latest project.map".
Example: Multiuser Sharing of .map Projects
Example: Convert an ESRI File Geodatabase into a .map Project
Example: Convert an ESRI Personal Geodatabase into a .map Project
