Manifold project files, called projects, in .map format are the native "documents" for Manifold. The File - Open command will open an existing Manifold .map project file and the File - Save command will save a Manifold project in .map project file format. Manifold .map format is also referred to as MAP format, using all-capital letters.
Manifold stores everything it does in a project within a Manifold .map file. A .map project file is an incredibly fast, very rich, file database that can store data of all kinds. It can store large, complex entities like maps, drawings, scripts, queries, tables and much more, and it can store links to other databases, like Oracle or PostgreSQL DBMS servers, and it can store connections to web data sources providing terabytes of data. A .map project file can have thousands of entities in it and itself be hundreds of gigabytes in size.
Manifold .map project format is an operational format: it is designed for everyday work. Manifold .map format is built for maximum speed, and for nesting of projects with zero performance loss. .map project format is incredibly fast, convenient, and very robust, a perfect format for daily work.
Manifold also has another format, .mxb format, that is designed to store projects as compactly as possible for archival storage and for exchanging data. .mxb format is also referred to as MXB format, using all-capital letters. MXB is slower to write and slower to open than MAP and is cannot be used operationally for nesting projects. We can open a .mxb file with Manifold, and Manifold will immediately decompress it to create a .map format project that it will use. MXB is a highly compressed and robust format that minimizes file size, perfect for archival storage and for minimizing the size of files for transmission. MXB is also useful in ensuring maximum data reliability. See the For Maximum Reliability topic.
Manifold .map files at the present time do not allow simultaneous read/write access by multiple Manifold sessions. If a .map file has been opened in one Manifold session it cannot be opened in another. We can, however, link a .map file into several different Manifold sessions if we create the data source for each with the read only check box checked.
If we require simultaneous read/write access to data by multiple Manifold sessions, such as by different people on different machines, we can store the data in a DBMS, such as PostgreSQL/PostGIS, Oracle, MySQL, SQL Server, DB2, or other DBMS.
A project is the collection of stuff we are working on in any given Manifold session. It is all the different things we have in our Manifold workspace in that session. The Project pane to the right of the Manifold desktop will show the contents of the project. See the Project Pane topic.
When we use File - Save we save the contents of the project into a Manifold .map project file. Stuff that has been imported into the project will be saved within the .map file. Stuff that has been linked into the project will remain saved out in whatever form from which it was linked, with the information needed to link it again saved into the .map file.
When we use File - Open to open a previously-saved project file we get that collection of stuff back, exactly the way we saved it for data imported into the project, complete with connection information, like paths and URLs, to links and data sources exactly the way they were when we saved the project.
To fit into this documentation, illustrations show an artificially small Manifold desktop, with only a few panes, docked to the right side. In real life we use a much larger Manifold desktop, and all panes would be turned on, with some panes docked to the left and others docked to the right. Right-click a pane's tab to change where it is docked. Manifold will remember that new arrangement for our next session.
Manifold always launches with a project, a new blank project called New Project if not opened with an existing project file. An asterisk * appears next to the project name if the contents have been changed since the last save.
A new project is empty but for a System Data folder containing tables Manifold uses to manage the project.
Clicking the + icon opens the System Data folder to show the special tables it contains, normally the mfd_meta and mfd_root tables.
Other tables, such as a mfd_meta_source table for some data sources, or mfd_collate and mfd_srid tables for DBMS data sources, might appear in the System Data folders of a data source when linking to various data sources.
Tables in System Data folders store operational information for the data source. We do not need to work with the System Data tables during routine Manifold use. However, more advanced users may want to use them to manipulate components. Since they are small, simple tables and they always are at hand we can use them to illustrate examples in this documentation.
In the illustration above we have used File - Open to open a previously-saved project called Aus Hydro.map. It all comes back, including a data source that is a Microsoft Bing image server that is ready to go.
Tech tip: Within the Project pane most Windows moves we know, like Copy and Paste, will work. For example, we can Copy a table and Paste it within a folder. We can drag and drop a table into a folder to move it there. One modification to a standard Windows key: drag and drop in Windows is a move command and Ctrl-drag and drop is a copy command. In Manifold, drag and drop also is a move command but Shift-drag and drop is a copy command.
We can organize what is in our project using folders, like the System Data folder seen above, a good idea if our projects include many components. The Project pane itself includes useful features, like filtering, to help us zero in on the components we want in a project that may contain very many.
When we add stuff to our project that stuff will appear in the Project pane as various items, called components. Components are items such as tables, images, scripts, data sources and so on depending on what sort of data is added. The project pane will show all components in our project with an icon next to each to indicate the type of component.
Tables are the center of everything in Manifold. All data in Manifold is stored in tables in one form or another. That's good because it means we can get at anything with SQL. Nothing is hidden.
Other components such as images or drawings are visualizations of data stored within some table. Map components show other visual components, such as images, drawings and labels, as layers within a map window.
Tables - Where all data within Manifold is stored in one form or another. Opening a table window shows a table in classic row and column presentation.
Drawings - A visual display of vector data stored within tables using a variety of geometry types. The same data can be seen in multiple different drawings using different styles, and the same drawing can be open in multiple windows with different selections and views in each.
Data Sources - Data linked into a project from an external source, which could be a file, a file database, a database serve or a web server, covering virtually every database and file type encountered in database or GIS work.
Images - A visual display of raster data stored within tables as tiles using a variety of different data types and channel combinations. All rasters are called images in Manifold even those which are terrain elevation or other non-photographic data.
Labels - Text annotations automatically created from the fields of a drawing's table.
Layouts - Layouts are compositions on virtual sheets of paper, allowing us to create documents showing maps or other components as PDF documents or hard copy printed sheets.
Locations - Locations save a particular geographic location with an associated scale, thus saving a given view in a map or other window to which we can return.
Maps - Consist of a stack of layers where each layer is a drawing, image or labels component.
Queries - Written in SQL, queries manipulate data and projects, for example, creating new tables and other components, performing calculations, altering the structure of databases and tables, and extracting, editing and analyzing subsets of data.
Scripts - Manifold includes built-in support for scripting in eleven different languages with six to eight always available and the remainder easy to install. Scripts provide custom capabilities and can automate virtually anything.
Comments - Text saved and displayed in a comments window provides a simple but convenient way to save notes about a project or to save text as a scratchpad.
Folders - We can organize our project by using folders.
Components we see in the Project pane may be a link to data stored somewhere else or they may be a local component for which the data is stored within the project file. Linked components don't take up any space in the .map file that stores the project. They're just pointers to the file or data source somewhere else where the data is stored. Local components are stored in the project's .map file, which will grow in size as we add local components to a project. .map files can get very big if we add large components to them.
A yellow database cylinder marks a linked file or data source in the Project pane. The yellow database cylinder tells us that everything within that hierarchy is coming into the project from some source outside of the project. The source will also have its own System Data folder with information on the linked source. In the above example the Products table seen under the products hierarchy is coming in from some data source outside of the project. In contrast, the bronze image and the bronze Tiles table are both local components that are stored within the project file.
Linking a data source into a project also creates a section of the project that is a portal to another world, where what we see in that data source is not in the project file but is a view into a different file or data source (like a database) which may or may not be writable. See the discussion in the Portals to Other Worlds section of the Project Pane topic.
When writing queries, we can refer to tables within a data source using the syntax [datasource]::[table]. For example, we could write a query that selects all records from the Products table within the products data source in the illustration above as:
SELECT * FROM [products]::[Products];
See the Example: Refer to a Table within a Data Source topic for more information on referring to tables within data sources.
When we use File - Save to save a project we write that project into a file database stored in Manifold .map file format. The .map file we create saves all the data for local components as well as saving links, Project pane folder hierarchies and other housekeeping information for the project. The next time we open the project file in Manifold we will get all of that back, with the Project Pane showing components and folders organized exactly like they were when we saved the project.
Manifold project files can save large amounts of data, hundreds of gigabytes, and provide performance second to none in database applications. That makes Manifold itself a fine choice for a file database system, using Manifold .map format project files for storage, without being forced to use an external DBMS just because the tables, images, drawings or other data we would like to store and to work with are very large.
At the same time Manifold is exceptionally adept at working with and very fast at connecting to DBMS servers like Oracle, PostgreSQL, DB2 or SQL Server, so fast that Manifold will take full advantage of the best performance those systems can provide. That makes Manifold a great console for working with multiple different DBMS databases at once, with big data stored locally in the project file, or with big data stored in external DBMS servers or both at the same time.
Manifold can also link other, previously created Manifold project files into a project with no loss of performance compared to importing the data into local storage within the project. For example, the products link seen in the illustration above is a Manifold .map project file called products that was linked into the project. Manifold can then work with such linked-in projects with zero loss of performance no matter how big the linked-in project may be, just as if the contents of those linked-in projects were saved locally within the session's own project file. Working with linked project files is so fast that we would never guess they were linked files from performance alone if we did not see in the project file they were marked with a yellow database cylinder.
There are many ways to get data into project:
Open an existing Manifold or Manifold .map project file.
Import data into the project from a file. See the Importing and Linking topic.
Link data into the project from a file. See the Importing and Linking topic.
Create - New Data Source, a more sophisticated form of linking, often used with servers. See the Importing and Linking topic.
Create new components such as tables or drawings and add data to them.
Copy and Paste data from different Manifold sessions.
Write scripts and programs to create new data or to acquire it.
Use File - Open to open a Manifold .map project file or a Manifold .mxb archive file.
Newer .map files in Release 9 or later versions will open instantly, populating the Project pane with the contents of the .map file.
Files in .mxb archival format will open more slowly than .map due to the process of decompression and reconstituting the project from highly compacted, archival form. Using File - Save to save the project will thereafter save to .map format, Manifold's everyday, working project format. Use File - Export Project to write out an .mxb archive file, if desired.
Older .map files from before Release 9, such as for Manifold Release 8, will be loaded and automatically migrated into Release 9 project format when opened by Release 9 . Caution: saving such a project to the same .map file will replace the old Release 8 format with new Release 9 .map format which older Manifold products (Release 8) will not be able to read. Use File - Save As to save the modernized project to a new and different file name to avoid overwriting the old format.
Large .map files in older Manifold formats, before Release 9, can take a long time to be converted into Release 9 .map file format. Be patient in such cases as once the project is saved by Manifold as a Release 9 .map file it will thereafter open instantly.
We can use the File - Create menu (or right-click into the Project pane and choose from the context menu) to create a new component such as a table or a drawing and then add data manually to that component, or add data programmatically, taking data from some process that generates data. The simplest way to add data manually is to create a table, specifying what fields we want in the table and to then proceed to add records one by one manually to the table. We are not going to manually add a billion records to a table but a few hundred are easy to add and some organizations over time manually will create tables with thousands of records.
We can also create a drawing and then manually create objects in that drawing. Manifold System provides exceptionally rich editing capabilities for drawings.
If we launch more than one Manifold session we can open different projects with those sessions and then Copy and Paste between projects. Because Manifold projects open instantly they make very good storage containers for collections of data. Users will often have terabytes of folders on disk that are full of Manifold projects that contain their data holdings. The same .map format is used for all Release 9 applications, so Manifold System can open and work with .map projects created and saved by any Release 9 version.
We can use Manifold projects by linking them into our project or by popping the project open in a separate Manifold session and then copying and pasting from it to our current session. The best way to get data from one Manifold project into another is to create a data source for that other project within the session where we are working. That is generally faster than making copies using the Windows Clipboard.
Tech tip: When copying records from a table and pasting into another table, to add records to the destination table we should use the Ctrl-Shift-V command for a Paste Append. A paste append will automatically avoid collisions between existing and pasted values of the mfd_id key field used in tables that contain data for drawings and images because the pasted records will automatically be given new, unique mfd_id values. If we neglect to use Ctrl-Shift-V we might not be able to paste any records because mfd_id values must be unique: if the records we would like to paste have any mfd_id values that already exit in the destination table they will not be pasted into that table.
We can utilize a variety of automated means to create new components and to populate them with data. We can use SQL to create new tables and other components, either from existing ones or as a result of computations. We can use SQL or scripting or a combination to connect to external sources, for example, external sensors such as GPS devices or cameras which generate data, and to capture that data within our project.
There are three main ways of saving data from a project:
Save the project to a .map project file.
Export data from a component into a file.
Automatic dynamic connections to writable linked files or data sources.
See the Exporting topic for information on how to export and save data from all types of components: tables, images, drawings, maps, and other components.
A few words on each option:
An asterisk * appears next to the project name if the contents have been changed since the last time we have saved the project. Using File - Save to save a project will save all data stored in that project within the .map file created for the project and it will also save all components in the project including links for linked components and connection information for data sources. Until we save the project any edits made to local components are maintained in temporary Manifold storage. If we quit without saving the project those changes to local components will not take effect in the project file. By default, Manifold will prompt us to save changes before closing a project.
Note that data within linked components and data sources is always stored within the files that were linked or the data sources. When we save a project file that only saves pointers to where the data is actually stored.
Saving .map project files is reasonably fault tolerant. See the Notes section in this topic for more info.
MXB archival files are for archiving and exchanging data using the smallest possible project file size. They are small, but slower to create or to open than a .map project file. To export a project as an .mxb archive file, choose File - Export Project, choose MXB as the file type, provide the name desired and press Save. Exporting to .mxb files takes significantly longer than saving a .map project file, which usually is very fast.
Right-clicking onto a component in the project pane and choosing Export in the pop up menu will allow export to those file formats Manifold supports for that type of component. See comments in the How to Edit a Single File topic.
If we have connected to a data source such as an Oracle Server, PostgreSQL Server, ESRI geodatabase, GPKG, MySQL, SQLite or any other data source, even a web server based data source, we can harvest the entire contents of that data source and export into either a .map project file or an .mxb archival project file. Want to capture an entire ESRI geodatabase in a single, high speed .map project file? Easy: link the geodatabase into a project as a data source, open it to show the hierarchy within, and then right-click onto the data source, choose Export and then export as a .map or .mxb project file as preferred. In one step we can convert a messy set of many dozens of files into a single, modern .map project that opens instantly and features massively fast and reliable storage.
Manifold connects to linked files and to data sources in read/write mode if possible. As with most DBMS products, editing the data in a writable linked file or a writable data source (the default) will immediately change the saved data. As with most DBMS products, there is no "undo" for such edits. See the discussion in the Importing and Linking topic.
See also the Where Manifold Data is Stored topic.
Manifold .map format is fast - really fast. For larger files it is usually far faster for spatial data on almost any machine than any of the classic GIS or spatial data formats. In many cases for spatial work it will be faster than even enterprise-class DBMS storage, even when large amounts of data are stored.
Manifold .map format is so fast that when building libraries of data that might be used in future projects it pays to create those libraries as collections of .map files, importing different data sets of interest from various older formats into .map files. Doing that will allow instantaneous opens and will also allow use of "stock" data by linking those .map files into new projects with zero performance penalties. The one-time tedium of importing a big file into Manifold .map format will be long forgotten in the many times thereafter the big file opens instantly and the data springs immediately into use. Experience such speed once with a big file and you will not want to go back to slower formats.
Even better, linking a Manifold .map file into a project or connecting to that .map file as a data source provides a means of cascading the contents of projects with zero loss of performance. Projects can be linked one within the other to many levels without any performance loss at all. When we link other Manifold .map files into our project we can use the contents of those files as if they were in the local file with absolutely zero delay. That remarkable characteristic allows us to organize our data by using .map files to group data, either stored within a .map file or via links within a .map file, into logical and convenient collections.
For example, we might have one project that contains one or more data sources where a complex, difficult to remember connection string is used to connect to those data sources. We can give those data sources simple, easy to remember names. Whenever we need to use one of those data sources we can connect to the .map file that hosts the desired data source.
To take another example, we might have a collection of standardized data sets for many different countries kept as a library of .map files. We can link in whichever one we need for a given project and performance will be as fast as if the data were imported into that project's own .map file. See the discussion for Nested Data Sources in the Tables topic.
Links and data sources take up virtually no space within a project file (the data remains stored in the linked file or the data source) so we can have many project files with links and data sources in them for organizational convenience without duplicating the potentially vast amount of storage space to which they refer.
Other applications, for example GIS packages such as Manifold Release 8 or other GIS packages, can connect to Manifold .map projects via Manifold's ODBC driver to:
Access data within Manifold .map projects. Applications can read and write data in Manifold tables, drawings or images, depending on the capability of the application. Most GIS applications, for example, can link to drawings stored within Manifold .map projects.
Connect through Manifold to access data from other sources that are linked into a Manifold project. This makes it possible for applications which do not have the very wide-ranging capabilities of connecting to various data sources to connect through Manifold to those data sources. LibreOffice, for example, cannot connect directly to an ESRI GDB file geodatabase, but by connecting to a Manifold project that has an ESRI GDB data source in it LibreOffice can utilize Manifold as an intermediary to link that GDB data into LibreOffice as if GDB was a built-in capability of LibreOffice.
See the examples listed in the Interoperability Examples topic for step by step instructions on how to connect to Manifold using ODBC.
Release 9 .map format is a different, much improved .map format that is not backward compatible with Release 8, the prior generation of Manifold GIS products. Release 9 can read old format Release 8 .map files, but Release 8 cannot read new format Release 9 .map files.
When Release 9 opens a Release 8 .map file the contents of the project are automatically converted, as much as possible, into their Release 9 equivalents. Linking a Release 8 .map file or creating a new data source based on a Release 8 .map file will also cause that .map file to be converted into Release 9 .map file format.
Make a backup copy of any pre-Release 9 .map files you wish to retain in the old format before you open them or use them in a Release 9 project.
If we open a Release 8 .map file in Release 9, if we try to save that project Manifold will automatically increment the name so we save it to a different name, not overwriting the original Release 8 file. This helps preserve the original Release 8 file for use with Release 8. If we insist on writing it to the same name we can do that, but in that case the new .map file will be a Release 9 .map and not readable by Release 8.
To exchange data between Release 9 and Release 8 consider the following:
To work simultaneously with the same data using both Release 9 and Release 8, use a multiuser database to store the data. Both Release 9 and Release 8 can connect to many different multiuser databases to read/write data.
If access does not need to be simultaneous:
To utilize data from Release 8 within Release 9, launch Release 9 and open the Release 8 .map file. From Release 9, save that project using a different name for the .map file since a .map file saved by Release 9 is not readable by Release 8.
To read/write data from either Release 9 or Release 8, and to maintain the data in Release 9 storage, keep the data in a Release 9 .map file and connect to it from Release 8 using Release 8's built in ODBC driver for connecting to Release 9. The most recent Release 8 updates include an ODBC driver for Release 9.
To read/write data from either Release 9 or Release 8, and to maintain the data in Release 8 storage, keep the data in a Release 8 .map file and connect to it from Release 9 using Release 9's built in ODBC driver for connecting to Release 8.
To interchange data using a third party format, export data from either Release 9 or Release 8 to a format such as shapefiles that both Release 9 and Release 8 can import.
Opening a Release 8 .map file in Release 9 will automatically migrate images with two nuances:
Migrated images will no longer have a Level field, which is not used in 9.
Compressed images within the Release 8 .map project will be migrated into a new ECW / JP2 image file created in the same folder as the opened Release 8 .map project, and automatically linked into the 9 project. The new file will be named by concatenating the Release 8 project name and an iterated number starting with 1. If we subsequently open the same Release 8 .map project in Release 9, the new ECW / JP2 image file will be used automatically, to avoid having to re-migrate the compressed image again.
In most cases, we will run out of machine resources before hitting any limitations of Manifold. There are a few limitations that may be encountered even in large machines:
256 GB max size for .map files - Using 64-bit Manifold, the amount of data stored within a .map file cannot be larger than 256 GB, although a .map file can contain data sources linking to a DBMS that contain far larger data, petabytes, in size. The maximum .map file size is 64 GB with 32-bit Manifold. The 64-bit limitation will be increased in future builds.
256 GB max size for changes to data - .map files can work with data far larger than 256 GB when that data is stored in external data sources, such as an enterprise class DBMS. Although there is no theoretical limit to the size of data Manifold can work with on the desktop, a hard limit on the amount of data that can be changed in a single operation is 256 GB. For example, if we have a single image that is 1 terabyte in size we can display that and manipulate it, but if we want to change all of the values in all of the pixels in the image we will have to do that in four operations, each up to 256 GB in size. The 256 GB limit is the current, somewhat arbitrary, setting for temporary caches utilized internally by Manifold. This limitation will be increased in future builds.
See the Limitations topic for a current list of limitations.
Importing bigger data - Importing bigger data into a project can be slow, possibly very slow for large files, but that only has to be done once. After that everything goes really fast. Once we save the imported data as a Manifold .map project it will usually be in far faster form than the original format. Save a 100GB image or a big GIS drawing as a Manifold project and Manifold thereafter will open it instantly with performance most other environments cannot match.
Fault Tolerant .map Saves - The process Manifold uses for saving a .map file is reasonably fault-tolerant, taking measures to greatly reduce the chances of .map file corruption as follows:
When saving to a new .map file using File - Save As, Manifold will not make any changes to the original .map file.
When saving to an existing .map file using File - Save, Manifold will first collect changes into a separate, accessory .SAVEDATA file created in the same folder as the .map file. Only after changes are saved to the .SAVEDATA file will Manifold proceed to write the same changes to the .map file. If any failure occurs while collecting changes in .SAVEDATA the .map file stays intact and uncorrupted.
If the collection of changes in .SAVEDATA works OK but there is a failure when writing changes to the .map file, Manifold will repeat the save operation when opening the .map file. As a result the .map file will either receive none of the changes or eventually will receive all of them.
Saving data keeps internal memory structures stable so that if the save fails, for example, due to not enough disk space, the combination of data in the partially-saved image and in memory remains coherent with no losses. The user can then take a few moments to throw off panic, restore a calm demeanor, free some disk space or find a different location for saving with more space, and repeat the save.
Fault tolerance through stable internal memory structures applies to all failed-save scenarios where Manifold is still running and has not itself been nuked by a catastrophic failure, such as a power outage with no battery backup. For example, it applies to saving to a new file with File - Save As and also to saving with File - Save to an existing file with or without use of .SAVEDATA.
Using .SAVEDATA has a performance cost, albeit a cost which is not overly large due to two factors: first, the time to save is proportional to the amount of changes and, second, much of the data between the two save stages, first into .SAVEDATA and next into the .map file, will end up being cached by the operating system. But since the cost of using .SAVEDATA is non-zero Manifold provides a connection string option to turn off the use of .SAVEDATA for a particular .map file.
In addition, when connecting to .map files as a data source we can turn off the use of .SAVEDATA by unchecking the option for Fault-tolerant save in the New Data Source dialog.
Important: By default .map file data sources use .SAVEDATA. If .SAVEDATA is not used and the system crashes during a save operation, literally between the writes, the resulting .map file will no longer load. Given that the universe is designed by Fate to ensure both the main power and battery backup together will fail exactly in the middle of a critical write, it is generally a bad idea to tempt Fate by turning off the use of .SAVEDATA. Fate always wins.
Open a .map file from a command prompt - For users who want to use a batch file to launch Manifold or Viewer with a particular .map project file, Manifold supports the usual Windows way of opening a file as a command line argument for the opening program: Running manifold.exe myproject.map will launch Manifold and open myproject.map in Manifold. Use double quotes around the file name if there are any special characters or spaces in the .map file name. For example, we would write manifold.exe "-this is my latest project.map".
Importing and Linking
User Interface Basics
File - Export Project
Example: Convert an ESRI File Geodatabase into a .map Project
Example: Convert an ESRI Personal Geodatabase into a .map Project
Where Manifold Data is Stored
Accessory Files Created
The MAPCACHE File
For Maximum Reliability