For Esri Users

This topic explains how Manifold is different from GIS packages that you may already know, such as Esri ArcGIS® Pro ("Pro") or QGIS ("Q").

Nomenclature

Manifold nomenclature is very similar to Esri nomenclature, often with simpler terms.  What Esri refers to as feature classes, Manifold simply calls drawings or vector layers.  All raster layers are called images or raster layers.  Within vector layers, what Esri calls features Manifold calls objects.  Instead of Esri's polygons, polylines, and points as basic vector objects, Manifold uses the terms areas, lines, and points.  

Completely New

Manifold Release 9 is a completely new system, millions of lines of code written from the ground up in the last few years. It is not based on older products or libraries as are Pro or Q.   That enables Manifold to include features such as parallelism throughout that are not possible in products that are based on legacy code.  

 

Manifold was designed to handle the rapidly increasing size and sophistication of GIS data and GIS projects.   Many years ago a GIS project that was tens of megabytes in size was considered big, but today projects routinely can be tens or hundreds of gigabytes in size.   Manifold's designers knew that the only way to handle larger data was by doing parallel processing all of the time, using all CPU and GPU cores in a system and all threads the operating system can launch, from data access all the way through processing and analysis to visualization.  

 

Manifold designers also knew that data storage and management are central to modern GIS: to get fast performance with bigger data the GIS itself has to be built on a fast, parallel, database.  

Manifold is a Database

GIS packages like Pro or Q leave data in the originating file or server.  When adding a layer from a shapefile to a project in Pro, for example, the data remains stored in the originating shapefile.  They link data into the GIS package and do not import the data into the GIS package's own storage.

 

That is a consequence of how those products have evolved: they started as viewers of GIS data stored in file formats or servers outside of the GIS, and to this day they have no internal database that can store GIS data.  The data is always linked in from some external storage format, where it is subject to all the limitations and quirks of that storage format.  That also makes it difficult to share projects which include data from multiple data sources.  If a Pro project includes data from several different file geodatabases, various shapefiles and various GeoTIFF files, it is a hassle to package all the data sources along with the package to share with a colleague in a different location.

 

In contrast, Manifold itself is a database system that has been engineered for the specific requirements of GIS work with spatial data.  It not only can handle over a thousand GB of data on the desktop, it does so with fully parallel, fully multithreaded architecture designed to handle the exceptionally "fat" operations that routinely occur in GIS work.  Besides providing full-featured, advanced, parallel, spatial SQL, Manifold's internal database system includes advanced features like full support for Unicode and collations essential to correctly handle international data.

 

Building out a GIS on the foundation of an internal database system provides many advantages:

Imports as well as Links

That gives Manifold users the option of importing data for storage within the Manifold project, in addition to linking data from external sources as Esri does.   Manifold projects are just a single .map file that provides very fast, parallel storage of potentially thousands of items.   A project can have hundreds of layers that are rasters or vectors with hundreds of tables and all that can be stored in a single .map that is easy to back up or to share with a colleague.  .map files are so fast that for most uses they are faster than storing data in even a fast DBMS like PostgreSQL/PostGIS with none of the hassles of operating a server.   Map files also open instantly:  A 100 GB .map file with hundreds of layers will open in 1/2 second.

 

Most Esri users very rapidly convert to importing their data into the project and saving all data as .map files.  If they ever need the data in some interchange format, like shapefiles or file gdb, they just export it to that format as needed.

Not Limited by Formats

Having its own, standard, rich database allows Manifold to map all data into that standard model, either when data is imported into the project or mapped on the fly if data is linked into the project. That enables user to work with a full range of capabilities without having to keep in mind the limitations of the originating format.

Greater Flexibility

Keeping data in a shapefile means that only one column can be a geometry column and there can be only one drawing associated with that.  Keeping data in a Manifold project means that a table can have as many geometry columns as you like, with multiple different drawings using the same column or different columns in that table, to provide multiple different views of data. Geometry columns can be different types as well, such as native Manifold geometry or interchange types such as WKB or WKT geometry.   Drawings can also be created on the fly based on query results.  

Greater Access

Because everything in Manifold runs on the internal database, basically everything within Manifold is stored in and exposed by a table.   Styles, for example, are simply human-readable JSON strings stored in properties tables.  That makes it easy to transform data about the contents, structure, and presentation of what is in the project either interactively or procedurally, using SQL or scripts, which greatly enhances automation.

Always On SQL

Packages like Pro or Q that do not have an internal database depend on whatever SQL is provided by data sources.  If the data source, such as file GDB or a shapefile,  does not provide an SQL then you have no SQL available.  If you connect to two different data sources that do provide SQLs, like SQL Server and Oracle, you must use a different SQL with each data source, and cannot write queries that combine the two data sources.    Manifold's internal database engine provides an SQL engine that is always on and available, no matter what the data source.  SQL is always there for all imported data, including rasters, and is also available for linked data sources.  That enables writing SQL queries that always work, and which can combine data from tables from many different data sources at the same time.   Do JOINs between tables stored in SQL Server, Oracle, and Postgres if you like.  

Nested Projects

A Manifold project can link into itself a different Manifold project.  The assets in that other, linked project can be accessed with zero performance loss.  That makes it possible to keep frequently used assets within a .map file and to nest projects within other projects, as many levels deep as desired.

Fully Parallel Operation

Manifold runs fully CPU parallel and GPU parallel.  When using point and click dialogs or transform tools there is no need to manually set any parallel processing factor, as parallel processing is always on by default.

 

The system automatically launches multiple threads to use all available cores for everything from data access to processing to rendering.  The parallelization optimizers will automatically balance CPU and GPU parallelism, dispatching tasks to massively parallel GPU processing when optimizers judge that would be faster.  Manifold will automatically use multiple CPUs in a multi-CPU motherboard, and will also automatically use multiple GPUs, including GPUs from different technology generations, if multiple GPUs have been installed.

 

In contrast, only approximately 80 of Esri's hundreds of geoprocessing tools have been parallelized, and only partially so, with the main Pro application remaining single-threaded.   Pro does not significantly use GPU parallelization.  Q is basically a single threaded system, with no built in parallelization for either CPU or GPU.  

 

Manifold's internal spatial database engine runs fully parallel and has been optimized for a mix of manycore CPU parallelism as well as massively parallel GPU parallelism.  When running parallel against linked storage from non-parallel formats or data sources, Manifold uses its internal parallel database as a parallel data cache, to enable loading many more threads for maximum multicore utilization.

Bigger Data

Manifold can handle hundreds of GB on the desktop, either vector or raster data or a mix of vector and raster.   The system can pop open projects with hundreds of layers and hundreds of GB in 1/2 second.   

 

That changes workflow as compared to Pro.  Manifold users often will work with large data even when only a small part of it is of interest, because there is no need to trim data to a smaller size to get tolerable performance.   There is no need to worry about having projects with many components or layers to them, reducing the number of layers so that projects do not take too long to open.   Saves are also very fast, so there is no hesitation about doing frequent saves in case of hardware failure.

Mixed Object Types in the Same Layer

Esri products like Pro require a feature layer contains only polygons, only lines, or only points.   Manifold vector layers can contain a mix of polygons, lines, and points, and those objects can be created from a mix of either straight segments or curvilinear segments.  Manifold will automatically, on the fly, split mixed object layers into single object type layers when writing to Esri formats such as file GDB or shapefiles.  

Quiet Cockpit

ArcGIS Pro has a visually appealing, but graphically busy, ribbon layout that uses a lot of screen real estate for large command icons and user interface elements that are not being used.  Q has very large numbers of buttons in a very busy layout.  

 

Manifold tries to reduce the amount of screen real estate consumed by buttons and commands so that more screen real estate is available for the GIS content of windows.  A Microsoft ribbon is not used because of the large amount of screen real estate it consumes.   Toolbars adapt so that in general only command icons are displayed that can be used.  

Non-Modal Panes

Key operations in Manifold are handled by non-modal panes in preference to modal dialogs.  Pro and Q, in comparison, use modal dialogs.   A "modal" dialog is one that grabs the user interface when launched.  It does not allow other GUI functionality until the user exists that dialog.  A "non-modal" pane does not grab the user interface, and it allows the user to do other things even while it is active.   

 

An example of non-modal panes is Manifold's Transform pane, which is a rough equivalent to Esri's geoprocessing toolboxes.  Clicking the Transform pane, picking a template and setting it up does not lock the interface to that transform tool: it will automatically adapt on the fly to whatever window has the focus.   If we click different map windows or different attribute table windows the Transform pane will simply adapt on the fly to whatever window has the focus, remembering and restoring context for each window when that window regains the focus.  

 

That makes it possible to start setting up a transform with the focus on one window, and then panning and zooming within that window or making measurements in it with a different tool to help us decide how we want to set up the transform, or even switching to a different window to do completely different work in that window while we decide how to proceed with the transform in the first window.

Fully Unicode

Manifold is fully Unicode, with full support for international collations, throughout.   Text types are nvarchar (full Unicode) by default.   Most users always use nvarchar text types so their work and the data they produce is fully compatible with internationalization.  

More Programming Languages

Manifold includes a built-in, always available, highly optimized, fully parallel SQL with many hundreds of SQL functions.  SQL is so highly optimized in Manifold that it is difficult even for experts writing in C++ to match the performance of queries in Manifold.   SQL is therefore usually the first choice even for analytics when working with data in tables, with no need to cobble up SQL-like functionality using Python with Pandas or other libraries.  

 

For scripting in tasks where scripting is preferred, Manifold allows use of ten different scripting languages.   In-line scripts can be called from SQL queries, allowing use of scripts where SQL constructs do not fit the task, and SQL can be called from scripts.   

 

Manifold tends to favor Microsoft style languages, for example, using C#, JScript, VB.NET, VBScript and using IronPython and PythonScript instead of cPython.  

Unified Package

Manifold Release 9 has been written from the ground up as a single, unified package.   It has not been assembled by packaging together a variety of other projects, and it does not depend on other packages such as GDAL or PROJ.   A general rule with fully parallel packages is that for the entire thing to be multithreaded all of the various libraries and components that go into the package must also be multithreaded.  Virtually none of the packages commonly used in older packages like Pro or Q are multithreaded, so they were not candidates for use within Manifold.  

 

A side effect of writing all portions of Manifold as fully parallel code starting from a clean slate is that more modern software techniques could be used in a more unified and more globally optimized architecture.  That also provides a more unified, orthogonal style for user interfaces, without the variation in user interfaces seen in packages assembled from many pieces.  

 

An additional benefit is that not relying on hundreds of older, or third party libraries makes it much easier to maintain and to evolve Manifold.  Instead of many different components issuing updates and evolving at different rates, the system can be maintained and evolved with a single release schedule for the entire package.  Designers can also implement significant upgrades that must happen throughout the system at once, without trying to coordinate many third party schedules.   A good example is the recent addition of parallel server functionality, where Manifold as a highly parallel client can connect using many-threaded connections to Manifold as a highly parallel server to enable multiuser use of data residing on the server.

High Reliability

The database internal to Manifold has been designed to work with very high reliability and to be resistant to data corruption caused by hardware and network failures.   Crashes are so rare that you can expect to work every day, for eight hours a day, for over ten years without encountering a crash.

 

 

Next:  Install and Launch