Transform Pane Guide

Panes are a key part of Manifold's user interface.  Unlike dialogs, which grab the mouse and keyboard until exited, panes allow us to work in other windows while the pane operates on whatever is the active window.   Panes report important information and they allow us to issue commands and make changes to the active window.

 

This topic provides a quick introduction to the Transform pane.  See the Transform Pane topic for more details and step by step examples.  

 

The Transform pane allows manipulation and editing of values in fields in tables, drawings or images, either by using pre-built transform templates or by using SQL expressions.  For example, by manipulating the value of geometry fields in tables used to drive drawings, the Transform pane manipulates objects in drawings, for example, reshaping them, clipping them or transforming them, like transforming area polygons into border lines for those polygons.  Likewise, by manipulating the value of tile fields in tables used to drive images, the Transform pane also manipulates pixel values in images.  The Transform pane works when any component that is based on a table has the focus, such as tables, drawings, labels or images.

 

The Transform pane provides functions like those found in the Geoprocessing toolboxes of products like Esri ArcGIS® Pro .

 

The Transform pane often groups many similar operations within a single template, allowing choice of the specific operation desired after launching the template.  For example, the Reshape template for geometry fields includes operation options such as rotate, scale, shift and similar.    Grouping operations that are similar or have similar parameters allows rapid application and reapplication of similar Reshape functions by changing fewer options or values, without having to repeatedly choose and re-choose parameters that have not changed.

Index Required

To use the Transform pane the component's table must have at least one identity index in the table, such as the usual btree index on the mfd_id identity field.  Indexes are added automatically when importing or linking data from GIS formats so normally we do not have to think about this requirement in GIS work.   Occasionally when importing data from tables in non-GIS formats we might have to manually add an index.   This is easy to do with one click in the Schema dialog. See the Add an Index to a Table topic.

 

Transforms on tile fields in images that depend on tile placement also require that the table have a spatial index on the picked tile field.  A spatial index is automatically added when importing or linking images.  However, if we create an image using programming or some other means that does not create an index, we can add a spatial index on the tiles, normally an rtree index, with a few clicks in the Schema dialog.

Operates on the Active Window and Specified Layer

The pane automatically configures for whichever window has the focus when the pane is open.  Click a different window to move the focus to that window and the pane automatically adjusts.  

 

When working with an active window like a map that has more than one layer, choose the layer desired from the pull down list of components in the topmost box of the pane.  The layer picked in the pane can be different from whatever is the active layer in the map.

 

The Transform pane opens to a top level list of pre-built commands, called templates, that may be used with the data type of the specified source field.  

 

Templates will automatically adapt to the data type of the specified source field.  The Filter box can be used to narrow down a long list of templates, or to find a template that does a particular operation.  The Filter box searches within templates for the names of operations within templates, as well as the top level names of templates.

 

A complete list of templates, organized by the data type of the field with which they work, appears in the Transform Reference topic.  

Two Step Workflow

The Transform pane uses two step workflow:  

 

Step 1: Pick a layer and a source field.  The pane then shows a list of templates that can be used with the data type of that source field.  

 

Step 2: Pick a template.  The template opens with options for operations and parameters that work with the data type of the source field.  Without returning to Step 1, we can change parameters, choose a different layer in the map with fields of the same data type, or choose a different template that works on fields of the same data type.

 

Two step workflow ends up being faster than single step, because much workflow is iterative, doing the same thing to a different field, or making a slight change and applying the same operation.   After the initial step we can rapidly repeat the second step over and over, with easier recycling of settings.  For example, once we choose a numeric field in the first step we can very rapidly apply operation after operation to many different numeric fields in different layers, with most settings remaining the same, persisting into the next operation.  

Pick a Template

Double-click a template within the list to launch that template using the specified component and source field.  We can also repeat-click a template, or focus on that template and press Enter, or focus on that template and press the Edit Parameters button.  That launches the template within the pane.

 

 

Templates often provide many operations that can be picked from a pull-down list, like the shift operation in action in the illustration above.    The pane will automatically configure to show options and parameter boxes that work with the chosen operation.  Templates will automatically adapt to the data type of the source field.    

 

Using the Transform pane:

 

  1. Choose the component to use.  Components that are layers in the active window will be available.
  2. Choose a field within that component.   The pane will populate with templates that can be used with fields of that data type.
  3. Double-click the desired template.
  4. The pane will display parameters for whatever operation in that template is chosen.  Some templates have many operations.
  5. Choose the desired operation with parameters as desired.
  6. Choose the Result destination desired if a mode other than same field is desired.
  7. Press the Transform button.

 

 

When the Transform pane has been used during a Manifold session, templates that have been used will appear, with options as used, in the Recent list at the top of the list of templates.  In addition, we can pin frequently used templates to the top of the list.

 

Component

The name of the component the template is using.  Automatically set to the active window.  When the active window has more than one layer, we can choose the component desired from a pull down list in the box.

Field

The field on which the template operates, also called the source field.   Choose any field from the active component.  When a drawing is the active component, the pane can operate using any field within the drawing's table.  Likewise, when a drawing's table is the active component, the pane can operate using the geometry field in that table, to transform object geometry displayed in the drawing.

Filter

Reduce long lists of templates, and find which template hosts a desired operation by entering text to filter by.  The pane will only show templates that have in their names, or in the names of options they provide, the text entered into the filter box.  

Templates

Templates provide commands that can be done with the chosen type of field.  Templates often include many operations or options within them.   Operations that do similar things or use similar parameters are grouped together within a template, to allow parameter and option choices to persist as much as possible within repetitive or iterative workflow using similar operations. Double-click a template to launch it.

Recent

The ten most recently used templates together with their secondary operational choices appear as shortcuts, such as the contains option within the Search template for text.   Double-click a recently used item to launch it.
  
The list of recently used templates persists within the same Manifold session even if we close the project and open a new project.  However, the list of recently used templates will disappear if we close Manifold and then launch a new Manifold session.

Pinned

Recently used shortcuts appear with a white pushpin.  Pinned shortcuts appear with a black pushpin at the top of the list.  Click a white pushpin to turn it black, and to pin that shortcut to the top of the template list. To unpin a shortcut, click the black pushpin to turn it white.   If the cursor is on a recent or pinned shortcut, the spacebar toggles the pushpin color.
  
Pinned shortcuts will persist at the top of the templates list for fields of that data type, even if we close and restart Manifold in a new session.   

(tooltips)

Hover the mouse cursor over a template to get a tooltip with a brief description of the template, usually providing some examples of operations it can do.

Cursor

Click a template to move the row cursor onto that template.  Move the cursor using the up and down arrow keys.  When we move the cursor onto a template, that puts the focus on that template.

Edit Parameters

Click to launch the template indicated by the cursor.

 

Picking a template launches it for use, with options and controls appearing as required for the data type of the chosen field and the operation desired.  The name of the template in use will appear at the top of the pane, with controls and parameter boxes appearing in the pane as required.

 

Typical Controls within a Template

Up one level.  Return to the main template list to allow choosing the component or field.  Use this button to choose a field that is a different data type from the current field.

<component name>

Gives the name of the component or layer that the template is using.   Change to any other layer in the same component that also has a field of the current data type.

Field

The subject/source field on which the template operates.   Choose any other field of the same data type.

Operation

Choose the operation option to use within this template.   Some templates can have very many operations.  

<parameter boxes>

Different templates, and different operations within templates, will automatically display the parameter boxes required for the operation.

Unit

Units of measure that can be used for the operation, based on the projection in use by the component.   Projections such as Latitude / Longitude that use angular units of measure will offer a choice of Arc Minute, Arc Second, Degree, or Radian, automatically converting the chosen unit into whatever unit is used by the projection (usually degrees).   Projections such as Pseudo Mercator that use linear units of measure will offer a very long list of all linear units of measurement known to Manifold, with Meter appearing by default.  The template will automatically convert the chosen unit into whatever linear unit (feet, survey feet, etc.) is used by the projection.   This can make it very easy to apply operations like shift: if we want to shift some objects three miles we can choose Mile as the unit of measure and the shift operation using 3 in the  X and Y boxes will shift objects by three miles even if the drawing's projection uses meters or feet for the coordinate system.

Field, Value, or Expression

Value boxes, such as the X and Y boxes for the shift operation, allow us to choose values from a different field, literal values we enter, or values generated by an  expression that is a snipped of SQL.

Collation  

A Collation option appears with some templates applied to text fields, for example, the Replace template that is used for text search and replace operations.   It is primarily used to specify case or no case searches.  Press the collation picker button to choose a different collation, for example, neutral for case sensitive search and replace.   Various collations offer different languages and different settings, such as reckoning accents on characters or not.

Result

Specify the destination for the result of the transform.

 

  • Same Field - Appears as the default when the template creates the same number of records as the source component.  Place the result of the transform into the same field that was used as the source of the transform.  This overwrites prior values in that field with new values that are the result of the transform.

 

  • Existing Field - Choose an existing field in the same table that is of a compatible data type. A list of existing fields will appear as choices when the template creates the same number of records as the source component.  Place the result of the transform into the specified field.  This overwrites prior values in that field with new values that are the result of the transform.  Tile fields are considered to be compatible when they have the same tile size and the same number of channels.

 

  • New Field - Create a new field of the specified type in the table, and populate it with the results of the transform, optionally also creating a new drawing (for geometry fields) or a new image (for tile fields).    Appears as a choice when the template creates the same number of records as the source component.   

 

  • New Table - Create a new table using the name specified in the New table box, and create a new field in that table into which the result of the transform will be placed. Also create a new drawing (for geometry fields) or a new image (for tile fields).  Appears as the only choice when a transform creates a different number of records as the source component.  

 

Automatic Table Names - When the destination for the result is a new table, entering the name of the new, destination drawing or image in the Transform pane automatically will generate a corresponding name for the new table. The name of the new table will be adjusted as required by the database within which it is created. For example, if the destination database does not allow names that contain spaces, spaces will be replaced by underscore characters.

 

Result Type

The data type to use for the field that will hold the result.   Appears when results are sent to a new field or a new table and there is an option to choose the data type.   For example, numeric results will usually allow a choice of any numeric data type, such as various integer or floating point numeric data types.   Geometry results will usually allow a choice of Manifold's native geom type, geommfd type, or geomWKB type.  For a tile field, the data type is that of the channel values.

New drawing

Appears when geometry results are sent to a new field in the same table or sent to a new table.  Automatically creates a new drawing of the specified name that uses the resulting geometry field.

New image

Appears when tile results are sent to a new field in the same table or sent to a new table.  Automatically creates a new image of the specified name that uses the resulting tile field.

New table

Appears when geometry or tile results are sent to a new table.  The name to use for the new table.

Resources

A choice of CPU and GPU parallelization resources the system is allowed to use:

 

  • all CPU cores - Allow parallelization up to using all CPU cores (threads) with no use of parallel GPU allowed.

 

  • all CPU cores, all GPU cores -  Allow parallelization up to using all CPU cores (threads) and parallel use of all GPU cores.

 

  • one CPU core - Allow use of only one CPU core (thread) with no use of parallel GPU allowed.

 

  • one CPU core, all GPU cores - Allow use of only one CPU core (thread) and parallel use of all GPU cores.

 

CPU "cores" are used in the Windows meaning of the word core, meaning hyperthread for CPUs that support hyperthreading when hyperthreading is turned on in the BIOS.   Since most modern CPUs and systems support hyperthreading, when Windows reports the number of cores it is really reporting the number of threads.

 

GPU cores are either used fully parallel for all cores or GPU is not used at all.

 

The Resources setting puts limits on what the system is allowed to use.  It does not force parallelization if that would result in slower operation.   Manifold will optimize on the fly to decide whether a given operation at that moment of machine loading and data in use will go faster if parallelized, and, if so, to what degree parallelization would be best.  

 

For example, on a 48 core CPU Manifold will not launch 96 parallel threads to add one plus one to get two, since that would be slower than simply doing the addition in one thread.  The optimizer might decide it is better to run only CPU parallel, or only GPU parallel, or a mixture of both simultaneously.  

Transform selection only

Check to apply the transform only to selected records (objects).    

 

Operations such as clip will in addition have a "selection only" option box, such as Clip with selection only, that allow using only selected objects in the clipping drawing.  

Transform

Apply the transform template.

Preview

Show a preview in blue preview color of what the transform operation will do, when possible.   A preview is just a temporary view and does not change anything.

 

Press the Preview button to launch a preview, or to update a preview after changing any parameters or controls in the pane.  A preview will stay in view until we hide it, or until a layer used to compute the preview is removed or refreshed.  We can add layers, pan and zoom, Alt-click objects to view attributes, and edit layers without losing the preview.

 

Closing a preview:  In map windows, right-click the blue preview caption bar at the top of the window and choose Hide Preview.  In table windows, right-click the blue preview column head and choose Hide Preview.

Edit Query

Pop open a Command Window loaded with an SQL query that accomplishes what this template does using the current settings.

Result Destinations

When presenting available Result destination options, the Transform pane automatically adapts to any restrictions, such as read-only limitations, on possible destinations.   For example, attempting to perform a transform on a computed field or on the built-in mfd_id identity field disables the Same Field result option. Attempting to perform a transform on a table that cannot alter its schema (common for result tables of queries, or for read-only tables in remote databases) disables the New Field result option.

 

The default name for a new field is set to a blank string to force entering it before running the transform. After the transform is run, the field name is left unaltered so that repeating the transform without any changes quickly fails with a name already in use error and does nothing, alerting the user to alter the name or to choose a different Result destination. The default names for new components likewise are set to blank strings to force entering them, and after the transform is run the component names are left unaltered so that repeating the transform without any changes quickly fails, alerting the user to change names or to choose a different destination.

Indexes in New Result Tables

Sending transform results into a new table automatically creates an autogenerated identity field and a unique index on that field within the new table that is created.   When such new tables are created in a Manifold .map project or data source, the autogenerated identity field and index will be the usual mfd_id and mfd_id_x field and index.   When such new tables are created in some external data source, such as a DBMS, whatever facilities available within that data source will be used to create an autogenerated identity field and unique index.

 

If the target database does not support autogenerated fields, the transform will fail with an error message.   Manifold supports autogenerated fields in the vast majority of databases to which Manifold can write.   The only notable exceptions are databases that use the Microsoft Jet engine, namely Access and Excel files.

 

However, for Access and Excel, in any event it is a much better idea to import data into the Manifold .map project, perform the transform within the .map project, and then export the result back into an Access or Excel file.   That is because Jet is fragile: performing the transform within the Manifold .map instead of directly on the data stored in place within the Access or Excel file limits the amount of time the Jet database is being written to, which helps ensure its integrity.   That is especially important when the Jet database is being opened from a network share.  

 

A limitation on creating new tables to accept the results of a transform occurs with tile results:   If the target database does not support spatial indexes that are required for the resulting field, the transform will fail with an error message.  Since at the present time only .map project files support creating spatial indexes on tile fields, this requirement limits the creation of new tables by the transform pane for the results of transforms to new tables within a .map project file.  This limitation may be removed in future builds.   It does not apply to new tables created to host geometry fields added by a transform, since creating new tables with a new geometry field can be done both within .map projects and also within most external databases.

Fields Copied into New Result Tables

When sending transform results into a new table, all fields from the existing source table are copied over into the new table along with the result field, except for geometry and tile fields.

 

When the result field of a transform is a geometry data type, the new table excludes all other original geometry fields and creates a spatial index only on the new geometry field. When the resulting field is a tile data type, the new table excludes all other original tile fields and creates a spatial index on the new tile field.

 

The copy/exclude logic arises from the most likely circumstances of workflow.   If in addition to the result of the transform we want to retain all prior geometry or tile fields in the table, which could be a large amount of data, the most effective way to do that is simply to save the result to a new field in the same table.  There is no need to create a new table.

 

In contrast, if we want to place the result in a new table, it is less likely we care about other geometry or tile fields in the prior table.  For example, it is often the case we may save intermediate transform results as new geometry or tile fields in the same table, and then when we save to a new table we want to start a new chain of operations and results starting with the field saved to the new table, with any prior geometry or tile fields left behind in the old, source table.

 

If we want to retain all prior geometry or tile fields in a new table, the straightforward way to do that is to first copy and paste the original table to make a copy, and to then apply the transform to that copy, which allows us to save the result as a new field into that table.  

Example

We begin with a drawing, seen below, that shows building footprints in Monaco as polygonal areas.  

 

 

One of the buildings has been selected by Ctrl-clicking it.

 

 

With the focus on the opened map window, choose the Transform pane.   The buildings drawing is automatically loaded as the subject component.  

 

Choose the Geom field in the buildings drawing, and then double-click the Buffer template.

 

 

The Transform pane loads the Buffer template.   Enter 30 in the Distance box.   For Unit, choose Meter from the long list of units available.

 

Press Preview to see what that will do.

 

 

The preview appears in the map window using shades of blue preview color, as if it were a layer above all the other layers in the map.   The preview by default is 100% opaque.  To see what is underneath the preview, right-click onto the Buffer preview title bar and choose 50%.

 

 

That changes the opacity of the preview layer to 50%, allowing us to see how the buffer objects have been created for the objects in the buildings layer.   Each buffer object extends 30 meters beyond the boundaries of the corresponding object in the buildings layer.

 

 

We can restrict the operation of the Buffer template to only selected objects.   Check the Transform selection only box, and again press the Preview button to see what that does.

 

 

That creates a buffer object only for the selected object in the buildings layer.   The preview is still set to 50% opacity.

 

So far, we are only looking at a preview of what the Buffer template will do.   To create the buffer object, we can press the Transform button.  

 

However, the template is currently set up to put the Result into the (same field), which would alter the geometry of the selected object to transform it into the buffer object.  If we do not want to do that, but instead we want to create a new buffer object in a new layer, we can change the setting in the Result box.

 

 

Click into the Result box and change the setting to New Table.   That will create a new table and put the geometry for the buffer object into that new table.

 

 

The (new table) setting also opens up a New drawing box, to automatically create a new drawing to visualize the geometry in the new table.  We enter the name buffers for the new drawing.  As we enter the name for the drawing, the pane will automatically fill in an analogous name for the table.   We can change that if we like.  We could name the new drawing and table however we want, but it makes sense to use a consistent naming style so it is clear in the Project pane which drawings use which tables.

 

Press Transform.   That creates a new table and a new drawing in the Project pane.   Drag and drop the new Buffers drawing into the map window.

 

 

The new drawing appears in the map window.   In the illustration above we have used the Layers pane to set the opacity of the buffers layer to 50% and we have used the Style pane to change the fill color for areas in the buffers drawing to bright green.

 

After running the template, the Transform pane is still set up the way we left it.

 

 

That makes it convenient to do repeated transforms that are variations of each other.  For example, we could change the Distance to 40 and then change the name of the drawing to buffers40 and the table to buffers40 Table and then press Transform again.  That would create another drawing and table pair, but this time containing a buffer of 40 meters around the selected object.   In a matter of moments we could create a series of different buffers.

 

Or, we could uncheck the Transform selection only box and change the name of the destination drawing and table to all buffers, and then press Transform to create buffers around all objects.    We could then use the Clip transform template between the buffers and all buffers layers to create a layer that has buffers around all objects, but which are clipped so that no buffer zone of any other object comes within 30 meters of the selected object.

 

 

 

Next:  Join Dialog