1) Batch Processing Using the WorkspaceRunner
The WorkspaceRunner transformer runs a different workspace from within another workspace. In conjunction with the Directory and File Pathname reader, it can be used as a device for batch processing.
The resulting output is a new schema feature output via the <Schema> output port. This new feature is also given the special attribute and value: fme_schema_handling = ‘schema_only’ which allows the feature to be recognized by a dynamic writer as a schema feature. If you wish to continue using the original input features, these are passed via the Output port
- You want to ensure your dynamic writer is receiving a valid schema
- You don’t know the schema of incoming data and want to make sure it meets certain standards before being used in a dynamic writer
- You want to modify the schema before it reaches the writer
- When used in a dynamic workspace, schema features should be output from the SchemaScanner first using the Output Schema Features Before Data parameter
- Most schemas contain attributes you don’t need in your final output dataset, so be sure to make use of the Ignore Attributes Containing parameter.
The Clipper takes a number of clip boundaries (Clipper input port) and a number of features to be clipped . Then the output is split into four output types (previously three output types in FME 2021.2 or prior):
- Inside: All Candidate/Clippee features that are entirely inside a Clipper feature
- Outside: All Candidate/Clippee features that are entirely outside all Clipper features
- Remnants: New for 2022.0, outputs features that are zero-width or nearly zero-width, that have no area, such as a zero-width spike.
- <Rejected>: Controlled by the “Invalid Parts Handling” parameter. This parameter controls the behavior of rejected features such as aggregate and multi geometries that contain features that are invalid for clipping (such as null geometries). These geometries without a spatial location can neither be classified as inside nor outside. Use this parameter to decide if you only want to reject the invalid parts or reject the entire feature.
A Python script can be added to a workspace that will run when the workspace completes. For example, you may want to copy files somewhere when a workspace is finished or open the output dataset with an application that is not Data Inspector.
A shutdown script runs after all of the transformers, readers, writers, and the log file are closed in a successful translation. In the case of a failed translation, the script will still execute but may fail as only fme.status and fme.failureMessage are guaranteed to contain valid values.
6) Creating a Unique Identifier | CRCCalculator
The CRCCalculator creates a unique ID for a feature based on its geometry and/or attributes. CRC stands for Cyclic Redundancy Check. This is particularly useful if the dataset has a "natural key." Natural keys are created from a list of attribute names and their values for a particular data record or feature. These can be cumbersome to work within an FME workflow (i.e. FeatureMerger) so combining the natural key into a single CRC unique ID can simplify the workspace.
Another use case is to use the CRC value for change detection - to make sure it hasn't been changed, corrupted, deleted, etc. It can also be used as a form of ChangeDetection (instead of using the ChangeDetector to compare geometry, use the Matcher to compare CRC values).
7)Remove Intersections From Line Networks
Occasionally it may be necessary to remove the intersections from a line network, possibly to optimize symbology for display. The LineJoiner transformer works well when the lines to be joined all share a unique identifier, but this is not always the case. In the case of a T intersection where all three lines share the same id, the LineJoiner will choose which two lines to join randomly.
FME incorporates the ability to locate and fix self-intersections using the GeometryValidator transformer.
In this example, we will look at identifying and fixing self-intersections in a dataset containing building outlines.
11) Python Scripted Parameters in FME
Spikes (or Outliers) in spatial data occur when a vertex in a feature has an x, y, or z value that is so incorrect as to result in a spike-like appearance.
Spikes are a fairly specific type of defect, and there is a specific FME transformer designed to handle them: the SpikeRemover.
12.b) Data QA: Identifying Bad Topology in Linear Networks
A linear network should consist of lines that meet at a point, without crossing. However, this is not always the case. There are various problems that can occur.
A misaligned point occurs where two lines are meant to meet at a known point, but one line does not properly connect:
An overshoot occurs where two lines are meant to meet at a known point, but one line extends beyond that intersection:
An undershoot occurs where two lines are meant to meet at a known point, but one line fails to reach that intersection:
A missing node occurs where two lines are meant to meet at a known point, but only one of those lines contains the intersection node:
In this case, the red (horizontal) line has an end node, but the green (vertical) line has no node or vertex at the same location.
In general, these issues are very small and usually invisible to the eye - else they would be easily detected without the need for special data validation techniques. There is no specific transformer for these scenarios, but we can use a combination of general geometry-handling transformers to do the job.
13) Using the Parameter Manager
In FME 2021.0, we have introduced a new way to manage and work with user parameters in FME Desktop. The new Parameter Manager allows you to see all of your parameters in a single window, as well as create new ones, reorder, as well as group them
In FME, when processing data from a non-streaming data source (e.g. database or file), because the data is at rest and finite, you can load the data into memory and then sort and group the data before analyzing it. When reading a data stream into FME, you cannot do this since the data is infinite, and it will never finish loading. This is where windowing comes in.
Windowing means that you take the data stream and, using time, break the stream up into groups. Once in groups, data is ready for analysis.
https://www.youtube.com/watch?v=70d9n4YQNWE
https://www.youtube.com/watch?v=fecdLImUl5U
19) Working with Geodatabase Domains: Writing A Coded Domain
A domain is a set of rules that define permitted values for an attribute. They are used to constrain data values in order to ensure data integrity. A domain is defined in a geodatabase as a unique entity and can be applied to any attribute in any feature class within the geodatabase that contains the domain definition.
There are two types of geodatabase domains (coded domains and range domains) and both are supported by FME. A coded domain is essentially a list of multiple valid values while a range domain is a single permitted range of numeric values.
20) A Guide to Better Geometry-QA/QC
Spatial applications use geometric data to represent spatial features from the real world. It’s common for these geometries to render differently between formats. These scenarios are frequently seen when translating GIS to CAD, and the reciprocal. Examples of geometric problems include overshoots, dangles, gaps, and duplicates. The act of correcting invalid geometry is often referred to as ‘cleaning’ data. These processes are also used to develop Quality Assurance and Control (QA & QC) procedures.
21) Using the Geometry Parameter in FME Server
The Geometry Parameter in FME Server allows users to draw an area of interest on a map and then have that area inputted into a workspace. This functionality is first set up in FME Workbench with a combination of the Geometry published parameter and the GeometryReplacer transformer, then in FME Server the user can access the map anywhere you can see published parameters (Run Workspace, FME Apps, etc.)
22) Data QA: Identifying Features Closer than a Minimum Distance
Features that are going to be used for cartographic production at a specific scale need to be a set distance apart in order to not overlap in the final map output. Testing for features closer than a set distance apart is therefore a useful ability in that scenario.
For example, here a user is mapping water fountains, but finds that - at the required scale - the chosen symbol overlaps with a building footprint:
The user will need to therefore identify these features, quantify the problem, and then fix the issue by displacing the features to the required distance.
Since LiDAR point cloud data contains information about any surface that returns its laser, we can use it to model terrain as well as building shapes. This is exactly what we set out to do in this demo, where a LAS point cloud is used to (1) build a triangular irregular network (TIN) terrain model and (2) extrude building footprints to their actual 3D height. Each of these two steps becomes a layer in our output: a 3D model written to PDF that captures the urban built environment in a Vancouver neighborhood.
There are two tricks to this workflow: (1) extracting and replacing the building footprint geometry (vector data), and (2) clipping the point cloud using the building footprints. In short, we will isolate the point clouds for each individual building footprint, and then extrude these footprints into 3D solids.
CAD Building Outlines:
LAS Point Cloud:
27) Tutorial: Getting Started with FME Form(Desktop)
https://community.safe.com/s/article/getting-started-with-fme-desktop-read-data
FME Form is designed to overcome your data integration challenges. Whether you need to extract, transform, load, integrate, validate, or share data, FME’s visual interface automates your workflows and eliminates coding. FME Form is a part of the FME Platform, including FME Flow (formerly FME Server) for data automation.
28) Tutorial: Getting Started with FME Flow(Server)
FME Flow automates the enterprise. By authoring workflows in FME Form and automating them with FME Flow, integrations can be run in real-time, in response to events, on time intervals, or according to business logic. No-code web applications can be built to deliver notifications and provide self-serve data submission and validation to anyone. FME Flow is a part of the FME Platform, including FME Form (formerly FME Desktop) for workflow/workspace authoring.
29) Tutorial: Getting Started with List Attributes
List attributes are FME's way of allowing multiple values per attribute. For example, rather than creating a text field named FRUITS that stores the value “Apple, Orange, Pear”, a user can create a list, which is more structured and can be broken down into constituent parts for processing:
Fruits{0}: Apple
Fruits{1}: Orange
Fruits{2}: Pear In FME, lists are denoted using curly brackets after the list name (e.g. this list is called "Fruits{}"), and a number inside the curly brackets represents the element's index inside the list (e.g. Orange is element 1 in the list). With this structure, lists can be reorganized, exploded into individual parts, analyzed statistically, and more.
30) Tutorial: Getting Started with Oracle
This tutorial will provide you with all the information necessary to begin working with Oracle/Oracle Spatial in FME. Databases are useful for long-term data storage, rather than short-term data transfers, so the key operations are getting data into and out of that store. This tutorial will focus on how to move data into and out of Oracle databases, and introduce new users to key concepts, as well as the Oracle Reader and Writer parameters.
FME provides non-spatial data support through the Oracle reader/writer, and spatial data support through the Oracle Spatial reader/writer. This tutorial series will focus on the use of the Oracle Spatial database.
The three key operations that occur within a spatial database, including Oracle Spatial are:
- Data Imports
- Data Updates
- Data Distribution
Application Programming Interfaces (APIs) are used as a communication medium between clients and servers. They allow applications to talk to one another, transmitting data and services back and forth for a variety of purposes. By leveraging an API, you are taking advantage of existing data, software, functions or code-shared by the developer. When you do, you can benefit in diverse ways which include:
- Streamlining your workflow
- Direct access to an outside database
- Added functionality to your own application
- Application integration for more fluid information delivery
- + many more!
A coordinate system is a mathematical model of the conversion between the data coordinates and a specific location on the Earth.
In order to be truly georeferenced, data needs to have a coordinate system associated to it. FME will gladly handle data without any coordinate system, but without a defined coordinate system, it will not be possible to accurately combine the dataset with those in other coordinate systems.
Coordinate system definitions are specified by a set of parameters that define this mathematical model, including the earth model (ellipsoid or datum), the units used to measure the coordinates, the projection type, and any parameters specific to the projection type.
No comments:
Post a Comment