FME Introduction - Session 6 | Jugaad-'We Are Solution'

Conditional Filtering

Transformers that filter, don’t transform data content.

What is Filtering?

Filtering is the technique of subdividing data as it flows through a workspace.

It’s the case where there are multiple output connections from a transformer, each of which carries data to be processed differently.

Here (for example) incoming stream A is filtered into two new streams, B and C:

A filtering transformer may be any transformer with multiple output ports, such as the GeometryFilter or Sampler transformers, the latter of which creates a sample selection of data and separates it out through Sampled and NotSampled output ports:

However, these are mostly in-built, fixed tests. Conditional filtering is where the decision about which features are output to which connection is decided by some form of user-defined test or condition. The Tester transformer is the most obvious example of this. It carries out a test and has different output ports for features that pass and fail the test.

Transformers that Filter

Many transformers in the Filters and Joins category carry out these tests and redirect data according to the results:

Although the Tester transformer is the most used of this category, there are many other transformers such as the TestFilter, GeometryFilter, AttributeFilter, SpatialFilter, and Sampler.

The Tester and TestFilter Transformers

The Tester and TestFilter are the two key transformers for conditional filtering. They test the values on attribute values.

Tester

The Tester transformer is generally for single tests that produce a Yes/No result.

For example, here we wish to decide whether to send out snow plows to a particular road based on whether the value of the Snowfall attribute is greater than 150 mm (approximately 6 inches):

If snowfall is greater than 150, the road feature will pass the test and snow plows will be sent.

Multiple Clauses

Each clause in the Tester is an individual test that allows a Passed/Failed result. For example, each of the following criteria might be separate tests:

Has there been more than 100 mm (4 inches) of snowfall?
Is this a major road?
Is the temperature less than zero degrees Celsius?
Was sand last applied more than 24 hours ago?

However, the Tester allows the combination of multiple tests, where a user can combine any number of clauses using an AND and OR statement. So instead of individual tests, I might ask:

Is this an Arterial (major) road AND has there been more than 100 mm of snow?

The Tester also allows the mixing of AND and OR statements using what is called a Composite Test. For example:

Is this an Arterial road AND (has there been more than 100 mm of snow OR (is the temperature less than zero AND was the road treated (with sand) more than 24 hours ago))?

But - however complex the test becomes - it still results in a single Yes/No (binary) result; features will either pass or fail this set of tests.

It's also worth remembering that we aren't restricted to simple tests of equality (A=B); in the above, there are also "greater than" and "less than" tests. That's because there are many different operators available for use in a test clause.

Operators

The list of operators available in the Tester transformer (or in many of the other locations that make use of the Tester dialog) looks like this:

Besides the usual operators, there are also some based on a SQL where clause. These include:

In
Like
Contains
Begins With
Ends With
Contains Regex

...plus there are other tests that check for the existence of attributes and values:

Attribute has a value
Attribute is Null
Attribute is Empty String
Attribute is Missing

Note :- "Attribute has a value" is the opposite of the three other tests; for example, this attribute is not Null, AND it is not an empty string, AND it is not missing. Incidentally, "missing" means the attribute does not exist at all on the feature being tested.

"Contains Regex" means only part of the string needs to match.

For example...

Attribute Value: abcd

Search String: ^ab

Contains Regex: Passed

TestFilter

The TestFilter carries out the same sort of tests as a Tester, but it can incorporate MULTIPLE tests.

The TestFilter allows a number of conditions to be tested, each of which can have a number of test clauses. Each condition is given an output port, with an additional output port for features that fail all of the test conditions.

The TestFilter is very similar to the CASE or SWITCH command in programming or scripting languages. In Workbench it looks like this:

Notice that there are multiple conditions, and an output port for each. Each condition/port combination is equivalent to a single Tester transformer; hence the TestFilter is a good way to combine multiple Tester transformers into one.

The TestFilter output ports can be given custom names, rather than a simple PASSED/FAILED, which is another advantage to this transformer over the Tester.

The TestFilter has the full set of operators available with the Tester such as equals, greater than, less than, and so forth. Each condition is tested in turn.

Features that pass are output through the matching output port. Features that fail are sent on to the next condition in the list. Therefore it’s very important to get the conditions in the correct order.

Note:- Because the TestFilter can carry out a single test (as well as multiple ones) it's possible to use it exclusively instead of the Tester transformer.

Other Key Filter Transformers

AttributeFilter

The AttributeFilter transformer directs features by values in a chosen attribute. It is not a binary test (Yes/No) but a way to separate many values for a single attribute, for example:

Is that road an Arterial, Collector, NonCity, Other, Private, Residential, or Secondary?

If you were using seven Tester transformers to separate this data, you could save space on the workspace canvas and replace that setup with just a single AttributeFilter.

Note:- Use the Import... button to quickly add attributes from existing datasets.

The AttributeFilter also works with numeric values; however, its only "operator" is to find equivalency (=), so you would rarely use it for arithmetical tests. In that scenario, the better solution is the AttributeRangeFilter.

AttributeRangeFilter

The AttributeRangeFilter carries out the same operation as the AttributeFilter, except that it can handle a range of numeric values instead of just a simple one-to-one match.

For example, we might want to separate data based on a range of snowfall values, like so:

Notice that the AttributeRangeFilter parameters dialog has a Generate button to generate ranges automatically from a set of user-defined extents.

GeometryFilter

The GeometryFilter directs features on the basis of geometry type; for example, point, line, area, ellipse:

It can even filter data based on null geometry (tabular records):

The GeometryFilter is useful for:

Filtering out unwanted geometry types; for example, removing non-linear features before using an AreaBuilder transformer
Validating geometry against a list of permitted types; for example, where the dataset is constrained to either point or area features (above)
Dividing up geometry types to write to separate destination Feature Types; for example, when writing to a geometry-restricted format such as Esri Shapefile.

Filtering Summary

	Single Test		Multiple Tests		Test Type		Operators	Attributes
	Single Clause	Multi Clause	Single Clause	Multi Clause	String	Numeric
Tester	Y	Y	–	–	Y	Y	16	Multiple
TestFilter	Y	Y	Y	Y	Y	Y	16	Multiple
AttributeFilter	Y	Y	–	–	Y	–	1	1
AttributeRangeFilter	Y	Y	–	–	–	Y	6	1

	Single Test		Multiple Tests		Test Type		Operators	Attributes
	Single Clause	Multi Clause	Single Clause	Multi Clause	String	Numeric	(Number of)	(Number of)
AttributeValueMapper	Y	Y	–	–	Y	–	1	1
AttributeRangeMapper	Y	Y	–	–	–	Y	6	6
AttributeCreator	Y	Y	Y	Y	Y	Y	16	Multiple