Python FME API Part1

Introduction

A high-level overview of how and where Python may be applied in FME.

Where Python can be used in FME ?

There are three main places where Python can be used in FME:

The PythonCreator transformer lets you write Python code to create new features.
The PythonCaller transformer lets you write Python code to manipulate existing features in a pipeline.
Each workspace can define Python scripts that run before or after workspace execution. These startup and shutdown scripts can be found in Navigator > Workspace Parameters > Scripting.

Python can also be used to create custom readers, writers, and transformers. Some of the readers, writers, and transformers included with FME rely on Python.

Python versions

Python versions. It can be found in Navigator > Workspace Parameters > Scripting > Python Compatibility. This setting in the workspace takes precedence over FME Options. All Python code executed by the workspace will run with the selected interpreter.

Python 2.7 is supported in FME 2016 and earlier. Python 2.7, 3.4, and 3.5 are supported from FME 2017 to 2018.1. Python 2.7 (deprecated), 3.5, 3.6, and 3.7 are supported as of FME 2019.0.

On Windows, the FME 2019.0 installation includes both Python 2.7 (optional component) and 3.7. On Mac, the FME 2019.0 installation includes Python 3.7. On Linux, FME uses the system Python interpreter.

In FME 2019.0, Python 3.7 is the default interpreter.In FME 2021.0,Python 3.8 is the default interpreter.

PythonCreator transformer

The PythonCreator transformer can be used to create new features in the workspace. It contains a Python code template that looks similar to this:

import fmeobjects

class FeatureCreator(object):

    def __init__(self):
        pass

    # create features before first reader feature is processed
    def input(self, feature):
        newFeature = fmeobjects.FMEFeature()
        self.pyoutput(newFeature)

    # output features in close() to create the features
    # after all reader features have been processed
    def close(self):
        pass 

The first line, import fmeobjects, imports the FME Objects Python API. In FeatureCreator.input(), an FMEFeature is constructed, then returned using self.pyoutput().

Creates features with a user-supplied Python script.

Typical Uses

Reading unsupported formats
Prototyping Python format plugins
Tasks where a transformer is not available

PythonCaller Transformer

The PythonCaller transformer can be used to manipulate existing features using Python. It contains a Python code template that looks similar to this:

import fmeobjects

class FeatureProcessor(object):

    def __init__(self):
        pass

    def input(self, feature):
        self.pyoutput(feature)

    def close(self):
        pass

Typical Uses

Tasks where a transformer is not available.
Using external modules for processing.
Performing complex manipulations on list attributes.

This template is similar to the one in the PythonCreator transformer. The difference is that its input() method receives an FMEFeature as an argument. Each feature entering the PythonCaller’s input port is passed to the input() method in the PythonCaller, where it can be worked with. self.pyoutput() then emits the feature from the PythonCaller’s output port.

Though the PythonCaller is intended for manipulating existing features in a pipeline, it’s also possible to use it to create new features, just like PythonCreator. self.pyoutput() can be called multiple times in order to emit more than one feature for each feature entering the input port. The same feature can also be emitted multiple times: each feature from the output port is a copy.

Using FME’s Python REPL

Aside from using the PythonCreator and PythonCaller transformers described above, another good way to experiment with the FME Objects Python API is the FME Python interpreter’s interactive shell. To use it, open a terminal or command prompt in your FME install directory, and enter:

> fme python
Python 2.7.10 (default, May 23 2015, 09:44:00) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import fmeobjects
>>>

How does it work?

The PythonCaller executes a Python script to manipulate features.

When a specialized task is required, such as custom statistical analysis of an attribute, but Workbench does not provide a transformer suited to the task, a Python script can perform specialized and complex operations on a feature's geometry, attributes, and coordinate system.

Interface Paradigm

The PythonCaller can interface with a Python script in two different ways - by a function or by a class.

Use the Function Interface when you intend to process a single feature at a time. (in versions older than 2021.0)
Use the Class Interface for more flexibility including group by functionality

The Class Interface is useful when you want to operate on a group of features together, such as collecting all the features received and then outputting them in a specific sort order.

Note: the Function interface has been removed from the PythonCaller template in 2021.0. However, existing Python functions will continue to run in FME 2021+.

Class Interface

The PythonCaller can interface with a class defined in a Python script. The calling sequence on the methods defined in a class depends on which mode the PythonCaller is in.

There are two modes - Standard and Group By.

Standard Mode

This is the operation mode when no attributes are set in the Group By parameter. In this mode, which is the most common, the PythonCaller will have the following calling sequence on a class:

__init__() - Called once, whether or not any features are processed.
input() - Called for each FME Feature that comes into the input port.
close() - Called once, after all features are processed (when no more FME Features remain). If no features are processed, the close() method will still be called.

Features that need to continue through the workspace for further processing must be explicitly written out using the pyoutput()method.

When the class interface is processing incoming FMEFeatures one at a time, the pyoutput() method is to be called once per incoming FMEFeature in the input() method. Conversely, when the class interface is operating on a group of FMEFeatures, the incoming FMEFeatures can be stored in a list, then processed and written out through pyoutput() in the close() method.

Default Python Code Template:-

init()

Default FMEFeature constructor.

Return type:	FMEFeature
Returns:	An instance of a Feature object.

__init__ Initialize self.

init(feature)

Create a copy of the passed in Feature object.

Parameters:	feature (FMEFeature) – The Feature object to create a copy of.
Return type:	FMEFeature
Returns:	An instance of a Feature object.

Getting and setting the feature type

When using a PythonCaller to work with features that originated from a reader, it may be useful to know the feature’s feature type. getFeatureType() is used to obtain the feature type.

setFeatureType() is used to set the feature type of a feature.

Getting and setting attributes

Use setAttribute() to set all non-null attributes. Most attribute types are supported. To set a list attribute, pass a list as the value. To set null attribute values, see the next section.

Below is an example of getting and setting various attributes:

>>> from fmeobjects import FMEFeature, FME_ATTR_REAL32

>>> feature = FMEFeature()
>>> feature.setAttribute('beavers', 10)
>>> feature.setAttribute('park_name', 'Rat Park')
>>> feature.setAttribute('rangers', ['Alice', 'Bob'])
>>> feature.getAttribute('beavers'), feature.getAttribute('park_name'), feature.getAttribute('rangers')
10, 'Rat Park', ['Alice', 'Bob']

Working with null or missing attributes

getAttribute() cannot be used to determine whether an attribute is missing or has a null value. Similarly, setAttribute() cannot be used to set null values on attributes. Separate methods are dedicated to this task.

To set null attribute values, use setAttributeNullWithType(), passing it the name of the attribute to set to null, and the attribute type it would have been if it weren’t null.

To determine whether an attribute is missing, has a null value, and its type, use getAttributeNullMissingAndType(). It returns a 3-element tuple.

Below is an example of getting and setting null and missing attributes:

>>> from fmeobjects import FMEFeature, FME_ATTR_REAL32

>>> feature = FMEFeature()
>>> feature.setAttributeNullWithType('park_area', FME_ATTR_REAL32)
>>> feature.getAttributeNullMissingAndType('park_area')
True, False, 8
>>> feature.setAttribute('raccoons', 64)
>>> feature.removeAttribute('raccoons')
>>> feature.getAttributeNullMissingAndType('raccoons')
False, True, 0
>>> feature.getAttributeNullMissingAndType('nonexistent')
False, True, 0

Working with geometry :

The FME Objects Python API has many geometry classes. The following is a simple example that constructs a 2D point and an offset 2D point, offsets the point by the offset point, fetches the new coordinates, and sets the new point to a feature in two different ways:

>>> from fmeobjects import FMEFeature, FMEPoint, FMEGeometryTools

>>> feature = FMEFeature()
>>> point = FMEPoint(10, 20)
>>> offset = FMEPoint(10, 10)
>>> point.offset(offset)
>>> point.getXYZ()
(20.0, 30.0, 0.0)
>>> feature.setGeometry(point)
>>> feature.getGeometry().getXYZ()
(20.0, 30.0, 0.0)
>>> offsetPoint = FMEGeomtryTools().offset(point, offset)
>>> offsetPoint.getXYZ()
(30.0, 40.0, 0.0)
>>> feature.setGeometry(offsetPoint)
>>> feature.getGeometry().getXYZ()
(30.0, 40.0, 0.0)

Many geometric operations are available through FMEGeometryTools. Consult their API for details.

Coordinate systems and reprojection

Coordinate systems are set at the feature level instead of the geometry level. By default, features do not have a coordinate system. To assign one, pass a coordinate system name to setCoordSys():

>>> from fmeobjects import FMEFeature, FMEPoint

>>> feature = FMEFeature()
>>> feature.setGeometry(FMEPoint(-122.842764, 49.177847))
>>> feature.setCoordSys('LL84')

When a feature has a coordinate system, it can be reprojected using reproject(). This operation is similar to the Reprojector transformer. For example:

>>> feature.reproject('SPHERICAL_MERCATOR')
>>> feature.getCoordSys()
'SPHERICAL_MERCATOR'
>>> feature.getGeometry().getXYZ()
(-13674793.936118279, 6305092.363454558, 0.0)

The FMECoordSysManager and FMEReprojector classes are available for more advanced operations with coordinate systems and reprojection.

Writing messages to the FME log

Python’s print() function can be used to log messages, but for best results in FME, use FMELogFile:

>>> from fmeobjects import FMELogFile

>>> FMELogFile().logMessageString("hello world")

If this code executed in Workbench, this immediately prints “hello world” as an informational message in the log.

Example Code

1) The example below calculates the total area of all the features processed and then outputs all the features with a new attribute containing the total area:

import fme
import fmeobjects
 
class FeatureProcessor(object):
    def __init__(self):
        self.feature_list = []
        self.total_area = 0.0
 
    def input(self, feature):
        self.feature_list.append(feature)
        self.total_area += feature.getGeometry().getArea()
 
    def close(self):
        for feature in self.feature_list:
            feature.setAttribute("total_area", self.total_area)
            self.pyoutput(feature)

    def process_group(self):
        pass