Flow Filter Anatomy

Flow filters are the basic unit of execution inside of Ascent, and all functionality is implemented as a Flow filter. The full interface to a Flow filter can be found in the Flow filter header file. Here is a summary of the functions relevant to a filter developer:

public:
  Filter();
  virtual ~Filter();

  // override and fill i with the info about the filter's interface
  virtual void          declare_interface(conduit::Node &i) = 0;

  // override to imp filter's work
  virtual void          execute() = 0;

  // optionally override to allow filter to verify custom params
  // (used as a guard when a filter instance is created in a graph)
  virtual bool          verify_params(const conduit::Node &params,
                                      conduit::Node &info);

A derived filter must minimally implement the declare_interface and execute methods, but it is highly encouraged that a new filter implement verify_params as well. verify_params alerts users to input errors and unexpected parameters.

Note

Developing a flow filter requires a working knowledge of the Conduit API. In the Introduction to Ascent section under Conduit Examples, there are several examples of basic Conduit usage. More Conduit tutorial resources can be found in the Conduit documentation.

Flow filter implementations are located in the src/libs/ascent/runtimes/flow_filters directory.

Implementing A New Filter

As a convenience, we have created the VTKHNoOp filter as staring point and reference. Although the NoOp filter demonstrates how to use a VTK-h filter, the implementation is relevant to anyone developing flow filters in Ascent regardless of whether VTK-h or Viskores is used.

Interface Declaration

void
VTKHNoOp::declare_interface(conduit::Node &i)
{
    i["type_name"]   = "vtkh_no_op";
    i["port_names"].append() = "in";
    i["output_port"] = "true";
    i["param_schema"] = conduit::Node();
}

  • type_name: declares the name of the filter to flow, and the only requirement is that this name be unique.

  • port_names: declares a list of input port names.

  • output_port: declares if this filter has an output of not. Valid values are true and false.

  • param_schema: defines a parameter schema as a conduit node expressing the required and optional parameters and their types.

The port_names parameter is a list of input port names that can be referenced by name or index when creating the filter within the runtime. The typical number of inputs is one, but there is no restriction on the input count. To add additional inputs, additional append() calls will add more inputs to the port list, and the input port names must be unique.

i["port_names"].append() = "in1";
i["port_names"].append() = "in2";

For the majority of developers, a transform (i.e., a filter that can be part of a pipeline) filter will have one input (e.g., the data set) and one output. If creating an extract, the output_port should be declared false indicating that this filter is a sink.

Parameter Verification

Parameters are passed through Ascent and then to filters. For detailed examples of filter in Ascent see the Pipelines section.

How Are Parameters Passed?

The parameters are passed to the Ascent API through Conduit nodes. A simple filter interface looks like this in c++:

conduit::Node filter;
filter["type"] = "filter_name";
filter["params/string_param"] = "string";
filter["params/double_param"] = 2.0;

or equivalently in yaml:

type: "filter_name"
params:
  string_param: "string"
  double_param: 2.0

The Ascent runtime looks for the params node and passes it to the filter upon creation. Parameters are verified when the filter is created during execution.

Filter Parameter Verification Schemas

Filter parameters are validated and checked for any surprises using Parameter Schemas for Validation and Surprise Checking. In practice, a filter’s interface param_schema describes the expected structure of the params node that users provide when creating a filter.

Note

In prior versions of Ascent, the verify_params method was used to allow the filter creator to define parameter verification and surprise checking for each individual flow filter. The base implementation of verify_params now uses the param_schema defined in the filter interface to validate parameters and check for surprises against the expected schema using Parameter Schemas for Validation and Surprise Checking.

Ascent Parameter Schema Helpers

To assist in constructing parameter schemas, Ascent has a number of small schema builder helpers defined in:

  • src/libs/ascent/runtimes/flow_filters/ascent_runtime_param_check.hpp

  • src/libs/ascent/runtimes/flow_filters/ascent_runtime_param_check.cpp

Each helper returns a Conduit node that represents a schema fragment (see Parameter Schemas for Validation and Surprise Checking). These fragments are typically inserted under param_schema["properties/<param_name>"].

Example pattern:

using namespace ascent::runtime::filters;

void MyFilter::declare_interface(conduit::Node &i)
{
    i["type_name"] = "my_filter";
    i["port_names"].append() = "in";
    i["output_port"] = "true";

    conduit::Node &param_schema = i["param_schema"];
    param_schema["type"] = "object";
    param_schema["additionalProperties"] = false;

    string_schema(param_schema["properties/field"], 1); // non-empty string
    param_schema["required"].append() = "field";
}
string_schema

Builds a string schema with optional length bounds.

conduit::Node &string_schema(conduit::Node &schema_node,
                             const std::size_t minLength = 0,
                             const std::size_t maxLength = std::numeric_limits<std::size_t>::max());
Example: Using ``string_schema`` (with length bounds)
// Example: require a non-empty string with a max length
conduit::Node &param_schema = i["param_schema"];
param_schema["type"] = "object";
param_schema["additionalProperties"] = false;

string_schema(param_schema["properties/field"], 1, 64);
param_schema["required"].append() = "field";

Resulting schema fragment:

type: string
minLength: 1
maxLength: 64

Example input accepted by this schema:

params:
  field: "pressure"

string_enum_schema

Builds a string schema restricted to an enumerated set of allowed values.

conduit::Node &string_enum_schema(conduit::Node &schema_node,
                                  const std::vector<std::string> &options);
Example: Using ``string_enum_schema`` (string enum)
// Example: restrict a string parameter to a fixed set of values
conduit::Node &param_schema = i["param_schema"];
param_schema["type"] = "object";
param_schema["additionalProperties"] = false;

string_enum_schema(param_schema["properties/interpolation"], {"nearest", "linear"});
param_schema["required"].append() = "interpolation";

Resulting schema fragment:

type: string
enum: ["nearest", "linear"]

Example input accepted by this schema:

params:
  interpolation: "nearest"

bool_schema

Builds a boolean-like schema represented as a string enum: "true" or "false".

conduit::Node &bool_schema(conduit::Node &schema_node);
Example: Using ``bool_schema`` ("true"/"false" string enum)
// Example: model a boolean-like parameter as "true"/"false" strings
conduit::Node &param_schema = i["param_schema"];
param_schema["type"] = "object";
param_schema["additionalProperties"] = false;

bool_schema(param_schema["properties/enabled"]);
param_schema["required"].append() = "enabled";

Resulting schema fragment:

type: string
enum: ["true", "false"]

Example input accepted by this schema:

# Note: the values must be strings, so quote them in YAML.
params:
  enabled: "true"

number_schema

Builds a numeric schema, optionally with bounds. If supports_expressions is true, the schema accepts either a number or an expression string (via oneOf).

conduit::Node &number_schema(conduit::Node &schema_node,
                             const bool supports_expressions = false,
                             const int minimum = std::numeric_limits<int>::lowest(),
                             const int maximum = std::numeric_limits<int>::max(),
                             const int exclusiveMinimum = std::numeric_limits<int>::lowest(),
                             const int exclusiveMaximum = std::numeric_limits<int>::max());
Example: Using ``number_schema`` (bounded number)
// Example: accept a number in [0, 10]
conduit::Node &param_schema = i["param_schema"];
param_schema["type"] = "object";
param_schema["additionalProperties"] = false;

number_schema(param_schema["properties/threshold"], false, 0, 10);
param_schema["required"].append() = "threshold";

Resulting schema fragment:

# Example: bounded number (no expressions)
type: number
minimum: 0
maximum: 10

Example input accepted by this schema:

params:
  threshold: 3.5

Example: Using ``number_schema`` (number or expression)
// Example: accept either a numeric value or an expression string
conduit::Node &param_schema = i["param_schema"];
param_schema["type"] = "object";
param_schema["additionalProperties"] = false;

number_schema(param_schema["properties/threshold"], true, 0, 10);
param_schema["required"].append() = "threshold";

Resulting schema fragment:

# Example: number OR expression string
oneOf:
  - {type: number, minimum: 0, maximum: 10}
  - {type: string, format: expression}

Example input accepted by this schema:

params:
  threshold: "1 + 2"

integer_schema

Builds an integer schema, optionally with bounds. If supports_expressions is true, the schema accepts either an integer or an expression string (via oneOf).

conduit::Node &integer_schema(conduit::Node &schema_node,
                              const bool supports_expressions = false,
                              const int minimum = std::numeric_limits<int>::lowest(),
                              const int maximum = std::numeric_limits<int>::max(),
                              const int exclusiveMinimum = std::numeric_limits<int>::lowest(),
                              const int exclusiveMaximum = std::numeric_limits<int>::max());
Example: Using ``integer_schema`` (bounded integer)
// Example: accept an integer >= 1
conduit::Node &param_schema = i["param_schema"];
param_schema["type"] = "object";
param_schema["additionalProperties"] = false;

integer_schema(param_schema["properties/levels"], false, 1);
param_schema["required"].append() = "levels";

Resulting schema fragment:

type: integer
minimum: 1

Example input accepted by this schema:

params:
  levels: 8

Example: Using ``integer_schema`` (integer or expression)
// Example: accept either an integer value or an expression string
conduit::Node &param_schema = i["param_schema"];
param_schema["type"] = "object";
param_schema["additionalProperties"] = false;

integer_schema(param_schema["properties/levels"], true, 1);
param_schema["required"].append() = "levels";

Resulting schema fragment:

oneOf:
  - {type: integer, minimum: 1}
  - {type: string, format: expression}

Example input accepted by this schema:

params:
  levels: "2 + 2"

vec3_schema

Builds an object schema that requires three numeric components. The default component names are x, y, and z.

conduit::Node &vec3_schema(conduit::Node &schema_node,
                           const bool supports_expressions = false);

conduit::Node &vec3_schema(conduit::Node &schema_node,
                           const std::string var1,
                           const std::string var2,
                           const std::string var3,
                           const bool supports_expressions = false);
Example: Using ``vec3_schema`` (required x/y/z object)
// Example: require all 3 components: x, y, z
conduit::Node &param_schema = i["param_schema"];
param_schema["type"] = "object";
param_schema["additionalProperties"] = false;

vec3_schema(param_schema["properties/offset"]);
param_schema["required"].append() = "offset";

Resulting schema fragment:

type: object
additionalProperties: false
required: ["x", "y", "z"]
properties:
  x: {type: number}
  y: {type: number}
  z: {type: number}

Example input accepted by this schema:

params:
  offset: {x: 0.0, y: 1.0, z: 2.0}

Example: Using ``vec3_schema`` with custom param names
// Example: require all 3 components: x, y, z
conduit::Node &param_schema = i["param_schema"];
param_schema["type"] = "object";
param_schema["additionalProperties"] = false;

vec3_schema(param_schema["properties/offset"], "var_i", "var_j", "var_k");
param_schema["required"].append() = "offset";

Resulting schema fragment:

type: object
additionalProperties: false
required: ["var_i", "var_j", "var_k"]
properties:
  var_i: {type: number}
  var_j: {type: number}
  var_k: {type: number}

Example input accepted by this schema:

params:
  offset: {var_i: 0.0, var_j: 1.0, var_k: 2.0}

vec3_schema_anyOf

Builds an object schema that allows any subset of three numeric components, but requires that at least one of them is present (via anyOf). The default component names are x, y, and z.

conduit::Node &vec3_schema_anyOf(conduit::Node &schema_node,
                                 const bool supports_expressions = false);

conduit::Node &vec3_schema_anyOf(conduit::Node &schema_node,
                                 const std::string var1,
                                 const std::string var2,
                                 const std::string var3,
                                 const bool supports_expressions = false);
Example: Using ``vec3_schema_anyOf`` (x/y/z object, at least one required)
// Example: allow x/y/z, but require at least one component to be present
conduit::Node &param_schema = i["param_schema"];
param_schema["type"] = "object";
param_schema["additionalProperties"] = false;

vec3_schema_anyOf(param_schema["properties/origin"]);
param_schema["required"].append() = "origin";

Resulting schema fragment:

type: object
additionalProperties: false
properties:
  x: {type: number}
  y: {type: number}
  z: {type: number}
anyOf:
  - {type: object, required: ["x"]}
  - {type: object, required: ["y"]}
  - {type: object, required: ["z"]}

Example input accepted by this schema:

params:
  origin: {x: 0.0}  # or {y: 1.0} / {z: 2.0} / {x: 0.0, y: 1.0}, etc.

array_schema

Builds an array schema. With item_schema, each element is validated against the provided schema.

conduit::Node &array_schema(conduit::Node &schema_node,
                            const conduit::Node &item_schema = conduit::Node(),
                            const std::size_t minItems = 0,
                            const std::size_t maxItems = std::numeric_limits<std::size_t>::max());
Example: Using ``array_schema`` (array of numbers)
// Example: require an array where each item is a number
conduit::Node &param_schema = i["param_schema"];
param_schema["type"] = "object";
param_schema["additionalProperties"] = false;

conduit::Node num_schema;
array_schema(param_schema["properties/iso_values"], number_schema(num_schema));
param_schema["required"].append() = "iso_values";

Resulting schema fragment:

# Array of numbers
type: array
items: {type: number}

Example input accepted by this schema:

params:
  iso_values: [0.1, 0.2, 0.3]

ignore_schema

Builds an object schema that is explicitly skipped during validation (via constraints/skip). This is useful when a parameter subtree is accepted but not yet formally specified, or when validation is handled elsewhere.

Note

Ignore schema allows a parameter to be surprise checked against (when additionalProperties is false) but the actual value will not be validated against any schema. This is an explicit way of saying “I want field to be a valid value passed to this filter but I am not going to worry about if it is a stirng or number or something else just yet”.

Note

Generally, if a validation schema can be provided it should be instead of using an ignore_schem.

conduit::Node &ignore_schema(conduit::Node &schema_node);
Example: Using ``ignore_schema`` (skip validation for a subtree)
// Example: accept a subtree but skip validating it
conduit::Node &param_schema = i["param_schema"];
param_schema["type"] = "object";
param_schema["additionalProperties"] = false;

ignore_schema(param_schema["properties/options"]);
param_schema["required"].append() = "options";

Resulting schema fragment:

type: object
constraints:
  skip: true

Example input accepted by this schema:

params:
  options:
    any_subtree: "is accepted"
    nested: {a: 1, b: [1, 2, 3]}

Execute

The execute() method does the real work. In our example, we are wrapping the VTKHNoOp filter which is a transform, i.e., a filter that can be called inside of a pipeline. Be default, transforms are passed VTK-h data sets and extracts are called with either Conduit Blueprint data sets (i.e., the data published by the simulation) or VTK-h data sets, when the extract consumes the result of a pipeline. The data type can be checked by the filter and converted by one of Ascent’s data adapters located in the src/libs/ascent/runtimes directory.

Listing 2 An example execute method
 1void
 2VTKHNoOp::execute()
 3{
 4
 5    if(!input(0).check_type<vtkh::DataSet>())
 6    {
 7        ASCENT_ERROR("vtkh_no_op input must be a vtk-h dataset");
 8    }
 9
10    std::string field_name = params()["field"].as_string();
11
12    vtkh::DataSet *data = input<vtkh::DataSet>(0);
13    vtkh::NoOp noop;
14
15    noop.SetInput(data);
16    noop.SetField(field_name);
17
18    noop.Update();
19
20    vtkh::DataSet *noop_output = noop.GetOutput();
21
22    set_output<vtkh::DataSet>(noop_output);
23}

Filter Inputs

Inputs to filters are always pointers. Lines 5-8 demonstrate how to check the type of data to the filter. input(0).check_type<SomeType>() returns true if the input pointer is of the same type as the template parameter. Alternatively, we could reference the input port by its declared interface name: input("in").check_type<SomeType>().

Warning

If you perform input data type conversion, the temporary converted data must be deleted before exiting the execute method.

Once the filter input type is known it is safe to call input<KnownType>(0) to retrieve the pointer to the input (line 12).

Flow filters have a member function params() that returns a reference to the Conduit node containing the filter parameters that were previously verified. Since we already verified the existence of the string parameter field, it is safe to grab that parameter without checking the type or path.

For optional parameters, care should be used when accessing node paths. Conduit nodes paths can be checked with params().has_path("some_path") Other methods exist to verify or convert their underlying types such as node["path"].is_numeric(). If you are expecting an integer the semantics between these two calls are very different:

  • node["path"].as_int32(): I am positive this is an int32 and I alone accept the consequences if it is not

  • node["path"].to_int32(): I am expecting an int32 and please convert if for me assuming whatever type it is can be converted to what I am expecting

Filter Output

A filter’s output is a pointer to a data sets. In the case of tranforms this type is expected to be a VTK-h data set. Output pointers are reference counted by Flow’s registry and will be deleted when no downstream filter needs the output of the current filter.

In the case of an extract, no output needs to be set.

Registering Filters With Ascent

Newly created filters need to be registered with the Ascent runtime. The file ascent_runtime_filters.cpp is where all builtin filter are registered. Following the NoOp example:

Listing 3 Ascent runtime filter registration
AscentRuntime::register_filter_type<VTKHNoOp>("transforms","noop");

Filter registration is templated on the filter type and takes two arguments.

  • arg1: the type of the filter. Valid values are transforms and extracts

  • arg2: the front-facing API name of the filter. This is what a user would declare in an actions file.

Accessing Metadata

We currently populate a limited set of metadata that is accessible to flow filters. We place a Conduit node containing the metadata inside the registry which can be accessed in the following manner:

Listing 4 Accessing the registry metadata inside a flow filter
conduit::Node * meta = graph().workspace().registry().fetch<Node>("metadata");
int cycle = -1;
float time = -1.f;
if(meta->has_path("cycle"))
{
  cycle = (*meta)["cycle"].to_int32();
}
if(meta->has_path("time"))
{
   time = (*meta)["time"].to_int32();
}

The above code is conservative, checking to see if the paths exist. The current metadata values Ascent populates are:

  • cycle: simulation cycle

  • time: simulation time

  • refinement_level: number of times a high-order mesh is refined

If these values are not provided by the simulation, then defaults are used.

Using the Registry (state)

Filters are created and destroyed every time the graph is executed. Filters might want to keep state associated with a particular execution of the filter. A conduit node is a convenient container for arbitrary data, but there is no restriction on the type of data that can go inside the registry.

Listing 5 Accessing the registry metadata inside a flow filter
conduit::Node *my_state_data = new conduit::Node();
// insert some data to the node

// adding the  node to the registry
graph().workspace().registry().add<conduit::Node>("my_state", my_state_data, 1);

// check for existence and retrieve
if(graph().workspace().registry().has_entry("my_state"))
{
  conduit::Node *data = graph().workspace().registry().fetch<conduit::Node>("my_state"))
  // do more stuff
}

Data kept in the registry will be destroyed when Ascent is torn down, but will persist otherwise. A problem that arises is how to tell different invocations of the same filter apart, since a filter can be called an arbitrary number of times every time ascent is executed. The Ascent runtime gives unique names to filters that can be accessed by a filter member function this->detailed_name(). One possible solution is to use this name to differentiate filter invocations. This approach is reasonable if the actions remain the same throughout the simulation, but if they might change, all bets are o ff.

Note

Advanced support of registry and workspace usage is only supported through the Ascent developers platinum support contract, which can be purchased with baby unicorn tears. Alternatively, you are encouraged to look at the flow source code, unit tests, and ask questions.

Using MPI Inside Ascent

Ascent creates two separate libraries for MPI and non-MPI (i.e., serial). In order to maintain the same interface for both versions of the library, MPI_Comm handles are represented by integers and are converted to the MPI implementations underlying representation by using the MPI_Comm_f2c function.

Code containing calls to MPI are protected by the define ASCENT_MPI_ENABLED and calls to MPI API calls must be guarded inside the code. In Ascent, the MPI comm handle is stored in and can be retrieved from the flow::Workspace which is accessible from inside a flow filter.

Listing 6 Example of code inside a filter that retrieves the MPI comm handle from the workspace
#ifdef ASCENT_MPI_ENABLED
  int comm_id = flow::Workspace::default_mpi_comm();
  MPI_Comm mpi_comm = MPI_Comm_f2c(comm_id);
  int rank;
  MPI_Comm_rank(comm, &rank);
#endif