Field Build Functions

These functions can be used in order to build and manipulate the fields

abs_value
FunctionDescriptionSourcesOutputArguments
abs_valueReturns the absolute value for a given numeric value.

numeric_value
(type: Number)
Number
{
  "abs_value_of_source": {
    "type": "numeric",
    "function": "abs_value",
    "source": "source_name"
  }
}
{
  "source_name": -6
}
{
  "abs_value_of_source": 6
}
all_checks
FunctionDescriptionSourcesOutputArguments
all_checksA directive to check for multiple conditions. Receives a sequence of
values, returns True iff all the values return True.
Note that it is possible and advised to use this method over results of
other boolean field functions (such as check_existence, equal_const and
negative_check) in order to create complex boolean fields.
Not suitable for a boolean array source, only for multiple boolean sources.

If remove_if_null is set to True, if one of the boolean_values is null, this field
will be removed altogether.

boolean_values
(type: Sequence[Optional[bool]])
Optional[bool]remove_if_null

{
  "all_sources_are_true": {
    "type": "boolean",
    "function": "all_checks",
    "sources": [
      "source_name_1",
      "source_name_2"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name_1": true,
  "source_name_2": true
}
<EXAMPLE OUTPUT>
{
  "all_sources_are_true": true
}
all_values_field_in_array
FunctionDescriptionSourcesOutputArguments
all_values_field_in_arrayA directive which returns an array of values from all the occurrences of
a key in an array of dictionaries.

array_value
(type: Sequence[Mapping])
Sequencekey
(type: str)
{
  "values_from_objects_in_array": {
    "type": "numeric",
    "function": "all_values_field_in_array",
    "source": "source_name",
    "args": [
      "name"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "name": "john",
      "age": 13
    },
    {
      "name": "bob",
      "age": 20
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "values_from_objects_in_array": [
    "john",
    "bob"
  ]
}
area
FunctionDescriptionSourcesOutputArguments
areaA directive for calculating the area of polygon.
The input should be a polygon in the format of a
GeoJson (json or WKT) or a TopoJson.
The returned area will be in square meter (m^2).
Set convert_to_km to true in order to get the result in square kilometer(km^2)

polygon
convert_to_km

{
  "area_of_polygon": {
    "type": "numeric",
    "function": "area",
    "source": "polygon"
  }
}
<EXAMPLE INPUT 1>
{
    "polygon": {
        "type": "Polygon",
        "coordinates": [[[-50,-90],[-50,90],[50,90],[-50,-90]]]
    }
}
<EXAMPLE INPUT 2>
{
    "polygon": "POLYGON ((-50 -90, -50 90, 50 90, 50 -90, -50 -90))"
}
<EXAMPLE OUTPUT >
{
    "area_of_polygon": 18000.0
}
array_intersection
FunctionDescriptionSourcesOutputArguments
array_intersectionA directive for getting the intersection of two arrays

first_array_value
(type: Sequence)

second_array_value
(type: Sequence)
Sequence
{
  "array_from_intersected_arrays": {
    "type": "string",
    "function": "array_intersection",
    "sources": [
      "source_name_1",
      "source_name_2"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name_1": [7,1,5,2,3,6],
  "source_name_2": [3,8,6,20,7]
}
<EXAMPLE OUTPUT>
{
  "array_from_intersected_arrays": [3,6,7]
}
array_length
FunctionDescriptionSourcesOutputArguments
array_lengthA directive for creating a field according to the length of an array value.

array_value
(type: Sequence)
intdefault_value

{
  "length_of_array": {
    "type": "numeric",
    "function": "array_length",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": [1,2,3,4,6,9,22]
}
<EXAMPLE OUTPUT>
{
  "length_of_array": 7
}
array_numbers
FunctionDescriptionSourcesOutputArguments
array_numbersA directive for creating an array which consists only of the numeric values
of a given array.
array_value
(type: Sequence)
Sequence[Number]default_value

{
  "numeric_values": {
    "type": "numeric",
    "function": "array_numbers",
    "source": "mixed_array"
  }
}
<EXAMPLE INPUT>
{
  "mixed_array": [1,5,3,true,"dog"]
}
<EXAMPLE OUTPUT>
{
  "numeric_values": [1,5,3]
}
array_top_value
FunctionDescriptionSourcesOutputArguments
array_top_valueA directive for creating a field according to the max numeric value of an array.
Note: given array must contain only numeric values.
array_value
(type: Sequence[Number])
Optional[Number]default_value

{
  "top_value_of_array": {
    "type": "numeric",
    "function": "array_top_value",
    "source": "array_source"
  }
}
<EXAMPLE INPUT>
{
  "array_source": [1,5,32,3,2.8]
}
<EXAMPLE OUTPUT>
{
  "top_value_of_array": 32
}
array_tops_delta
FunctionDescriptionSourcesOutputArguments
array_tops_deltaA directive for creating a field according to the delta between the two largest
numeric values of an array.
Note: given array must contain only numeric values.
array_value
(type: Sequence[Number])
Optional[Number]default_value

{
  "delta_of_array_top_values": {
    "type": "numeric",
    "function": "array_top_value",
    "source": "array_source"
  }
}
<EXAMPLE INPUT>
{
  "array_source": [32,1,5,12]
}
<EXAMPLE OUTPUT>
{
  "delta_of_array_top_values": 20
}
averages_delta
FunctionDescriptionSourcesOutputArguments
averages_deltaReturns the delta between the average value of the two arrays.

first_array_value
(type: Sequence[Optional[Number]])

second_array_value
(type: Sequence[Optional[Number]])
Optional[Number]
{
  "delta_of_averages_of_two_arrays": {
    "type": "numeric",
    "function": "averages_delta",
    "sources": [
      "source_name_1",
      "source_name_2"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name_1": [3,7,10,20],
  "source_name_2": [1,5,9]
}
<EXAMPLE OUTPUT>
{
  "delta_of_averages_of_two_arrays": 5
}
boolean_via_numeric
FunctionDescriptionSourcesOutputArguments
boolean_via_numericReturn the boolean value of the numeric value that the given str describes.

value
(type: str)
bool
{
  "boolean_value_field": {
    "type": "boolean",
    "function": "boolean_via_numeric",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": "0"
}
<EXAMPLE OUTPUT>
{
  boolean_value_field": false
}
clean_null_str
FunctionDescriptionSourcesOutputArguments
clean_null_strA workaround directive to nullify "null" strings coming by mistake from the user.

actual_value
{
  "nullified_field_name": {
    "type": "string",
    "source": "source_name",
    "function": "clean_null_str"
  },
  "not_nullified_field_name": {
    "type": "string",
    "source": "source_name_2",
    "function": "clean_null_str"
}
<EXAMPLE INPUT>
{
  "source_name": "null",
  "source_name_2": "dog"
}
<EXAMPLE OUTPUT>
{
  "nullified_field_name": null,
  "not_nullified_field_name": "dog"
}
concat
FunctionDescriptionSourcesOutputArguments
concatA directive which takes a sequence field, and returns its values concatenated
as a string.

actual_value
(type: Union[Mapping, Sequence])
Optional[str]
{
  "concatenated_array": {
    "type": "string",
    "function": "concat",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    "John",
    "Bob",
    "Sharon",
    "Joe"
  ]
}
<EXAMPLE OUTPUT>
{
  "concatenated_array": [
    "John_Bob_Sharon_Joe"
  ]
}
concat_inner_array
FunctionDescriptionSourcesOutputArguments
concat_inner_arrayReturns an array concatenation of all values in the the inner_array_key in the
objects originally found in the source array.

outer_array_value
(type: Sequence[Mapping])
Sequenceinner_array_key
(type: str)
{
  "array_of_concatenated_arrays": {
    "type": "string",
    "function": "concat_inner_array",
    "source": "source_name",
    "args": [
      "a"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "a": [1,2,3],
      "b": ["a","b","c"]
    },
    {
      "a": [4,5,6],
      "b": ["d","e","f"]
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "array_of_concatenated_arrays": ["1","2","3","4","5","6"]
}
concat_strings
FunctionDescriptionSourcesOutputArguments
concat_stringsA directive for concatenating all the given string/strings sequences sources.
If sources are strings sequences, concat each index of the sources.

If values contains str arrays, in case of a None arg in one of the arrays, the
returned value in the corresponding index will be None.

values
(type: Sequence[Union[str, Sequence[str]]])
Union[str, Sequence[str]]separator
(type: str)
{
  "concatenated_string": {
    "type": "string",
    "function": "concat_strings",
    "sources": [
      "source_name_1",
      "source_name_2"
    ],
    "args": [
      ", "
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name_1": "Jack",
  "source_name_2": "Black"
}
<EXAMPLE OUTPUT>
{
  "concatenated_string": "Jack, Black"
}
concat_strings_recursive_arrays
FunctionDescriptionSourcesOutputArguments
concat_strings_recursive_arraysA directive for generating a sequence of strings, concatanated from keys of objects
within arrays. It currently has a very specific usage: It expects a source which is
an array of objects, then a sequence of keys, the first of which should be a key to
look for a string value, the second should be a key to look for a new array of
objects, the third is again a key for a string value within this new array, and the
forth is again a key for looking for a new array of objects, and so on.

top_array_value
(type: Sequence[Optional[Mapping]])
Optional[str]internal_keys
(type: Sequence[str])

separator
(type: str)
{
  "concatenated_strings_in_recursive_arrays": {
    "type": "string",
    "source": "source_name",
    "function": "concat_strings_recursive_arrays",
    "args": [
      [
        "feature_id",
        "attributes",
        "attribute_1",
        "values",
        "unique_id"
      ]
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "feature_id": 3833,
      "attributes": [
        {
          "attribute_1": 1242,
          "values": [
            {
              "unique_id": 12
            }
          ]
        }
      ]
    },
    {
      "feature_id": 2336,
      "attributes": [
        {
          "attribute_1": 923,
          "values": [
            {
              "unique_id": 132
            }
          ]
        },
        {
          "attribute_1": 1759,
          "values": [
            {
              "unique_id": 243
            }
          ]
        }
      ]
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "concatenated_strings_in_recursive_arrays": [
    "3833_1242_12",
    "2336_923_132",
    "2336_1759_243"
  ]
}
const_is_in_array
FunctionDescriptionSourcesOutputArguments
const_is_in_arrayReturn true if the given array contains the given constant value, false otherwise.

array_value
(type: Sequence)
boolvalue

{
  "does_source_contains_const": {
    "type": "boolean",
    "function": "const_is_in_array",
    "source": array_source,
    "args": [
      "blue"
    ]
  }
}
<EXAMPLE INPUT>
{
  "array_source": [red, yellow, ,blue, orange]
}
<EXAMPLE OUTPUT>
{
  "does_source_contains_const": true
}
convert_to_float
FunctionDescriptionSourcesOutputArguments
convert_to_floatConvert string to float safely (if it's already int/float - no conversion needed).
In case the given value is not representing a number the return value is None.
Values in null_values are replaced with null.
original_value
(type: Optional[Number])
Numbernull_values

count_characters
FunctionDescriptionSourcesOutputArguments
count_charactersA directive which gets a string value or array, and returns the amount of
characters (if array - for each element).

string_or_array_value
int
{
  "count_characters_field_name": {
    "type": "int",
    "function": "count_characters",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
    "source_name": ["Hi there, my name is John.", "Hello"]
}
<EXAMPLE OUTPUT>
{
    "count_characters_field_name": [26, 5]
}
count_occurrences_in_array
FunctionDescriptionSourcesOutputArguments
count_occurrences_in_arrayA directive that, given an array field, counts the amount of items in that array in
which the value for given key is equal to the given inner_value.

array_value
(type: Sequence[Mapping])
intkey
(type: str)

inner_value


default_value

{
  "number_of_value_occurrences": {
    "type": "numeric",
    "function": "count_occurrences_in_array",
    "source": "source_name",
    "args": [
      "name",
      "Bob"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "name": "John",
      "age": 13
    },
    {
      "name": "Bob",
      "age": 20
    },
    {
      "name": "Sharon",
      "age": 17
    },
    {
      "name": "Bob",
      "age": 10
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "number_of_value_occurrences": 2
}
count_recursively
FunctionDescriptionSourcesOutputArguments
count_recursivelyA directive which counts all internal elements in the given array, recursively.

array_value
(type: Sequence)
int
{
  "number_of_elements_recursively": {
    "type": "numeric",
    "function": "count_recursively",
    "source": "source_name"
  }
}
{
  "source_name": [
    "bob",
    [
      "dog",
      "cat"
    ],
    "sharon"
  ]
}
{
  "number_of_elements_recursively": 4
}
count_words
FunctionDescriptionSourcesOutputArguments
count_wordsA directive which gets a string value or array, and returns the amount of
words (if array - for each element).
The amount of words is calculated based on the standard python split method
without args, i.e. words are separated by any number and form of whitespace
chars.

string_or_array_value
int
{
  "count_words_field_name": {
    "type": "int",
    "function": "count_words",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
    "source_name": ["Hi there, my name is John.", "Hello"]
}
<EXAMPLE OUTPUT>
{
    "count_words_field_name": [6, 1]
}
day_of_week_from_timestamp_seconds
FunctionDescriptionSourcesOutputArguments
day_of_week_from_timestamp_secondsReturns the day of week (UTC) from the given timestamp.

timestamp_value
(type: Number)
{
  "day_of_timestamp": {
    "type": "string",
    "function": "day_of_week_from_timestamp_seconds",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": 1610569800
}
<EXAMPLE OUTPUT>
{
  "day_of_timestamp": "Wednesday"
}
dedup_array_of_objects_by_key
FunctionDescriptionSourcesOutputArguments
dedup_array_of_objects_by_keyReturns the given array, deduped by the values in the given key.

array_value
(type: Sequence)
Sequencededup_key
(type: str)
{
  "deduped_sub_array": {
    "type": "string",
    "function": "dedup_array_of_objects_by_key",
    "source": "source_name",
    "args": [
      "name"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "name": "John",
      "age": 20
    },
    {
      "name": "Bob",
      "age": 30
    },
    {
      "name": "Max",
      "age": 19
    },
    {
      "name": "Bob",
      "age": 10
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "deduped_sub_array": [
    {
      "name": "John",
      "age": 20
    },
    {
      "name": "Bob",
      "age": 30
    },
    {
      "name": "Max",
      "age": 19
    }
  ]
}
delta
FunctionDescriptionSourcesOutputArguments
deltaA directive for calculating delta of two values.

first_value
(type: Optional[Number])

second_value
(type: Optional[Number])
Optional[Number]default_value

{
  "delta_between_values": {
    "type": "numeric",
    "sources": [
      "numeric_source_1",
      "numeric_source_2"
    ],
    "function": "delta"
  }
}
<EXAMPLE INPUT>
{
  "numeric_source_1": 50,
  "numeric_source_2": 2
}
<EXAMPLE OUTPUT>
{
  "delta_between_values": 48
}
delta_between_first_and_second
FunctionDescriptionSourcesOutputArguments
delta_between_first_and_secondReturns the delta between the max value and the second max value from a list of
(dynamic) multiple numeric value sources. Not suitable for numerical array sources.

numeric_values
(type: Sequence[Optional[Number]])
Optional[Number]
{
  "delta_of_two_highest_values": {
    "type": "numeric",
    "function": "delta_between_first_and_second",
    "sources": [
      "source_name_1",
      "source_name_2",
      "source_name_3"
    ]
  }
}
{
  "source_name_1": 5,
  "source_name_2": 10,
  "source_name_3": 7
}
{
  "delta_of_two_highest_values": 3
}
delta_from_const
FunctionDescriptionSourcesOutputArguments
delta_from_constA directive for calculating delta between a given number and a const.

first_value
(type: Optional[Number])
Optional[Number]const_to_subtract
(type: Number)

default_value

{
  "delta_between_source_and_const": {
    "type": "numeric",
    "function": "delta_from_const",
    "source": "numeric_source_name",
    "args": [
      10
    ]
  }
}
<EXAMPLE INPUT>
{
  "numeric_source_name": 50
}
<EXAMPLE OUTPUT>
{
  "delta_between_source_and_const": 40
}
distance
FunctionDescriptionSourcesOutputArguments
distanceA directive for calculating the distance between two points.
The input should be two given Geopoints.
The returned distance will be in meters (m).
Set convert_to_km to true in order to get the result in kilometers (km)

point1

point2
convert_to_km

{
    "distance_between_two_points": {
    "function": "distance",
    "sources": ["point1", "point2"],
    "type": "numeric"
    }
  }
<EXAMPLE INPUT>
{
    [{
        "point1": {"location" : "POINT (-71.34 41.12)" },
        "point2": {"location" : "POINT (52.406374 16.9251681)" }
    }]
}
<EXAMPLE OUTPUT>
{
    "distance_between_two_points": 11365307.89
}
divide
FunctionDescriptionSourcesOutputArguments
divideA directive for dividing one value by another.

numerator_value
(type: Optional[Number])

denominator_value
(type: Optional[Number])
Optional[Number]
{
  "quotient_of_division": {
    "type": "numeric",
    "function": "divide",
    "sources": [
      "numeric_source_name_1",
      "numeric_source_name_2"
    ]
  }
}
<EXAMPLE INPUT>
{
  "numeric_source_name_1": 50,
  "numeric_source_name_2": 10
}
<EXAMPLE OUTPUT>
{
  "quotient_of_division": 5
}
divide_by_const
FunctionDescriptionSourcesOutputArguments
divide_by_constA directive for dividing one value by a const.

numerator_value
(type: Optional[Number])
Optional[Number]denominator
(type: Optional[Number])
{
  "quotient_of_division_with_const": {
    "type": "numeric",
    "function": "divide_by_const",
    "source": "numeric_source_name",
    "args": [
      "5"
    ]
  }
}
<EXAMPLE INPUT>
{
  "numeric_source_name_": 50
}
<EXAMPLE OUTPUT>
{
  "quotient_of_division_with_const": 10
}
dynamic_field_value_from_object
FunctionDescriptionSourcesOutputArguments
dynamic_field_value_from_objectReturns the value of a dynamically chosen field within an object.
dict_object_value
(type: Mapping)

field_name_value
{
  "value_of_dynamic_field": {
    "type": "string",
    "function": "dynamic_field_value_from_object",
    "sources": [
      "source_name",
      "field_key_name"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": {
    "name": "Bob",
    "age": 23,
    "city": "New York"
  },
  "field_key_name": "city"
}
<EXAMPLE OUTPUT>
{
  "value_of_dynamic_field": "New York"
}
empty_to_null
FunctionDescriptionSourcesOutputArguments
empty_to_nullA directive which is just like the identity function for strings, only it returns
"null" for empty strings.

original_value
(type: str)
str
{
  "example_field_name": {
    "type": "string",
    "function": "empty_to_null",
    "sources": [
      "source_name"
    ]
  },
  "empty_field_name": {
    "type": "string",
    "function": "empty_to_null",
    "sources": [
      "source_name_2"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": "dog",
  "source_name_2": ""
}
<EXAMPLE OUTPUT>
{
  "example_field_name": "dog",
  "empty_field_name": null
}
equality_check
FunctionDescriptionSourcesOutputArguments
equality_checkA directive to check for equality between two values.

first_value

second_value
Optional[bool]
{
  "is_equale": {
    "type": "boolean",
    "function": "equality_check",
    "sources": [
      "source_name_1",
      "source_name_2"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name_1": 10,
  "source_name_2": 10
}
<EXAMPLE OUTPUT>
{
  "is_equale": true
}
equality_check_const
FunctionDescriptionSourcesOutputArguments
equality_check_constA directive to check for equality between a value and a constant value.

value
const

{
  "is_equal_to_const": {
    "type": "boolean",
    "function": "equality_check_const",
    "source": "source_name",
    "args": [
      10
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": 10
}
<EXAMPLE OUTPUT>
{
  "is_equal_to_const": true
}
existence_check
FunctionDescriptionSourcesOutputArguments
existence_checkA directive to check for an existence of a value.

value
boolnon_existent_value

{
  "value_exists": {
    "type": "boolean",
    "function": "existence_check",
    "source": "source_name"
  },
  "value_exists_2": {
    "type": "boolean",
    "function": "existence_check",
    "source": "source_name_2"
  },
  "value_exists_3": {
    "type": "boolean",
    "function": "existence_check",
    "source": "source_name_3"
  }
}
<EXAMPLE INPUT>
{
  "source_name": "John",
  "source_name_2": null
}
<EXAMPLE OUTPUT>
{
  "value_exists": true,
  "value_exists_2": false,
  "value_exists_3": false
}
field_delta
FunctionDescriptionSourcesOutputArguments
field_deltaA directive to get a field delta between the first and second values of a
specific key in an array field.

NOTE: If no second value is found, we use zero
instead.

array_value
(type: Sequence[Mapping[str, Number]])
Optional[Number]key
(type: str)

sort
(type: bool)

reversed_sort
(type: bool)

default_value

{
  "delta_of_first_and_second_values": {
    "type": "string",
    "function": "field_delta",
    "source": "source_name",
    "args": [
      "age"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "name": "John",
      "age": 20
    },
    {
      "name": "Bob",
      "age": 13
    },
    {
      "name": "Sharon",
      "age": 17
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "delta_of_first_and_second_values": 7
}
field_value_from_object
FunctionDescriptionSourcesOutputArguments
field_value_from_objectReturns the value of a field within an object

dict_object_value
(type: Mapping)
field_name

{
  "city_name_from_obj": {
    "type": "string",
    "function": "field_value_from_object",
    "source": "source_name",
    "args": [
      "city"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": {
    "name": "Bob",
    "age": 23,
    "city": "New York"
  }
}
<EXAMPLE OUTPUT>
{
  "city_name_from_obj": "New York"
}
filter_according_to_boolean_array
FunctionDescriptionSourcesOutputArguments
filter_according_to_boolean_arrayReturns the original array, filtered according to a same-length array of boolean
values.

original_array_value

boolean_array_value
should_invert

{
  "filtered_array": {
    "type": "string",
    "function": "filter_according_to_boolean_array",
    "sources": [
      "array_source_name",
      "boolean_array_source_name"
    ]
  }
}
<EXAMPLE INPUT>
{
  "array_source_name": [
    "cat",
    "cow",
    "dog",
    "sheep"
  ],
  "boolean_array_source_name": [
    true,
    true,
    false,
    true
  ]
}
<EXAMPLE OUTPUT>
{
  "filtered_array": [
    "cat",
    "cow",
    "sheep"
  ]
}
filter_array
FunctionDescriptionSourcesOutputArguments
filter_arrayReturns a sub array for the given array of objects, according to whether the value
in the given key of the object is from the possible values parameter.

If the bool_evaluation parameter is set, will cast the value of the given key to
bool before checking.

if the key parameter is None, will evaluate the given values in the array directly,
and will assume they are not dictionaries.

The returned value is still an array of objects, and should therefore never be
"saved" in the schema (use "do_not_save": true).

array_value
(type: Sequence[Mapping])
Sequencekey
(type: str)

possible_values
(type: Sequence)

bool_evaluation
(type: bool)
{
  "filtered_sub_array": {
    "type": "string",
    "function": "filter_array",
    "source": "source_name",
    "args": [
      "name",
      [
        "John",
        "Bob"
      ]
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "name": "John",
      "age": 13
    },
    {
      "name": "Bob",
      "age": 20
    },
    {
      "name": "Sharon",
      "age": 17
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "filtered_sub_array": [
    {
      "name": "John",
      "age": 13
    },
    {
      "name": "Bob",
      "age": 20
    }
  ]
}
filter_array_by_blacklist
FunctionDescriptionSourcesOutputArguments
filter_array_by_blacklistReturns a sub array for the given array of objects, according to whether the value
in the given key of the object is not from the blacklisted_values.

The returned value is still an array of objects, and should therefore never be
"saved" in the schema (use "do_not_save": true).

array_value
(type: Sequence[Mapping])
Sequence[Mapping]key
(type: str)

blacklisted_values
(type: Sequence)
{
  "filtered_sub_array": {
    "type": "string",
    "function": "filter_array_by_blacklist",
    "source": "source_name",
    "args": [
      "name",
      [
        "Bob",
        "Sharon"
      ]
    ],
    "do_not_use": true
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "name": "John",
      "age": 14
    },
    {
      "name": "Bob",
      "age": 33
    },
    {
      "name": "Max",
      "age": 45
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "filtered_sub_array": [
    {
      "name": "John",
      "age": 14
    },
    {
      "name": "Max",
      "age": 45
    }
  ]
}
filter_array_by_valid_ranges
FunctionDescriptionSourcesOutputArguments
filter_array_by_valid_rangesReturns a sub array for the given array of objects, according to whether the value
in the given key of the object is in at least one of the given valid_ranges.

If the given key is None, then this directive assumes the array isn't of objects,
but of the actual numbers needed to be filtered according to the ranges.

valid_ranges should be an array of pairs, which represent the minimum (inclusive)
and maximum (exclusive) for each range.
null (pythonic: None) can be used to suggest no minimum or no maximum.

array_value
(type: Sequence[Union[Mapping[str, Number], Number]])
Sequencekey
(type: str)

valid_ranges
(type: Sequence[Sequence[Number]])
{
  "filtered_sub_array": {
    "function": "filter_array_by_valid_ranges",
    "source": "source_name",
    "args": [
      "age",
      [
        [12,18],
        [30,35]
      ]
    ],
    "do_not_save": true
  },
  "filtred_sub_array_length": {
    "type": "numeric",
    "function": "array_length",
    "source": "filtered_sub_array"
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "name": "John",
      "age": 13
    },
    {
      "name": "Bob",
      "age": 20
    },
    {
      "name": "Sharon",
      "age": 17
    },
    {
      "name": "Bob",
      "age": 10
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "filtered_sub_array": [
    {
      "name": "John",
      "age": 13
    },
    {
      "name": "Sharon",
      "age": 17
    }
  ],
  "filtred_sub_array_length": 2
}
filter_array_static_key_and_dynamic_value
FunctionDescriptionSourcesOutputArguments
filter_array_static_key_and_dynamic_valueReturns a sub array for the given array of objects, according to whether the
value in the given key of the object is equal to the compared value given
dynamically as a source.

The returned value is still an array of objects, and should therefore never
be "saved" in the schema (use "do_not_save": true).

array_value
(type: Sequence[Mapping])

compared_value
Sequencekey
(type: str)
{
  "dynamically_filtered_sub_array": {
    "type": "string",
    "function": "filter_array_static_key_and_dynamic_value",
    "sources": [
      "array_value_source_name",
      "dynamic_value_source_name"
    ]
    "args": [
      "name"
    ]
  }
}
<EXAMPLE INPUT>
{
  "array_value_source_name": [
    {
      "name": "John",
      "age": 13
    },
    {
      "name": "Bob",
      "age": 20
    },
    {
      "name": "Sharon",
      "age": 17
    }
  ],
  "dynamic_value_source_name": "John"
}
<EXAMPLE OUTPUT>
{
  "dynamically_filtered_sub_array": [
    {
      "name": "John",
      "age": 13
    }
  ]
}
filter_duplicates
FunctionDescriptionSourcesOutputArguments
filter_duplicatesA directive for getting a new array, with filtered duplicates.
If the given value is not an array - return the value as is.

array_value
(type: Sequence)
Sequence
{
  "filtered_sub_array": {
    "type": "string",
    "function": "filter_duplicates",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": [1,2,2,2,3,4]
}
<EXAMPLE OUTPUT>
{
  "filtered_sub_array": [1,2,3,4]
}
first_in_array
FunctionDescriptionSourcesOutputArguments
first_in_arrayReturns the first value in an array.

array_value
(type: Sequence)
default_value

{
  "first_value_in_array": {
    "type": "string",
    "function": "first_in_array",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    "John",
    "Bob",
    "Max"
  ]
}
<EXAMPLE OUTPUT>
{
  "first_value_in_array": "John"
}
flatten_array_recursive
FunctionDescriptionSourcesOutputArguments
flatten_array_recursiveA directive to flatten an array with nested arrays to one level.
array_value
(type: Sequence)
Sequence
{
  "flattened_array": {
    "type": "string",
    "function": "flatten_array_recursive",
    "source": "array_source_name"
  }
}
{
  "array_source_name": [
    "bob",
    [
      "dog",
      "cat"
    ],
    "sharon"
  ]
}
{
  "flattened_array": [
    "bob",
    "dog",
    "cat",
    "sharon"
  ]
}
format_string
FunctionDescriptionSourcesOutputArguments
format_stringA directive for string formatting.
If some of the input values are arrays, this function will output an array for all
the possible combinations of the inputs.

input_values
(type: Sequence)
strtemplate
(type: str)
{
  "formatted_string_field": {
    "type": "string",
    "function": "format_string",
    "sources": [
      "source_name_1",
      "source_name_2"
    ],
    "args": [
      "{} and {} are co-workers"
    ]
  }
}
<EXAMPLE INPUT>
{
    "source_name_1": "John",
    "source_name_2": "Bob
}
<EXAMPLE OUTPUT>
{
    "formatted_string_field": "John and Bob are co-workers"
}
from_array_index
FunctionDescriptionSourcesOutputArguments
from_array_indexReturns the value in the given index in the given array.
array_value
index

{
  "value_of_index": {
    "type": "numeric",
    "function": "from_array_index",
    "source": array_source,
    "args": [
      2
    ]
  }
}
<EXAMPLE INPUT>
{
  "array_source": [10,12,6,3,11]
}
<EXAMPLE OUTPUT>
{
  "value_of_index": 6
}
geo_intersect
FunctionDescriptionSourcesOutputArguments
geo_intersectA directive for calculating the intersection of two given polygons.
The input should be GeoJson or TopoJson format or WKT (as a json or a json string)
and the output will be returned as WKT string.

polygon1

polygon2
{
  "intersection_of_polygons": {
    "function": "geo_intersect",
    "sources": ["poly1", "poly2"],
    "type": "string"
  }
}
<EXAMPLE INPUT>
{
    [{
      "poly1": {
        "type": "Polygon",
        "coordinates": [[
          [-180,-90],
          [-180,90],
          [180,90],
          [180,-90],
          [-180,-90]
        ]]
      },
      "poly2": {
        "type": "Polygon",
        "coordinates": [[
            [-122.801742, 45.48565],
            [-122.801742, 45.60491],
            [-122.584762, 45.60491],
            [-122.584762, 45.48565],
            [-122.801742, 45.48565]
        ]]
      }
    }]
}
<EXAMPLE OUTPUT>
{
    ""intersection_of_polygons": POLYGON ((-122.801742 45.60491,
                -122.584762 45.60491,
                -122.584762 45.48565,
                -122.801742 45.48565,
                -122.801742 45.60491
    ))"
}
get_dict_key_on_check
FunctionDescriptionSourcesOutputArguments
get_dict_key_on_checkReturn the value of a dict's key according to a check of check_key being
equal to check_value (static value supplied from the configuration as an
argument)

dict_objects_array_value
(type: Sequence[Mapping])
key
(type: str)

check_key
(type: str)

check_value


compare_as_floats
(type: bool)
{
  "value_of_key_by_args": {
    "type": "string",
    "source": "source_name",
    "function": "get_dict_key_on_check",
    "args": [
      "b",
      "a",
      3
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "a": 1,
      "b": 2
    },
    {
      "a": 3,
      "b": 4
    },
    {
      "a": 5,
      "b": 6
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "value_of_key_by_args": "4"
}
get_dict_key_on_dynamic_check
FunctionDescriptionSourcesOutputArguments
get_dict_key_on_dynamic_checkReturn the value of a dict's key according to a check of check_key being
equal to dynamic value supplied from a source.

check_value

dict_objects_array_value
(type: Sequence[Mapping])
key
(type: str)

check_key
(type: str)

compare_as_floats
(type: bool)
{
  "value_of_key_by_dynamic_check": {
    "type": "string",
    "sources": [
      "check_value_source",
      "source_name"
    ],
    "function": "get_dict_key_on_dynamic_check",
    "args": [
      "b",
      "a"
    ]
  }
}
<EXAMPLE INPUT>
{
  "check_value_source": 3,
  "source_name": [
    {
      "a": 1,
      "b": 2
    },
    {
      "a": 3,
      "b": 4
    },
    {
      "a": 5,
      "b": 6
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "value_of_key_by_dynamic_check": "4"
}
get_nth_highest_key
FunctionDescriptionSourcesOutputArguments
get_nth_highest_keyReturns the key of the nth highest value in a given dict.

dict_value
(type: Mapping[str, Number])
n
(type: int)
{
  "key_of_nth_highest_value": {
    "type": "numeric",
    "function": "get_nth_highest_key",
    "source": "source_name",
    "args": [
      1
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": {
    "average_age_1": 34,
    "average_age_2": 33,
    "average_age_3": 42
  }
}
<EXAMPLE OUTPUT>
{
  "key_of_nth_highest_value": "average_age_2"
}
get_nth_highest_value
FunctionDescriptionSourcesOutputArguments
get_nth_highest_valueReturns the nth highest value of all values in a given dict.

dict_value
(type: Mapping[str, Number])
n
(type: int)
{
  "nth_highest_value_of_keys": {
    "type": "numeric",
    "function": "get_nth_highest_value",
    "source": "source_name",
    "args": [
      1
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": {
    "average_age_1": 34,
    "average_age_2": 33,
    "average_age_3": 42
  }
}
<EXAMPLE OUTPUT>
{
  "nth_highest_value_of_keys": 33
}
get_top_key
FunctionDescriptionSourcesOutputArguments
get_top_key Returns the key with the highest number in its value, from dictionary which maps
strings to numbers.

dict_value
(type: Mapping[str, Number])
{
   "key_of_highest_value": {
     "type": "string",
     "function": "get_top_key",
     "source": "source_name"
   }
 }
 <EXAMPLE INPUT>
 {
   "source_name": {
     "key_1": 4,
     "key_2": 12,
     "key_3": 7,
     "key_4": 22
   }
 }
 <EXAMPLE OUTPUT>
{
   "key_of_highest_value": "key_4"
 }
get_top_value
FunctionDescriptionSourcesOutputArguments
get_top_valueReturns the highest value of all values in a given dict.

dict_value
(type: Mapping[str, Number])
{
  "highest_value_of_keys": {
    "type": "numeric",
    "function": "get_top_value",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": {
    "key_1": 4,
    "key_2": 12,
    "key_3": 7,
    "key_4": 22
  }
}
<EXAMPLE OUTPUT>
{
  "highest_value_of_keys": 22
}
hour_of_day_from_timestamp_seconds
FunctionDescriptionSourcesOutputArguments
hour_of_day_from_timestamp_secondsReturns the hour of the day (UTC) from the given timestamp.

timestamp_value
(type: Number)
{
  "hour_of_timestamp": {
    "type": "numeric",
    "function": "hour_of_day_from_timestamp_seconds",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": 1610569800
}
<EXAMPLE OUTPUT>
{
  "hour_of_timestamp": 20
}
identity
FunctionDescriptionSourcesOutputArguments
identityA default identity field build function.value
import_from_other_context_class
FunctionDescriptionSourcesOutputArguments
import_from_other_context_classImports the value of the field from other context class.
Using the given foreign id value, fetch a corresponding existing CRC and returns
the values in the foreign field.

The string_to_replace_dot param allows to replace the dot character in the
foreign_ids to a provided string.

foreign_ids
(type: Union[str, list, tuple])
foreign_field
(type: str)

foreign_context_class
(type: str)

foreign_id_prefix


string_to_replace_dot
(type: str)
{
  "imported_value": {
    "type": "string,
    "function": "import_from_other_context_class",
    "source": "some_id",
    "args": [
      "name_of_field_in_other_context_class",
      "NAME_OF_OTHER_CONTEXT_CLASS"
    ]
  }
}
{
  "some_id": "532n-kl52-gs53-weg5"
}
{
  "imported_value": "value_of_field_in_other_context_class"
}
is_in_dynamic_set
FunctionDescriptionSourcesOutputArguments
is_in_dynamic_setA directive to check if a value is in a set of values.

value

set_of_values
(type: Sequence)
Optional[bool]
{
  "fields": {
    "value_is_in_set": {
      "type": "boolean",
      "function": "is_in_dynamic_set",
      "sources": ["source_name", "source_name_2"],
    },
  },
}
<EXAMPLE INPUT>
{
  "source_name": "John",
  "source_name_2": ["John", "Bob", "Frank", "Paul"]
}
<EXAMPLE OUTPUT>
{
  "value_is_in_set": true
}
is_in_other_array_mapper
FunctionDescriptionSourcesOutputArguments
is_in_other_array_mapperA mapper directive for boolean values according to whether each value is in
another array or not

first_array_value
(type: Sequence)

second_array_value
(type: Sequence)
Sequence[bool]
{
  "boolean_values_according_to_sources": {
    "type": "boolean",
    "function": "is_in_other_array_mapper",
    "sources": [
      "source_name_1",
      "source_name_2"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name_1": [
    "John",
    "Bob",
    "Sharon",
    "Joe"
  ],
  "source_name_2": [
    "Max",
    "Bob",
    "Peter",
    "Joe"
  ]
}
<EXAMPLE OUTPUT>
{
  "boolean_values_according_to_sources": [
    false,
    true,
    false,
    true
  ]
}
is_in_set
FunctionDescriptionSourcesOutputArguments
is_in_setA directive to check if a value is in a given values set.

value
Optional[bool]set_key
(type: str)

treat_missing_as_false
(type: bool)
{
  "fields": {
    "value_is_in_set": {
      "type": "boolean",
      "function": "is_in_set",
      "source": "source_name",
      "args": [
        "set_name"
      ]
    },
  },
  "data": {
    "set_name": [
      "John",
      "Bob",
      "Frank",
      "Paul"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": "John"
}
<EXAMPLE OUTPUT>
{
  "value_is_in_set": true
}
is_monotone
FunctionDescriptionSourcesOutputArguments
is_monotoneA directive which checks if the given array is monotone.

array_value
(type: Sequence)
boolascending


use_weak

{
  "array_is_monotone": {
    "type": "boolean",
    "function": "is_monotone",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": [1,3,4,6,12]
}
<EXAMPLE OUTPUT>
{
  "array_is_monotone": true
}
keep_only_digits
FunctionDescriptionSourcesOutputArguments
keep_only_digitsA directive which takes a string (or array of strings) and returns that string only
with its digits, stripping away all non-digit characters.

actual_value
(type: Union[str, Sequence[str]])
{
  "digits_from_string": {
    "type": "string",
    "source": "source_name",
    "function": "keep_only_digits"
  }
}
<EXAMPLE INPUT>
{
  "source_name": "fe223"
}
<EXAMPLE OUTPUT>
{
  "digits_from_string": "223"
}
last_in_array
FunctionDescriptionSourcesOutputArguments
last_in_arrayReturns the last value in an array.

array_value
(type: Sequence)
default_value

{
  "last_value_in_array": {
    "type": "string",
    "function": "last_in_array",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    "John",
    "Bob",
    "Max"
  ]
}
<EXAMPLE OUTPUT>
{
  "last_value_in_array": "Max"
}
max_delta
FunctionDescriptionSourcesOutputArguments
max_deltaA directive which returns the delta between the min and max numbers in a given
source array.

array_value
(type: Sequence[Number])
Number
{
  "delta_of_highest_and_lowest_values": {
    "type": "numeric",
    "function": "max_delta",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": [1,3,12,6,4]
}
<EXAMPLE OUTPUT>
{
  "delta_of_highest_and_lowest_values": 11
}
max_value
FunctionDescriptionSourcesOutputArguments
max_valueReturns the max value from list of (dynamic) multiple numeric value sources.
Not suitable for numerical array sources.

numeric_values
(type: Sequence[Optional[Number]])
Optional[Number]
{
  "highest_value_from_sources": {
    "type": "numeric",
    "function": "max_value",
    "sources": [
      "source_name_1",
      "source_name_2",
      "source_name_3"
    ]
  }
}
{
  "source_name_1": 5,
  "source_name_2": 10,
  "source_name_3": 7
}
{
  "highest_value_from_sources": 10
}
month_from_timestamp
FunctionDescriptionSourcesOutputArguments
month_from_timestampA directive to get the month of the year of a given timestamp.

timestamp_value
(type: Number)
Number
{
  "month_in_number": {
    "type": "numeric",
    "function": "month_from_timestamp",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": 1610569800
}
<EXAMPLE OUTPUT>
{
  "month_in_number": 1
}
multiply
FunctionDescriptionSourcesOutputArguments
multiplyReturns the product of all the input values. If the inputs are arrays, then they
have to be of the same length, and the result will be an array of that same length
with all the products for each index.

input_values
{
  "product_of_multiplication": {
    "type": "numeric",
    "sources": [
      "numeric_source_1",
      "numeric_source_2"
    ],
    "function": "multiply"
  }
}
<EXAMPLE INPUT>
{
  "numeric_source_1": 5,
  "numeric_source_2": 10
}
<EXAMPLE OUTPUT>
{
  "product_of_multiplication": 50
}
multiply_by_const
FunctionDescriptionSourcesOutputArguments
multiply_by_constA directive for multiplying a value by a constant given as an arg.
# TODO(adi): We assume here that zero means null, we should split this
# function to multiply_by_const_no_zero.

original_value
(type: Optional[Number])
Numbermultiplier

{
  "product_of_multiplication": {
    "type": "numeric",
    "source": "numeric_source",
    "function": "multiply_by_const",
    "args": [
      "10"
    ]
  }
}
<EXAMPLE INPUT>
{
  "numeric_source": 5
}
<EXAMPLE OUTPUT>
{
  "product_of_multiplication": 50
}
negative_check
FunctionDescriptionSourcesOutputArguments
negative_checkA directive to check for the negative value of the specified value.
Returns True if the value is False.

boolean_value
(type: bool)
Optional[bool]treat_missing_as_false
(type: bool)
{
  "value_is_negative": {
    "type": "boolean",
    "function": "negative_check",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": false
}
<EXAMPLE OUTPUT>
{
  "value_is_negative": true
}
object_values_as_array
FunctionDescriptionSourcesOutputArguments
object_values_as_arrayReturns the values of a dict as an array. Fields using this function should never be
saved to crc (use "do_not_save": true in field config).

TODO(itai): Create a decorator which tags such functions and doesn't allow them to
be saved.

dict_value
(type: Optional[Mapping])
Sequence
{
  "array_of_values_from_object": {
    "type": "string",
    "function": "object_values_as_array",
    "source": "source_name",
    "do_not_save": true
  }
}
<EXAMPLE INPUT>
{
  "source_name": {
    "name": "John",
    "age": 30,
    "city": "New York"
  }
}
<EXAMPLE OUTPUT>
{
  "array_of_values_from_object": [
    "John",
    30,
    "New York"
  ]
}
parse_datetime_to_timestamp_seconds
FunctionDescriptionSourcesOutputArguments
parse_datetime_to_timestamp_secondsA directive for parsing a datetime object into epoch timestamp seconds.
If the allow_timestamps flag is set to true, the directive will not reject plain
timestamps and will return a given timestamp in seconds. Can work on lists as well.

value
(type: Union[str, int, Sequence[str], Sequence[int]])
Union[Number, Sequence[Number]]python_datetime_formats
(type: Union[str, Sequence[str]])

allow_timestamps
(type: bool)
{
  "timestamp_from_date": {
    "type": "numeric",
    "source": "datetime_object_source",
    "function": "parse_datetime_to_timestamp_seconds",
    "args": [
      ["%m/%d/%Y, %H:%M:%S", "%H:%M:%S-%m-%d-%Y"]
    ]
  }
}
<EXAMPLE INPUT>
{
  "datetime_object_source": "02/14/2021, 20:30:00"
}
<EXAMPLE OUTPUT>
{
  "timestamp_from_date": 1613334600
}
<EXAMPLE INPUT>
{
  "datetime_object_source": ["02/14/2021, 20:30:00", "02/18/2021, 08:30:00"]
}
<EXAMPLE OUTPUT>
{
  "timestamp_from_date": [1613334600, 1613637000]
}
<EXAMPLE INPUT>
{
  "datetime_object_source": "1613334600000",
  "allow_timestamps": "true"
}
<EXAMPLE OUTPUT>
{
  "timestamp_from_date": 1613334600
}
powerset_concat
FunctionDescriptionSourcesOutputArguments
powerset_concatA directive which returns the powerset of the given array, with its values
concatanated to one string for each subset.

array_value
(type: Sequence)
Sequence[str]separator
(type: str)

min_subset_size
(type: int)

max_subset_size
(type: int)
{
  "stringified_concatenated_powerset": {
    "type": "string",
    "source": "source_name",
    "function": "powerset_concat",
    "args": [
      ", "
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [1,2,3]
}
<EXAMPLE OUTPUT>
{
  "stringified_concatenated_powerset": [
    "1.0",
    "2.0",
    "3.0",
    "1.0, 2.0",
    "1.0, 3.0",
    "2.0, 3.0",
    "1.0, 2.0, 3.0"
  ]
}
put_value_according_to_bool
FunctionDescriptionSourcesOutputArguments
put_value_according_to_boolA directive which returns a value according a source boolean.

bool_value
(type: bool)
use_on_false


use_on_true

{
  "value_according_to_boolean": {
    "type": "string",
    "source": "boolean_source",
    "function": "put_value_according_to_bool",
    "args": [
      "dogs",
      "cats"
    ]
  },
  "value_according_to_boolean_2": {
    "type": "string",
    "source": "boolean_source_2",
    "function": "put_value_according_to_bool",
    "args": [
      "dogs",
      "cats"
    ]
  }
}
<EXAMPLE INPUT>
{
  "boolean_source": true,
  "boolean_source_2": false
}
<EXAMPLE OUTPUT>
{
  "value_according_to_boolean": "cats",
  "value_according_to_boolean_2": "dogs"
}
range_check
FunctionDescriptionSourcesOutputArguments
range_checkChecks if a value is in one of the given ranges. If given an array as source input,
will return a boolean response for each index in the array.

numeric_value
ranges
(type: Sequence[Mapping[str, Mapping[str, Union[Number, bool]]]])
{
  "value_is_in_range": {
    "type": "boolean",
    "function": "range_check",
    "source": "source_name",
    "args": [
      [
        {
          "left": {
            "value": 100,
            "inclusive": false
          }
        }
      ]
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": 150
}
<EXAMPLE OUTPUT>
{
  "value_is_in_range": true
}
remove_if_false
FunctionDescriptionSourcesOutputArguments
remove_if_falseReturns first value if the boolean value is true, else None.

first_value

boolean_value
(type: bool)
{
  "first_value_if_true": {
    "type": "numeric",
    "function": "remove_if_false",
    "sources": [
      "source_name_1",
      "source_name_2"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name_1": 10,
  "source_name_2": false
}
<EXAMPLE OUTPUT>
{
  "first_value_if_true": null
}
replace_char
FunctionDescriptionSourcesOutputArguments
replace_charA directive which gets a string or list of strings value, and returns it
with char_to_replace replaced by char_to_insert.

original_string_or_string_list
char_to_replace


char_to_insert

{
  "string_with_char_replaced_field_name": {
    "type": "string",
    "function": "replace_char",
    "source": "source_string"
    "args": "p", "l"
  }
  "string_list_with_char_replaced_field_name": {
    "type": "string",
    "function": "replace_char",
    "source": "source_string_array"
    "args": " ", ""
}
<EXAMPLE INPUT>
{
    "source_string": "hocus pocus"
    "source_string_array": ["hocus pocus", "abra cadabra"]
}
<EXAMPLE OUTPUT>
{
    "string_with_char_replaced_field_name": "hucus locus"
    "string_list_with_char_replaced_field_name": ["hocuspocus", "abracadabra"]
}
season_from_timestamp
FunctionDescriptionSourcesOutputArguments
season_from_timestampA directive to get the season of the year of a given timestamp.

timestamp_value
(type: Number)
is_northern_hemisphere

{
  "season_of_timestamp_north": {
    "type": "string",
    "function": "season_from_timestamp",
    "source": "source_name",
    "args": [
      true
    ]
  },
  "season_of_timestamp_south": {
    "type": "string",
    "function": "season_from_timestamp",
    "source": "source_name",
    "args": [
      false
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": 1610569800
}
<EXAMPLE OUTPUT>
{
  "season_of_timestamp_north": "winter",
  "season_of_timestamp_south": "summer"
}
shannon_entropy
FunctionDescriptionSourcesOutputArguments
shannon_entropyReturns the shannon entropy of the values in the given array.
the entropy is calculated as S = -sum(array_value * log(array_value), axis=axis).
This routine will normalize the values if they don’t sum to 1.
When use_base_2=false, the log base is e.

array_value
use_base_2

{
  "array_shannon_entropy": {
    "type": "numeric",
    "function": "shannon_entropy",
    "source": "array_source_name"
  }
},
{
  "array_shannon_entropy_base_2": {
    "type": "numeric",
    "function": "shannon_entropy",
    "source": "array_source_name",
    "args":[
        true
    ]
  }
}
<EXAMPLE INPUT>
{
  "array_source_name": [1/2,1/2]
}
<EXAMPLE OUTPUT>
{
  "array_shannon_entropy": 0.6931471805599453
},
{
  "array_shannon_entropy_base_2": 1
}
sort_and_concat_mapper
FunctionDescriptionSourcesOutputArguments
sort_and_concat_mapperA mapper directive which, for every (sequence) element in the source array, sorts
and concats as string using the given separator string.

array_value
(type: Sequence[Sequence])
Sequenceseparator

{
  "mapped_and_sorted_arrays": {
    "type": "string",
    "function": "sort_and_concat_mapper",
    "source": "source_name",
    "args": [
      ", "
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    [
      "John",
      "Bob",
      "Sharon",
      "Joe"
    ],
    [
      3,
      2,
      6,
      4
    ],
    [
      "dog",
      "cat",
      "monkey"
    ]
  ]
}
<EXAMPLE OUTPUT>
{
  "mapped_and_sorted_arrays": [
    [
      "Bob, Joe, John, Sharon"
    ],
    [
      "2, 3, 4, 6"
    ],
    [
      "cat, dog, monkey"
    ]
  ]
}
sort_array_of_objects_by_key
FunctionDescriptionSourcesOutputArguments
sort_array_of_objects_by_keyReturns the given array, sorted by the sort_key. If there a none value under
sort_key, we remove it before sorting.

array_value
(type: Sequence)
Optional[Sequence]sort_key
(type: str)

reverse

{
  "sorted_array": {
    "function": "sort_array_of_objects_by_key",
    "source": "source_name",
    "do_not_save": true,
    "args": [
      "age"
    ]
  },
  "first_in_sorted_array": {
    "function": "first_in_array",
    "source": "sorted_array",
    "do_not_save": true
  },
  "age_of_first_in_sorted_array": {
    "type": "numeric",
    "function": "field_value_from_object",
    "source": "first_in_sorted_array",
    "args": [
      "age"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "name": "John",
      "age": 20
    },
    {
      "name": "Bob",
      "age": 30
    },
    {
      "name": "Max",
      "age": 19
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "age_of_first_in_sorted_array": "19"
}
split_string_to_array
FunctionDescriptionSourcesOutputArguments
split_string_to_arrayA directive to split a string into an array.

string_value
(type: str)
Sequence[str]separator
(type: str)

array_wrapping_char
(type: str)

inner_string_wrapping_char
(type: str)
{
  "array_of_strings_after_split": {
    "type": "string",
    "function": "split_string_to_array",
    "source": "string_source_name",
    "args": [
      ", "
    ]
  }
}
{
  "string_source_name": "cat, dog, frog"
}
{
  "array_of_strings_after_split": ["cat","dog","frog"]
}
sub_array
FunctionDescriptionSourcesOutputArguments
sub_arrayReturns a sub array for the given array

array_value
(type: Sequence)
Optional[Sequence]start_index
(type: int)

end_index
(type: int)

default_value

{
  "sub_array_according_to_args": {
    "type": "string",
    "function": "sub_array",
    "source": "source_name",
    "args": [
      2,
      5
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [1,2,3,4,5,6,7,8]
}
<EXAMPLE OUTPUT>
{
  "sub_array_according_to_args": [3,4,5,6]
}
sum_array
FunctionDescriptionSourcesOutputArguments
sum_arrayReturns the sum of all values in the given array.

array_value
{
  "sum_of_array": {
    "type": "numeric",
    "function": "sum_array",
    "source": "array_source_name"
  }
}
<EXAMPLE INPUT>
{
  "array_source_name": [1,3,4]
}
<EXAMPLE OUTPUT>
{
  "sum_of_array": 8
}
sum_field_in_array
FunctionDescriptionSourcesOutputArguments
sum_field_in_arrayA directive which returns the sum of the values of a key in an array of
dict objects.

array_value
(type: Sequence[Mapping[str, Number]])
Numberkey
(type: str)
{
  "sum_of_values_by_key": {
    "type": "numeric",
    "function": "sum_field_in_array",
    "source": "source_name",
    "args": [
      "age"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "name": "John",
      "age": 13
    },
    {
      "name": "Bob",
      "age": 20
    },
    {
      "name": "Sharon",
      "age": 17
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "sum_of_values_by_key": 50
}
sum_formula
FunctionDescriptionSourcesOutputArguments
sum_formulaA directive for summing (dynamic) multiple numeric value sources. Not
suitable for numerical array sources.

numeric_values
(type: Sequence[Optional[Number]])
Number
{
  "sum_of_formula_sources": {
    "type": "numeric",
    "function": "sum_formula",
    "sources": [
      "source_name_1",
      "source_name_2",
      "source_name_3"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name_1": 5,
  "source_name_2": 6,
  "source_name_3": 10
}
<EXAMPLE OUTPUT>
{
  "sum_of_formula_sources": 21
}
timestamp_delta_days
FunctionDescriptionSourcesOutputArguments
timestamp_delta_daysA directive for calculating days delta of two timestamps (provided in
seconds/millis).

first_timestamp
(type: Optional[Number])

second_timestamp
(type: Optional[Number])
Optional[Number]default_value

{
  "delta_of_timestamps_in_days": {
    "type": "numeric",
    "sources": [
      "primary_source",
      "secondary_source"
    ],
    "function": "timestamp_delta_days"
  }
}
<EXAMPLE INPUT>
{
  "primary_source": 1613334600,
  "secondary_source": 1613248200
}
<EXAMPLE OUTPUT>
{
  "delta_of_timestamps_in_days": 1
}
to_int_string
FunctionDescriptionSourcesOutputArguments
to_int_stringA directive which gets a numeric (or numeric-convertible) value, and returns its
string representation, rounded, without decimal point.

original_value
str
{
  "numeric_to_int_field_name": {
    "type": "string",
    "function": "to_int_string",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
    "source_name": 4.55
}
<EXAMPLE OUTPUT>
{
    "numeric_to_int_field_name": "4"
}
use_value_mapping
FunctionDescriptionSourcesOutputArguments
use_value_mappingA directive which returns the mapped value of a key value.

key_value
(type: str)
mapper
(type: str)
{
  "fields": {
    "mapped_value_of_key": {
      "type": "string",
      "function": "use_value_mapping",
      "source": "key_source_name",
      "args": [
        "mapper_data_name"
      ]
    }
  },
  "data": {
    "second_set": {
      "A": "Dogs",
      "B": "Cats",
      "F": "Monkeys",
      "D": "Cows",
      "E": "Birds"
    }
  }
}
<EXAMPLE INPUT>
{
  "key_source_name": "B"
}
<EXAMPLE OUTPUT>
{
  "mapped_value_of_key": "Cats"
}
zip_to_us_state
FunctionDescriptionSourcesOutputArguments
zip_to_us_stateA directive which gets a US zip code value and returns the US state code it belongs
to. If the zip not found (or given an invalid value) - returns INVALID string.

string_or_int_value
(type: Union[str, int])
str
{
  "zip_code_to_state_field_name": {
    "type": "string",
    "function": "zip_to_us_state",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
    "source_name": 90401
}
<EXAMPLE OUTPUT>
{
    "zip_code_to_state_field_name": "CA"
}