Field Build Functions

These functions can be used in order to build and manipulate the fields

Suggest Edits
abs_value

Function

Description

Sources

Output

Arguments

abs_value

Returns the absolute value for a given numeric value.

numeric_value
(type: Number)

Number

{
  "abs_value_of_source": {
    "type": "numeric",
    "function": "abs_value",
    "source": "source_name"
  }
}

{
  "source_name": -6
}

{
  "abs_value_of_source": 6
}
all_checks

Function

Description

Sources

Output

Arguments

all_checks

A directive to check for multiple conditions. Receives a sequence of
values, returns True iff all the values return True.
Note that it is possible and advised to use this method over results of
other boolean field functions (such as check_existence, equal_const and
negative_check) in order to create complex boolean fields.
Not suitable for a boolean array source, only for multiple boolean sources.

If remove_if_null is set to True, if one of the boolean_values is null, this field
will be removed altogether.

boolean_values
(type: Sequence[Union[bool, NoneType]])

Union[bool, NoneType]

remove_if_null

{
  "all_sources_are_true": {
    "type": "boolean",
    "function": "all_checks",
    "sources": [
      "source_name_1",
      "source_name_2"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name_1": true,
  "source_name_2": true
}
<EXAMPLE OUTPUT>
{
  "all_sources_are_true": true
}
all_values_field_in_array

Function

Description

Sources

Output

Arguments

all_values_field_in_array

A directive which returns an array of values from all the occurrences of
a key in an array of dictionaries.

array_value
(type: Sequence[Mapping])

Sequence

key
(type: str)

{
  "values_from_objects_in_array": {
    "type": "numeric",
    "function": "all_values_field_in_array",
    "source": "source_name",
    "args": [
      "name"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "name": "john",
      "age": 13
    },
    {
      "name": "bob",
      "age": 20
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "values_from_objects_in_array": [
    "john",
    "bob"
  ]
}
array_intersection

Function

Description

Sources

Output

Arguments

array_intersection

A directive for getting the intersection of two arrays

first_array_value
(type: Sequence)

second_array_value
(type: Sequence)

Sequence

{
  "array_from_intersected_arrays": {
    "type": "string",
    "function": "array_intersection",
    "sources": [
      "source_name_1",
      "source_name_2"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name_1": [7,1,5,2,3,6],
  "source_name_2": [3,8,6,20,7]
}
<EXAMPLE OUTPUT>
{
  "array_from_intersected_arrays": [3,6,7]
}
array_length

Function

Description

Sources

Output

Arguments

array_length

A directive for creating a field according to the length of an array value.

array_value
(type: Sequence)

int

default_value

{
  "length_of_array": {
    "type": "numeric",
    "function": "array_length",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": [1,2,3,4,6,9,22]
}
<EXAMPLE OUTPUT>
{
  "length_of_array": 7
}
array_numbers

Function

Description

Sources

Output

Arguments

array_numbers

A directive for creating an array which consists only of the numeric values
of a given array.

array_value
(type: Sequence)

Sequence[Number]

default_value

{
  "numeric_values": {
    "type": "numeric",
    "function": "array_numbers",
    "source": "mixed_array"
  }
}
<EXAMPLE INPUT>
{
  "mixed_array": [1,5,3,true,"dog"]
}
<EXAMPLE OUTPUT>
{
  "numeric_values": [1,5,3]
}
array_top_value

Function

Description

Sources

Output

Arguments

array_top_value

A directive for creating a field according to the max numeric value of an array.
Note: given array must contain only numeric values.

array_value
(type: Sequence[Number])

Union[Number, NoneType]

default_value

{
  "top_value_of_array": {
    "type": "numeric",
    "function": "array_top_value",
    "source": "array_source"
  }
}
<EXAMPLE INPUT>
{
  "array_source": [1,5,32,3,2.8]
}
<EXAMPLE OUTPUT>
{
  "top_value_of_array": 32
}
array_tops_delta

Function

Description

Sources

Output

Arguments

array_tops_delta

A directive for creating a field according to the delta between the two largest
numeric values of an array.
Note: given array must contain only numeric values.

array_value
(type: Sequence[Number])

Union[Number, NoneType]

default_value

{
  "delta_of_array_top_values": {
    "type": "numeric",
    "function": "array_top_value",
    "source": "array_source"
  }
}
<EXAMPLE INPUT>
{
  "array_source": [32,1,5,12]
}
<EXAMPLE OUTPUT>
{
  "delta_of_array_top_values": 20
}
averages_delta

Function

Description

Sources

Output

Arguments

averages_delta

Returns the delta between the average value of the two arrays.

first_array_value
(type: Sequence[Union[Number, NoneType]])

second_array_value
(type: Sequence[Union[Number, NoneType]])

Union[Number, NoneType]

{
  "delta_of_averages_of_two_arrays": {
    "type": "numeric",
    "function": "averages_delta",
    "sources": [
      "source_name_1",
      "source_name_2"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name_1": [3,7,10,20],
  "source_name_2": [1,5,9]
}
<EXAMPLE OUTPUT>
{
  "delta_of_averages_of_two_arrays": 5
}
boolean_via_numeric

Function

Description

Sources

Output

Arguments

boolean_via_numeric

Return the boolean value of the numeric value that the given str describes.

value
(type: str)

bool

{
  "boolean_value_field": {
    "type": "boolean",
    "function": "boolean_via_numeric",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": "0"
}
<EXAMPLE OUTPUT>
{
  boolean_value_field": false
}
clean_null_str

Function

Description

Sources

Output

Arguments

clean_null_str

A workaround directive to nullify "null" strings coming by mistake from the user.

actual_value

{
  "nullified_field_name": {
    "type": "string",
    "source": "source_name",
    "function": "clean_null_str"
  },
  "not_nullified_field_name": {
    "type": "string",
    "source": "source_name_2",
    "function": "clean_null_str"
}
<EXAMPLE INPUT>
{
  "source_name": "null",
  "source_name_2": "dog"
}
<EXAMPLE OUTPUT>
{
  "nullified_field_name": null,
  "not_nullified_field_name": "dog"
}
concat

Function

Description

Sources

Output

Arguments

concat

A directive which takes a sequence field, and returns its values concatenated
as a string.

actual_value
(type: Union[Mapping, Sequence])

Union[str, NoneType]

{
  "concatenated_array": {
    "type": "string",
    "function": "concat",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    "John",
    "Bob",
    "Sharon",
    "Joe"
  ]
}
<EXAMPLE OUTPUT>
{
  "concatenated_array": [
    "John_Bob_Sharon_Joe"
  ]
}
concat_inner_array

Function

Description

Sources

Output

Arguments

concat_inner_array

Returns an array concatenation of all values in the the inner_array_key in the
objects originally found in the source array.

outer_array_value
(type: Sequence[Mapping])

Sequence

inner_array_key
(type: str)

{
  "array_of_concatenated_arrays": {
    "type": "string",
    "function": "concat_inner_array",
    "source": "source_name",
    "args": [
      "a"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "a": [1,2,3],
      "b": ["a","b","c"]
    },
    {
      "a": [4,5,6],
      "b": ["d","e","f"]
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "array_of_concatenated_arrays": ["1","2","3","4","5","6"]
}
concat_strings

Function

Description

Sources

Output

Arguments

concat_strings

A directive for concatenating all the given string/strings sequences sources.
If sources are strings sequences, concat each index of the sources.

If values contains str arrays, in case of a None arg in one of the arrays, the
returned value in the corresponding index will be None.

values
(type: Sequence[Union[str, Sequence[str]]])

Union[str, Sequence[str]]

separator
(type: str)

{
  "concatenated_string": {
    "type": "string",
    "function": "concat_strings",
    "sources": [
      "source_name_1",
      "source_name_2"
    ],
    "args": [
      ", "
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name_1": "Jack",
  "source_name_2": "Black"
}
<EXAMPLE OUTPUT>
{
  "concatenated_string": "Jack, Black"
}
concat_strings_recursive_arrays

Function

Description

Sources

Output

Arguments

concat_strings_recursive_arrays

A directive for generating a sequence of strings, concatanated from keys of objects
within arrays. It currently has a very specific usage: It expects a source which is
an array of objects, then a sequence of keys, the first of which should be a key to
look for a string value, the second should be a key to look for a new array of
objects, the third is again a key for a string value within this new array, and the
forth is again a key for looking for a new array of objects, and so on.

top_array_value
(type: Sequence[Union[Mapping, NoneType]])

Union[str, NoneType]

internal_keys
(type: Sequence[str])

separator
(type: str)

{
  "concatenated_strings_in_recursive_arrays": {
    "type": "string",
    "source": "source_name",
    "function": "concat_strings_recursive_arrays",
    "args": [
      [
        "feature_id",
        "attributes",
        "attribute_1",
        "values",
        "unique_id"
      ]
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "feature_id": 3833,
      "attributes": [
        {
          "attribute_1": 1242,
          "values": [
            {
              "unique_id": 12
            }
          ]
        }
      ]
    },
    {
      "feature_id": 2336,
      "attributes": [
        {
          "attribute_1": 923,
          "values": [
            {
              "unique_id": 132
            }
          ]
        },
        {
          "attribute_1": 1759,
          "values": [
            {
              "unique_id": 243
            }
          ]
        }
      ]
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "concatenated_strings_in_recursive_arrays": [
    "3833_1242_12",
    "2336_923_132",
    "2336_1759_243"
  ]
}
const_is_in_array

Function

Description

Sources

Output

Arguments

const_is_in_array

Return true if the given array contains the given constant value, false otherwise.

array_value
(type: Sequence)

bool

value

{
  "does_source_contains_const": {
    "type": "boolean",
    "function": "const_is_in_array",
    "source": array_source,
    "args": [
      "blue"
    ]
  }
}
<EXAMPLE INPUT>
{
  "array_source": [red, yellow, ,blue, orange]
}
<EXAMPLE OUTPUT>
{
  "does_source_contains_const": true
}
convert_to_float

Function

Description

Sources

Output

Arguments

convert_to_float

Convert string to float safely (if it's already int/float - no conversion needed).
In case the given value is not representing a number the return value is None.
Values in null_values are replaced with null.

original_value
(type: Union[Number, NoneType])

Number

null_values

count_characters

Function

Description

Sources

Output

Arguments

count_characters

A directive which gets a string value or array, and returns the amount of
characters (if array - for each element).

string_or_array_value

int

{
  "count_characters_field_name": {
    "type": "int",
    "function": "count_characters",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
    "source_name": ["Hi there, my name is John.", "Hello"]
}
<EXAMPLE OUTPUT>
{
    "count_characters_field_name": [26, 5]
}
count_occurrences_in_array

Function

Description

Sources

Output

Arguments

count_occurrences_in_array

A directive that, given an array field, counts the amount of items in that array in
which the value for given key is equal to the given inner_value.

array_value
(type: Sequence[Mapping])

int

key
(type: str)

inner_value

default_value

{
  "number_of_value_occurrences": {
    "type": "numeric",
    "function": "count_occurrences_in_array",
    "source": "source_name",
    "args": [
      "name",
      "Bob"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "name": "John",
      "age": 13
    },
    {
      "name": "Bob",
      "age": 20
    },
    {
      "name": "Sharon",
      "age": 17
    },
    {
      "name": "Bob",
      "age": 10
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "number_of_value_occurrences": 2
}
count_recursively

Function

Description

Sources

Output

Arguments

count_recursively

A directive which counts all internal elements in the given array, recursively.

array_value
(type: Sequence)

int

{
  "number_of_elements_recursively": {
    "type": "numeric",
    "function": "count_recursively",
    "source": "source_name"
  }
}

{
  "source_name": [
    "bob",
    [
      "dog",
      "cat"
    ],
    "sharon"
  ]
}

{
  "number_of_elements_recursively": 4
}
count_words

Function

Description

Sources

Output

Arguments

count_words

A directive which gets a string value or array, and returns the amount of
words (if array - for each element).
The amount of words is calculated based on the standard python split method
without args, i.e. words are separated by any number and form of whitespace
chars.

string_or_array_value

int

{
  "count_words_field_name": {
    "type": "int",
    "function": "count_words",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
    "source_name": ["Hi there, my name is John.", "Hello"]
}
<EXAMPLE OUTPUT>
{
    "count_words_field_name": [6, 1]
}
day_of_week_from_timestamp_seconds

Function

Description

Sources

Output

Arguments

day_of_week_from_timestamp_seconds

Returns the day of week (UTC) from the given timestamp.

timestamp_value
(type: Number)

{
  "day_of_timestamp": {
    "type": "string",
    "function": "day_of_week_from_timestamp_seconds",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": 1610569800
}
<EXAMPLE OUTPUT>
{
  "day_of_timestamp": "Wednesday"
}
dedup_array_of_objects_by_key

Function

Description

Sources

Output

Arguments

dedup_array_of_objects_by_key

Returns the given array, deduped by the values in the given key.

array_value
(type: Sequence)

Sequence

dedup_key
(type: str)

{
  "deduped_sub_array": {
    "type": "string",
    "function": "dedup_array_of_objects_by_key",
    "source": "source_name",
    "args": [
      "name"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "name": "John",
      "age": 20
    },
    {
      "name": "Bob",
      "age": 30
    },
    {
      "name": "Max",
      "age": 19
    },
    {
      "name": "Bob",
      "age": 10
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "deduped_sub_array": [
    {
      "name": "John",
      "age": 20
    },
    {
      "name": "Bob",
      "age": 30
    },
    {
      "name": "Max",
      "age": 19
    }
  ]
}
delta

Function

Description

Sources

Output

Arguments

delta

A directive for calculating delta of two values.

first_value
(type: Union[Number, NoneType])

second_value
(type: Union[Number, NoneType])

Union[Number, NoneType]

default_value

{
  "delta_between_values": {
    "type": "numeric",
    "sources": [
      "numeric_source_1",
      "numeric_source_2"
    ],
    "function": "delta"
  }
}
<EXAMPLE INPUT>
{
  "numeric_source_1": 50,
  "numeric_source_2": 2
}
<EXAMPLE OUTPUT>
{
  "delta_between_values": 48
}
delta_between_first_and_second

Function

Description

Sources

Output

Arguments

delta_between_first_and_second

Returns the delta between the max value and the second max value from a list of
(dynamic) multiple numeric value sources. Not suitable for numerical array sources.

numeric_values
(type: Sequence[Union[Number, NoneType]])

Union[Number, NoneType]

{
  "delta_of_two_highest_values": {
    "type": "numeric",
    "function": "delta_between_first_and_second",
    "sources": [
      "source_name_1",
      "source_name_2",
      "source_name_3"
    ]
  }
}

{
  "source_name_1": 5,
  "source_name_2": 10,
  "source_name_3": 7
}

{
  "delta_of_two_highest_values": 3
}
delta_from_const

Function

Description

Sources

Output

Arguments

delta_from_const

A directive for calculating delta between a given number and a const.

first_value
(type: Union[Number, NoneType])

Union[Number, NoneType]

const_to_subtract
(type: Number)

default_value

{
  "delta_between_source_and_const": {
    "type": "numeric",
    "function": "delta_from_const",
    "source": "numeric_source_name",
    "args": [
      10
    ]
  }
}
<EXAMPLE INPUT>
{
  "numeric_source_name": 50
}
<EXAMPLE OUTPUT>
{
  "delta_between_source_and_const": 40
}
divide

Function

Description

Sources

Output

Arguments

divide

A directive for dividing one value by another.

numerator_value
(type: Union[Number, NoneType])

denominator_value
(type: Union[Number, NoneType])

Union[Number, NoneType]

{
  "quotient_of_division": {
    "type": "numeric",
    "function": "divide",
    "sources": [
      "numeric_source_name_1",
      "numeric_source_name_2"
    ]
  }
}
<EXAMPLE INPUT>
{
  "numeric_source_name_1": 50,
  "numeric_source_name_2": 10
}
<EXAMPLE OUTPUT>
{
  "quotient_of_division": 5
}
divide_by_const

Function

Description

Sources

Output

Arguments

divide_by_const

A directive for dividing one value by a const.

numerator_value
(type: Union[Number, NoneType])

Union[Number, NoneType]

denominator
(type: Union[Number, NoneType])

{
  "quotient_of_division_with_const": {
    "type": "numeric",
    "function": "divide_by_const",
    "source": "numeric_source_name",
    "args": [
      "5"
    ]
  }
}
<EXAMPLE INPUT>
{
  "numeric_source_name_": 50
}
<EXAMPLE OUTPUT>
{
  "quotient_of_division_with_const": 10
}
dynamic_field_value_from_object

Function

Description

Sources

Output

Arguments

dynamic_field_value_from_object

Returns the value of a dynamically chosen field within an object.

dict_object_value
(type: Mapping)

field_name_value

{
  "value_of_dynamic_field": {
    "type": "string",
    "function": "dynamic_field_value_from_object",
    "sources": [
      "source_name",
      "field_key_name"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": {
    "name": "Bob",
    "age": 23,
    "city": "New York"
  },
  "field_key_name": "city"
}
<EXAMPLE OUTPUT>
{
  "value_of_dynamic_field": "New York"
}
empty_to_null

Function

Description

Sources

Output

Arguments

empty_to_null

A directive which is just like the identity function for strings, only it returns
"null" for empty strings.

original_value
(type: str)

str

{
  "example_field_name": {
    "type": "string",
    "function": "empty_to_null",
    "sources": [
      "source_name"
    ]
  },
  "empty_field_name": {
    "type": "string",
    "function": "empty_to_null",
    "sources": [
      "source_name_2"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": "dog",
  "source_name_2": ""
}
<EXAMPLE OUTPUT>
{
  "example_field_name": "dog",
  "empty_field_name": null
}
equality_check

Function

Description

Sources

Output

Arguments

equality_check

A directive to check for equality between two values.

first_value

second_value

Union[bool, NoneType]

{
  "is_equale": {
    "type": "boolean",
    "function": "equality_check",
    "sources": [
      "source_name_1",
      "source_name_2"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name_1": 10,
  "source_name_2": 10
}
<EXAMPLE OUTPUT>
{
  "is_equale": true
}
equality_check_const

Function

Description

Sources

Output

Arguments

equality_check_const

A directive to check for equality between a value and a constant value.

value

const

{
  "is_equal_to_const": {
    "type": "boolean",
    "function": "equality_check_const",
    "source": "source_name",
    "args": [
      10
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": 10
}
<EXAMPLE OUTPUT>
{
  "is_equal_to_const": true
}
existence_check

Function

Description

Sources

Output

Arguments

existence_check

A directive to check for an existence of a value.

value

bool

non_existent_value

{
  "value_exists": {
    "type": "boolean",
    "function": "existence_check",
    "source": "source_name"
  },
  "value_exists_2": {
    "type": "boolean",
    "function": "existence_check",
    "source": "source_name_2"
  },
  "value_exists_3": {
    "type": "boolean",
    "function": "existence_check",
    "source": "source_name_3"
  }
}
<EXAMPLE INPUT>
{
  "source_name": "John",
  "source_name_2": null
}
<EXAMPLE OUTPUT>
{
  "value_exists": true,
  "value_exists_2": false,
  "value_exists_3": false
}
field_delta

Function

Description

Sources

Output

Arguments

field_delta

A directive to get a field delta between the first and second values of a
specific key in an array field.

NOTE: If no second value is found, we use zero
instead.

array_value
(type: Sequence[Mapping[str, Number]])

Union[Number, NoneType]

key
(type: str)

sort
(type: bool)

reversed_sort
(type: bool)

default_value

{
  "delta_of_first_and_second_values": {
    "type": "string",
    "function": "field_delta",
    "source": "source_name",
    "args": [
      "age"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "name": "John",
      "age": 20
    },
    {
      "name": "Bob",
      "age": 13
    },
    {
      "name": "Sharon",
      "age": 17
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "delta_of_first_and_second_values": 7
}
field_value_from_object

Function

Description

Sources

Output

Arguments

field_value_from_object

Returns the value of a field within an object

dict_object_value
(type: Mapping)

field_name

{
  "city_name_from_obj": {
    "type": "string",
    "function": "field_value_from_object",
    "source": "source_name",
    "args": [
      "city"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": {
    "name": "Bob",
    "age": 23,
    "city": "New York"
  }
}
<EXAMPLE OUTPUT>
{
  "city_name_from_obj": "New York"
}
filter_according_to_boolean_array

Function

Description

Sources

Output

Arguments

filter_according_to_boolean_array

Returns the original array, filtered according to a same-length array of boolean
values.

original_array_value

boolean_array_value

should_invert

{
  "filtered_array": {
    "type": "string",
    "function": "filter_according_to_boolean_array",
    "sources": [
      "array_source_name",
      "boolean_array_source_name"
    ]
  }
}
<EXAMPLE INPUT>
{
  "array_source_name": [
    "cat",
    "cow",
    "dog",
    "sheep"
  ],
  "boolean_array_source_name": [
    true,
    true,
    false,
    true
  ]
}
<EXAMPLE OUTPUT>
{
  "filtered_array": [
    "cat",
    "cow",
    "sheep"
  ]
}
filter_array

Function

Description

Sources

Output

Arguments

filter_array

Returns a sub array for the given array of objects, according to whether the value
in the given key of the object is from the possible values parameter.

If the bool_evaluation parameter is set, will cast the value of the given key to
bool before checking.

if the key parameter is None, will evaluate the given values in the array directly,
and will assume they are not dictionaries.

The returned value is still an array of objects, and should therefore never be
"saved" in the schema (use "do_not_save": true).

array_value
(type: Sequence[Mapping])

Sequence

key
(type: str)

possible_values
(type: Sequence)

bool_evaluation
(type: bool)

{
  "filtered_sub_array": {
    "type": "string",
    "function": "filter_array",
    "source": "source_name",
    "args": [
      "name",
      [
        "John",
        "Bob"
      ]
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "name": "John",
      "age": 13
    },
    {
      "name": "Bob",
      "age": 20
    },
    {
      "name": "Sharon",
      "age": 17
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "filtered_sub_array": [
    {
      "name": "John",
      "age": 13
    },
    {
      "name": "Bob",
      "age": 20
    }
  ]
}
filter_array_by_blacklist

Function

Description

Sources

Output

Arguments

filter_array_by_blacklist

Returns a sub array for the given array of objects, according to whether the value
in the given key of the object is not from the blacklisted_values.

The returned value is still an array of objects, and should therefore never be
"saved" in the schema (use "do_not_save": true).

array_value
(type: Sequence[Mapping])

Sequence[Mapping]

key
(type: str)

blacklisted_values
(type: Sequence)

{
  "filtered_sub_array": {
    "type": "string",
    "function": "filter_array_by_blacklist",
    "source": "source_name",
    "args": [
      "name",
      [
        "Bob",
        "Sharon"
      ]
    ],
    "do_not_use": true
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "name": "John",
      "age": 14
    },
    {
      "name": "Bob",
      "age": 33
    },
    {
      "name": "Max",
      "age": 45
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "filtered_sub_array": [
    {
      "name": "John",
      "age": 14
    },
    {
      "name": "Max",
      "age": 45
    }
  ]
}
filter_array_by_valid_ranges

Function

Description

Sources

Output

Arguments

filter_array_by_valid_ranges

Returns a sub array for the given array of objects, according to whether the value
in the given key of the object is in at least one of the given valid_ranges.

If the given key is None, then this directive assumes the array isn't of objects,
but of the actual numbers needed to be filtered according to the ranges.

valid_ranges should be an array of pairs, which represent the minimum (inclusive)
and maximum (exclusive) for each range.
null (pythonic: None) can be used to suggest no minimum or no maximum.

array_value
(type: Sequence[Union[Mapping[str, Number], Number]])

Sequence

key
(type: str)

valid_ranges
(type: Sequence[Sequence[Number]])

{
  "filtered_sub_array": {
    "function": "filter_array_by_valid_ranges",
    "source": "source_name",
    "args": [
      "age",
      [
        [12,18],
        [30,35]
      ]
    ],
    "do_not_save": true
  },
  "filtred_sub_array_length": {
    "type": "numeric",
    "function": "array_length",
    "source": "filtered_sub_array"
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "name": "John",
      "age": 13
    },
    {
      "name": "Bob",
      "age": 20
    },
    {
      "name": "Sharon",
      "age": 17
    },
    {
      "name": "Bob",
      "age": 10
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "filtered_sub_array": [
    {
      "name": "John",
      "age": 13
    },
    {
      "name": "Sharon",
      "age": 17
    }
  ],
  "filtred_sub_array_length": 2
}
filter_array_static_key_and_dynamic_value

Function

Description

Sources

Output

Arguments

filter_array_static_key_and_dynamic_value

Returns a sub array for the given array of objects, according to whether the
value in the given key of the object is equal to the compared value given
dynamically as a source.

The returned value is still an array of objects, and should therefore never
be "saved" in the schema (use "do_not_save": true).

array_value
(type: Sequence[Mapping])

compared_value

Sequence

key
(type: str)

{
  "dynamically_filtered_sub_array": {
    "type": "string",
    "function": "filter_array_static_key_and_dynamic_value",
    "sources": [
      "array_value_source_name",
      "dynamic_value_source_name"
    ]
    "args": [
      "name"
    ]
  }
}
<EXAMPLE INPUT>
{
  "array_value_source_name": [
    {
      "name": "John",
      "age": 13
    },
    {
      "name": "Bob",
      "age": 20
    },
    {
      "name": "Sharon",
      "age": 17
    }
  ],
  "dynamic_value_source_name": "John"
}
<EXAMPLE OUTPUT>
{
  "dynamically_filtered_sub_array": [
    {
      "name": "John",
      "age": 13
    }
  ]
}
filter_duplicates

Function

Description

Sources

Output

Arguments

filter_duplicates

A directive for getting a new array, with filtered duplicates.
If the given value is not an array - return the value as is.

array_value
(type: Sequence)

Sequence

{
  "filtered_sub_array": {
    "type": "string",
    "function": "filter_duplicates",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": [1,2,2,2,3,4]
}
<EXAMPLE OUTPUT>
{
  "filtered_sub_array": [1,2,3,4]
}
first_in_array

Function

Description

Sources

Output

Arguments

first_in_array

Returns the first value in an array.

array_value
(type: Sequence)

default_value

{
  "first_value_in_array": {
    "type": "string",
    "function": "first_in_array",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    "John",
    "Bob",
    "Max"
  ]
}
<EXAMPLE OUTPUT>
{
  "first_value_in_array": "John"
}
flatten_array_recursive

Function

Description

Sources

Output

Arguments

flatten_array_recursive

A directive to flatten an array with nested arrays to one level.

array_value
(type: Sequence)

Sequence

{
  "flattened_array": {
    "type": "string",
    "function": "flatten_array_recursive",
    "source": "array_source_name"
  }
}

{
  "array_source_name": [
    "bob",
    [
      "dog",
      "cat"
    ],
    "sharon"
  ]
}

{
  "flattened_array": [
    "bob",
    "dog",
    "cat",
    "sharon"
  ]
}
format_string

Function

Description

Sources

Output

Arguments

format_string

A directive for string formatting.
If some of the input values are arrays, this function will output an array for all
the possible combinations of the inputs.

input_values
(type: Sequence)

str

template
(type: str)

{
  "formatted_string_field": {
    "type": "string",
    "function": "format_string",
    "sources": [
      "source_name_1",
      "source_name_2"
    ],
    "args": [
      "{} and {} are co-workers"
    ]
  }
}
<EXAMPLE INPUT>
{
    "source_name_1": "John",
    "source_name_2": "Bob
}
<EXAMPLE OUTPUT>
{
    "formatted_string_field": "John and Bob are co-workers"
}
from_array_index

Function

Description

Sources

Output

Arguments

from_array_index

Returns the value in the given index in the given array.

array_value

index

{
  "value_of_index": {
    "type": "numeric",
    "function": "from_array_index",
    "source": array_source,
    "args": [
      2
    ]
  }
}
<EXAMPLE INPUT>
{
  "array_source": [10,12,6,3,11]
}
<EXAMPLE OUTPUT>
{
  "value_of_index": 6
}
get_dict_key_on_check

Function

Description

Sources

Output

Arguments

get_dict_key_on_check

Return the value of a dict's key according to a check of check_key being
equal to check_value (static value supplied from the configuration as an
argument)

dict_objects_array_value
(type: Sequence[Mapping])

key
(type: str)

check_key
(type: str)

check_value

compare_as_floats
(type: bool)

{
  "value_of_key_by_args": {
    "type": "string",
    "source": "source_name",
    "function": "get_dict_key_on_check",
    "args": [
      "b",
      "a",
      3
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "a": 1,
      "b": 2
    },
    {
      "a": 3,
      "b": 4
    },
    {
      "a": 5,
      "b": 6
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "value_of_key_by_args": "4"
}
get_dict_key_on_dynamic_check

Function

Description

Sources

Output

Arguments

get_dict_key_on_dynamic_check

Return the value of a dict's key according to a check of check_key being
equal to dynamic value supplied from a source.

check_value

dict_objects_array_value
(type: Sequence[Mapping])

key
(type: str)

check_key
(type: str)

compare_as_floats
(type: bool)

{
  "value_of_key_by_dynamic_check": {
    "type": "string",
    "sources": [
      "check_value_source",
      "source_name"
    ],
    "function": "get_dict_key_on_dynamic_check",
    "args": [
      "b",
      "a"
    ]
  }
}
<EXAMPLE INPUT>
{
  "check_value_source": 3,
  "source_name": [
    {
      "a": 1,
      "b": 2
    },
    {
      "a": 3,
      "b": 4
    },
    {
      "a": 5,
      "b": 6
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "value_of_key_by_dynamic_check": "4"
}
get_nth_highest_key

Function

Description

Sources

Output

Arguments

get_nth_highest_key

Returns the key of the nth highest value in a given dict.

dict_value
(type: Mapping[str, Number])

n
(type: int)

{
  "key_of_nth_highest_value": {
    "type": "numeric",
    "function": "get_nth_highest_key",
    "source": "source_name",
    "args": [
      1
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": {
    "average_age_1": 34,
    "average_age_2": 33,
    "average_age_3": 42
  }
}
<EXAMPLE OUTPUT>
{
  "key_of_nth_highest_value": "average_age_2"
}
get_nth_highest_value

Function

Description

Sources

Output

Arguments

get_nth_highest_value

Returns the nth highest value of all values in a given dict.

dict_value
(type: Mapping[str, Number])

n
(type: int)

{
  "nth_highest_value_of_keys": {
    "type": "numeric",
    "function": "get_nth_highest_value",
    "source": "source_name",
    "args": [
      1
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": {
    "average_age_1": 34,
    "average_age_2": 33,
    "average_age_3": 42
  }
}
<EXAMPLE OUTPUT>
{
  "nth_highest_value_of_keys": 33
}
get_top_key

Function

Description

Sources

Output

Arguments

get_top_key

Returns the key with the highest number in its value, from dictionary which maps
strings to numbers.

dict_value
(type: Mapping[str, Number])

{
   "key_of_highest_value": {
     "type": "string",
     "function": "get_top_key",
     "source": "source_name"
   }
 }
 <EXAMPLE INPUT>
 {
   "source_name": {
     "key_1": 4,
     "key_2": 12,
     "key_3": 7,
     "key_4": 22
   }
 }
 <EXAMPLE OUTPUT>
{
   "key_of_highest_value": "key_4"
 }
get_top_value

Function

Description

Sources

Output

Arguments

get_top_value

Returns the highest value of all values in a given dict.

dict_value
(type: Mapping[str, Number])

{
  "highest_value_of_keys": {
    "type": "numeric",
    "function": "get_top_value",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": {
    "key_1": 4,
    "key_2": 12,
    "key_3": 7,
    "key_4": 22
  }
}
<EXAMPLE OUTPUT>
{
  "highest_value_of_keys": 22
}
hour_of_day_from_timestamp_seconds

Function

Description

Sources

Output

Arguments

hour_of_day_from_timestamp_seconds

Returns the hour of the day (UTC) from the given timestamp.

timestamp_value
(type: Number)

{
  "hour_of_timestamp": {
    "type": "numeric",
    "function": "hour_of_day_from_timestamp_seconds",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": 1610569800
}
<EXAMPLE OUTPUT>
{
  "hour_of_timestamp": 20
}
identity

Function

Description

Sources

Output

Arguments

identity

A default identity field build function.

value

import_from_other_context_class

Function

Description

Sources

Output

Arguments

import_from_other_context_class

Imports the value of the field from other context class.
Using the given foreign id value, fetch a corresponding existing CRC and returns
the values in the foreign field.

The string_to_replace_dot param allows to replace the dot character in the
foreign_ids to a provided string.

foreign_ids
(type: Union[str, list, tuple])

foreign_field
(type: str)

foreign_context_class
(type: str)

foreign_id_prefix

string_to_replace_dot
(type: str)

{
  "imported_value": {
    "type": "string,
    "function": "import_from_other_context_class",
    "source": "some_id",
    "args": [
      "name_of_field_in_other_context_class",
      "NAME_OF_OTHER_CONTEXT_CLASS"
    ]
  }
}

{
  "some_id": "532n-kl52-gs53-weg5"
}

{
  "imported_value": "value_of_field_in_other_context_class"
}
is_in_dynamic_set

Function

Description

Sources

Output

Arguments

is_in_dynamic_set

A directive to check if a value is in a set of values.

value

set_of_values
(type: Sequence)

Union[bool, NoneType]

{
  "fields": {
    "value_is_in_set": {
      "type": "boolean",
      "function": "is_in_dynamic_set",
      "sources": ["source_name", "source_name_2"],
    },
  },
}
<EXAMPLE INPUT>
{
  "source_name": "John",
  "source_name_2": ["John", "Bob", "Frank", "Paul"]
}
<EXAMPLE OUTPUT>
{
  "value_is_in_set": true
}
is_in_other_array_mapper

Function

Description

Sources

Output

Arguments

is_in_other_array_mapper

A mapper directive for boolean values according to whether each value is in
another array or not

first_array_value
(type: Sequence)

second_array_value
(type: Sequence)

Sequence[bool]

{
  "boolean_values_according_to_sources": {
    "type": "boolean",
    "function": "is_in_other_array_mapper",
    "sources": [
      "source_name_1",
      "source_name_2"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name_1": [
    "John",
    "Bob",
    "Sharon",
    "Joe"
  ],
  "source_name_2": [
    "Max",
    "Bob",
    "Peter",
    "Joe"
  ]
}
<EXAMPLE OUTPUT>
{
  "boolean_values_according_to_sources": [
    false,
    true,
    false,
    true
  ]
}
is_in_set

Function

Description

Sources

Output

Arguments

is_in_set

A directive to check if a value is in a given values set.

value

Union[bool, NoneType]

set_key
(type: str)

treat_missing_as_false
(type: bool)

{
  "fields": {
    "value_is_in_set": {
      "type": "boolean",
      "function": "is_in_set",
      "source": "source_name",
      "args": [
        "set_name"
      ]
    },
  },
  "data": {
    "set_name": [
      "John",
      "Bob",
      "Frank",
      "Paul"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": "John"
}
<EXAMPLE OUTPUT>
{
  "value_is_in_set": true
}
is_monotone

Function

Description

Sources

Output

Arguments

is_monotone

A directive which checks if the given array is monotone.

array_value
(type: Sequence)

bool

ascending

use_weak

{
  "array_is_monotone": {
    "type": "boolean",
    "function": "is_monotone",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": [1,3,4,6,12]
}
<EXAMPLE OUTPUT>
{
  "array_is_monotone": true
}
keep_only_digits

Function

Description

Sources

Output

Arguments

keep_only_digits

A directive which takes a string (or array of strings) and returns that string only
with its digits, stripping away all non-digit characters.

actual_value
(type: Union[str, Sequence[str]])

{
  "digits_from_string": {
    "type": "string",
    "source": "source_name",
    "function": "keep_only_digits"
  }
}
<EXAMPLE INPUT>
{
  "source_name": "fe223"
}
<EXAMPLE OUTPUT>
{
  "digits_from_string": "223"
}
last_in_array

Function

Description

Sources

Output

Arguments

last_in_array

Returns the last value in an array.

array_value
(type: Sequence)

default_value

{
  "last_value_in_array": {
    "type": "string",
    "function": "last_in_array",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    "John",
    "Bob",
    "Max"
  ]
}
<EXAMPLE OUTPUT>
{
  "last_value_in_array": "Max"
}
max_delta

Function

Description

Sources

Output

Arguments

max_delta

A directive which returns the delta between the min and max numbers in a given
source array.

array_value
(type: Sequence[Number])

Number

{
  "delta_of_highest_and_lowest_values": {
    "type": "numeric",
    "function": "max_delta",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": [1,3,12,6,4]
}
<EXAMPLE OUTPUT>
{
  "delta_of_highest_and_lowest_values": 11
}
max_value

Function

Description

Sources

Output

Arguments

max_value

Returns the max value from list of (dynamic) multiple numeric value sources.
Not suitable for numerical array sources.

numeric_values
(type: Sequence[Union[Number, NoneType]])

Union[Number, NoneType]

{
  "highest_value_from_sources": {
    "type": "numeric",
    "function": "max_value",
    "sources": [
      "source_name_1",
      "source_name_2",
      "source_name_3"
    ]
  }
}

{
  "source_name_1": 5,
  "source_name_2": 10,
  "source_name_3": 7
}

{
  "highest_value_from_sources": 10
}
month_from_timestamp

Function

Description

Sources

Output

Arguments

month_from_timestamp

A directive to get the month of the year of a given timestamp.

timestamp_value
(type: Number)

Number

{
  "month_in_number": {
    "type": "numeric",
    "function": "month_from_timestamp",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": 1610569800
}
<EXAMPLE OUTPUT>
{
  "month_in_number": 1
}
multiply

Function

Description

Sources

Output

Arguments

multiply

Returns the product of all the input values. If the inputs are arrays, then they
have to be of the same length, and the result will be an array of that same length
with all the products for each index.

input_values

{
  "product_of_multiplication": {
    "type": "numeric",
    "sources": [
      "numeric_source_1",
      "numeric_source_2"
    ],
    "function": "multiply"
  }
}
<EXAMPLE INPUT>
{
  "numeric_source_1": 5,
  "numeric_source_2": 10
}
<EXAMPLE OUTPUT>
{
  "product_of_multiplication": 50
}
multiply_by_const

Function

Description

Sources

Output

Arguments

multiply_by_const

A directive for multiplying a value by a constant given as an arg.

TODO(adi): We assume here that zero means null, we should split this

function to multiply_by_const_no_zero.

original_value
(type: Union[Number, NoneType])

Number

multiplier

{
  "product_of_multiplication": {
    "type": "numeric",
    "source": "numeric_source",
    "function": "multiply_by_const",
    "args": [
      "10"
    ]
  }
}
<EXAMPLE INPUT>
{
  "numeric_source": 5
}
<EXAMPLE OUTPUT>
{
  "product_of_multiplication": 50
}
negative_check

Function

Description

Sources

Output

Arguments

negative_check

A directive to check for the negative value of the specified value.
Returns True if the value is False.

boolean_value
(type: bool)

Union[bool, NoneType]

treat_missing_as_false
(type: bool)

{
  "value_is_negative": {
    "type": "boolean",
    "function": "negative_check",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
  "source_name": false
}
<EXAMPLE OUTPUT>
{
  "value_is_negative": true
}
object_values_as_array

Function

Description

Sources

Output

Arguments

object_values_as_array

Returns the values of a dict as an array. Fields using this function should never be
saved to crc (use "do_not_save": true in field config).

TODO(itai): Create a decorator which tags such functions and doesn't allow them to
be saved.

dict_value
(type: Union[Mapping, NoneType])

Sequence

{
  "array_of_values_from_object": {
    "type": "string",
    "function": "object_values_as_array",
    "source": "source_name",
    "do_not_save": true
  }
}
<EXAMPLE INPUT>
{
  "source_name": {
    "name": "John",
    "age": 30,
    "city": "New York"
  }
}
<EXAMPLE OUTPUT>
{
  "array_of_values_from_object": [
    "John",
    30,
    "New York"
  ]
}
parse_datetime_to_timestamp_seconds

Function

Description

Sources

Output

Arguments

parse_datetime_to_timestamp_seconds

A directive for parsing a datetime object into epoch timestamp seconds.
If the allow_timestamps flag is set to true, the directive will not reject plain
timestamps and will return a given timestamp in seconds. Can work on lists as well.

value
(type: Union[str, int, Sequence[str], Sequence[int]])

Union[Number, Sequence[Number]]

python_datetime_formats
(type: Union[str, Sequence[str]])

allow_timestamps
(type: bool)

{
  "timestamp_from_date": {
    "type": "numeric",
    "source": "datetime_object_source",
    "function": "parse_datetime_to_timestamp_seconds",
    "args": [
      ["%m/%d/%Y, %H:%M:%S", "%H:%M:%S-%m-%d-%Y"]
    ]
  }
}
<EXAMPLE INPUT>
{
  "datetime_object_source": "02/14/2021, 20:30:00"
}
<EXAMPLE OUTPUT>
{
  "timestamp_from_date": 1613334600
}
<EXAMPLE INPUT>
{
  "datetime_object_source": ["02/14/2021, 20:30:00", "02/18/2021, 08:30:00"]
}
<EXAMPLE OUTPUT>
{
  "timestamp_from_date": [1613334600, 1613637000]
}
<EXAMPLE INPUT>
{
  "datetime_object_source": "1613334600000",
  "allow_timestamps": "true"
}
<EXAMPLE OUTPUT>
{
  "timestamp_from_date": 1613334600
}
powerset_concat

Function

Description

Sources

Output

Arguments

powerset_concat

A directive which returns the powerset of the given array, with its values
concatanated to one string for each subset.

array_value
(type: Sequence)

Sequence[str]

separator
(type: str)

min_subset_size
(type: int)

max_subset_size
(type: int)

{
  "stringified_concatenated_powerset": {
    "type": "string",
    "source": "source_name",
    "function": "powerset_concat",
    "args": [
      ", "
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [1,2,3]
}
<EXAMPLE OUTPUT>
{
  "stringified_concatenated_powerset": [
    "1.0",
    "2.0",
    "3.0",
    "1.0, 2.0",
    "1.0, 3.0",
    "2.0, 3.0",
    "1.0, 2.0, 3.0"
  ]
}
put_value_according_to_bool

Function

Description

Sources

Output

Arguments

put_value_according_to_bool

A directive which returns a value according a source boolean.

bool_value
(type: bool)

use_on_false

use_on_true

{
  "value_according_to_boolean": {
    "type": "string",
    "source": "boolean_source",
    "function": "put_value_according_to_bool",
    "args": [
      "dogs",
      "cats"
    ]
  },
  "value_according_to_boolean_2": {
    "type": "string",
    "source": "boolean_source_2",
    "function": "put_value_according_to_bool",
    "args": [
      "dogs",
      "cats"
    ]
  }
}
<EXAMPLE INPUT>
{
  "boolean_source": true,
  "boolean_source_2": false
}
<EXAMPLE OUTPUT>
{
  "value_according_to_boolean": "cats",
  "value_according_to_boolean_2": "dogs"
}
range_check

Function

Description

Sources

Output

Arguments

range_check

Checks if a value is in one of the given ranges. If given an array as source input,
will return a boolean response for each index in the array.

numeric_value

ranges
(type: Sequence[Mapping[str, Mapping[str, Union[Number, bool]]]])

{
  "value_is_in_range": {
    "type": "boolean",
    "function": "range_check",
    "source": "source_name",
    "args": [
      [
        {
          "left": {
            "value": 100,
            "inclusive": false
          }
        }
      ]
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": 150
}
<EXAMPLE OUTPUT>
{
  "value_is_in_range": true
}
remove_if_false

Function

Description

Sources

Output

Arguments

remove_if_false

Returns first value if the boolean value is true, else None.

first_value

boolean_value
(type: bool)

{
  "first_value_if_true": {
    "type": "numeric",
    "function": "remove_if_false",
    "sources": [
      "source_name_1",
      "source_name_2"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name_1": 10,
  "source_name_2": false
}
<EXAMPLE OUTPUT>
{
  "first_value_if_true": null
}
replace_char

Function

Description

Sources

Output

Arguments

replace_char

A directive which gets a string or list of strings value, and returns it
with char_to_replace replaced by char_to_insert.

original_string_or_string_list

char_to_replace

char_to_insert

{
  "string_with_char_replaced_field_name": {
    "type": "string",
    "function": "replace_char",
    "source": "source_string"
    "args": "p", "l"
  }
  "string_list_with_char_replaced_field_name": {
    "type": "string",
    "function": "replace_char",
    "source": "source_string_array"
    "args": " ", ""
}
<EXAMPLE INPUT>
{
    "source_string": "hocus pocus"
    "source_string_array": ["hocus pocus", "abra cadabra"]
}
<EXAMPLE OUTPUT>
{
    "string_with_char_replaced_field_name": "hucus locus"
    "string_list_with_char_replaced_field_name": ["hocuspocus", "abracadabra"]
}
season_from_timestamp

Function

Description

Sources

Output

Arguments

season_from_timestamp

A directive to get the season of the year of a given timestamp.

timestamp_value
(type: Number)

is_northern_hemisphere

{
  "season_of_timestamp_north": {
    "type": "string",
    "function": "season_from_timestamp",
    "source": "source_name",
    "args": [
      true
    ]
  },
  "season_of_timestamp_south": {
    "type": "string",
    "function": "season_from_timestamp",
    "source": "source_name",
    "args": [
      false
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": 1610569800
}
<EXAMPLE OUTPUT>
{
  "season_of_timestamp_north": "winter",
  "season_of_timestamp_south": "summer"
}
shannon_entropy

Function

Description

Sources

Output

Arguments

shannon_entropy

Returns the shannon entropy of the values in the given array.
the entropy is calculated as S = -sum(array_value * log(array_value), axis=axis).
This routine will normalize the values if they don’t sum to 1.
When use_base_2=false, the log base is e.

array_value

use_base_2

{
  "array_shannon_entropy": {
    "type": "numeric",
    "function": "shannon_entropy",
    "source": "array_source_name"
  }
},
{
  "array_shannon_entropy_base_2": {
    "type": "numeric",
    "function": "shannon_entropy",
    "source": "array_source_name",
    "args":[
        true
    ]
  }
}
<EXAMPLE INPUT>
{
  "array_source_name": [1/2,1/2]
}
<EXAMPLE OUTPUT>
{
  "array_shannon_entropy": 0.6931471805599453
},
{
  "array_shannon_entropy_base_2": 1
}
sort_and_concat_mapper

Function

Description

Sources

Output

Arguments

sort_and_concat_mapper

A mapper directive which, for every (sequence) element in the source array, sorts
and concats as string using the given separator string.

array_value
(type: Sequence[Sequence])

Sequence

separator

{
  "mapped_and_sorted_arrays": {
    "type": "string",
    "function": "sort_and_concat_mapper",
    "source": "source_name",
    "args": [
      ", "
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    [
      "John",
      "Bob",
      "Sharon",
      "Joe"
    ],
    [
      3,
      2,
      6,
      4
    ],
    [
      "dog",
      "cat",
      "monkey"
    ]
  ]
}
<EXAMPLE OUTPUT>
{
  "mapped_and_sorted_arrays": [
    [
      "Bob, Joe, John, Sharon"
    ],
    [
      "2, 3, 4, 6"
    ],
    [
      "cat, dog, monkey"
    ]
  ]
}
sort_array_of_objects_by_key

Function

Description

Sources

Output

Arguments

sort_array_of_objects_by_key

Returns the given array, sorted by the sort_key. If there a none value under
sort_key, we remove it before sorting.

array_value
(type: Sequence)

Union[Sequence, NoneType]

sort_key
(type: str)

reverse

{
  "sorted_array": {
    "function": "sort_array_of_objects_by_key",
    "source": "source_name",
    "do_not_save": true,
    "args": [
      "age"
    ]
  },
  "first_in_sorted_array": {
    "function": "first_in_array",
    "source": "sorted_array",
    "do_not_save": true
  },
  "age_of_first_in_sorted_array": {
    "type": "numeric",
    "function": "field_value_from_object",
    "source": "first_in_sorted_array",
    "args": [
      "age"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "name": "John",
      "age": 20
    },
    {
      "name": "Bob",
      "age": 30
    },
    {
      "name": "Max",
      "age": 19
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "age_of_first_in_sorted_array": "19"
}
split_string_to_array

Function

Description

Sources

Output

Arguments

split_string_to_array

A directive to split a string into an array.

string_value
(type: str)

Sequence[str]

separator
(type: str)

array_wrapping_char
(type: str)

inner_string_wrapping_char
(type: str)

{
  "array_of_strings_after_split": {
    "type": "string",
    "function": "split_string_to_array",
    "source": "string_source_name",
    "args": [
      ", "
    ]
  }
}

{
  "string_source_name": "cat, dog, frog"
}

{
  "array_of_strings_after_split": ["cat","dog","frog"]
}
sub_array

Function

Description

Sources

Output

Arguments

sub_array

Returns a sub array for the given array

array_value
(type: Sequence)

Union[Sequence, NoneType]

start_index
(type: int)

end_index
(type: int)

default_value

{
  "sub_array_according_to_args": {
    "type": "string",
    "function": "sub_array",
    "source": "source_name",
    "args": [
      2,
      5
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [1,2,3,4,5,6,7,8]
}
<EXAMPLE OUTPUT>
{
  "sub_array_according_to_args": [3,4,5,6]
}
sum_array

Function

Description

Sources

Output

Arguments

sum_array

Returns the sum of all values in the given array.

array_value

{
  "sum_of_array": {
    "type": "numeric",
    "function": "sum_array",
    "source": "array_source_name"
  }
}
<EXAMPLE INPUT>
{
  "array_source_name": [1,3,4]
}
<EXAMPLE OUTPUT>
{
  "sum_of_array": 8
}
sum_field_in_array

Function

Description

Sources

Output

Arguments

sum_field_in_array

A directive which returns the sum of the values of a key in an array of
dict objects.

array_value
(type: Sequence[Mapping[str, Number]])

Number

key
(type: str)

{
  "sum_of_values_by_key": {
    "type": "numeric",
    "function": "sum_field_in_array",
    "source": "source_name",
    "args": [
      "age"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name": [
    {
      "name": "John",
      "age": 13
    },
    {
      "name": "Bob",
      "age": 20
    },
    {
      "name": "Sharon",
      "age": 17
    }
  ]
}
<EXAMPLE OUTPUT>
{
  "sum_of_values_by_key": 50
}
sum_formula

Function

Description

Sources

Output

Arguments

sum_formula

A directive for summing (dynamic) multiple numeric value sources. Not
suitable for numerical array sources.

numeric_values
(type: Sequence[Union[Number, NoneType]])

Number

{
  "sum_of_formula_sources": {
    "type": "numeric",
    "function": "sum_formula",
    "sources": [
      "source_name_1",
      "source_name_2",
      "source_name_3"
    ]
  }
}
<EXAMPLE INPUT>
{
  "source_name_1": 5,
  "source_name_2": 6,
  "source_name_3": 10
}
<EXAMPLE OUTPUT>
{
  "sum_of_formula_sources": 21
}
timestamp_delta_days

Function

Description

Sources

Output

Arguments

timestamp_delta_days

A directive for calculating days delta of two timestamps (provided in
seconds/millis).

first_timestamp
(type: Union[Number, NoneType])

second_timestamp
(type: Union[Number, NoneType])

Union[Number, NoneType]

default_value

{
  "delta_of_timestamps_in_days": {
    "type": "numeric",
    "sources": [
      "primary_source",
      "secondary_source"
    ],
    "function": "timestamp_delta_days"
  }
}
<EXAMPLE INPUT>
{
  "primary_source": 1613334600,
  "secondary_source": 1613248200
}
<EXAMPLE OUTPUT>
{
  "delta_of_timestamps_in_days": 1
}
to_int_string

Function

Description

Sources

Output

Arguments

to_int_string

A directive which gets a numeric (or numeric-convertible) value, and returns its
string representation, rounded, without decimal point.

original_value

str

{
  "numeric_to_int_field_name": {
    "type": "string",
    "function": "to_int_string",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
    "source_name": 4.55
}
<EXAMPLE OUTPUT>
{
    "numeric_to_int_field_name": "4"
}
use_value_mapping

Function

Description

Sources

Output

Arguments

use_value_mapping

A directive which returns the mapped value of a key value.

key_value
(type: str)

mapper
(type: str)

{
  "fields": {
    "mapped_value_of_key": {
      "type": "string",
      "function": "use_value_mapping",
      "source": "key_source_name",
      "args": [
        "mapper_data_name"
      ]
    }
  },
  "data": {
    "second_set": {
      "A": "Dogs",
      "B": "Cats",
      "F": "Monkeys",
      "D": "Cows",
      "E": "Birds"
    }
  }
}
<EXAMPLE INPUT>
{
  "key_source_name": "B"
}
<EXAMPLE OUTPUT>
{
  "mapped_value_of_key": "Cats"
}
zip_to_us_state

Function

Description

Sources

Output

Arguments

zip_to_us_state

A directive which gets a US zip code value and returns the US state code it belongs
to. If the zip not found (or given an invalid value) - returns INVALID string.

string_or_int_value
(type: Union[str, int])

str

{
  "zip_code_to_state_field_name": {
    "type": "string",
    "function": "zip_to_us_state",
    "source": "source_name"
  }
}
<EXAMPLE INPUT>
{
    "source_name": 90401
}
<EXAMPLE OUTPUT>
{
    "zip_code_to_state_field_name": "CA"
}

Updated 27 days ago

Did this page help you?