Sudden Changes

The third and last basic "Average" verse we will define is "AverageSuddenChange". This verse configures Mona to find timeseries-based anomalies in the data where the average of a metric has changed significantly in the last day, compared to a defined time period prior to it.

In our case, we want to check if the average of the following metrics: "credit_score", "offered_amount", "approved_amount", "credit_label_delta" and "offered_approved_delta_normalized", has significantlly changed in the last day, in one of the values of our segmentations: "occupation", "purpose", "stage", "model_version" and "city".

Step 1. Add new AverageSuddenChange verse.

1. On the configurations page, under the "Stanzas" tab, go to the "general" stanza and click on the edit button on the right.
2. Go to the "Verses" tab and click on "Add new verse"
3. Again, on the left, you will see a list of all possible verse types. Let's choose AverageSuddenChange.

The verse window is divided into different categories, for different possible params that can be defined in the verse. The first category is "Basic", and it holds all the basic params needed to configure the verse.

📘

All verse params have a default value which will be defined if no other value is given to override

Step 2. Define "Basic" verse params.

In the following params, click on "override" and add the following:

metrics - "credit_score", "offered_amount", "approved_amount", "credit_label_delta", "offered_approved_delta_normalized".
min_anomaly_level - 2
min_culprit_size - 10
min_segment_size_fraction - 0.05
segment_by - "occupation", "purpose","city"
time_series_points - 30

All verses added or edited in the GUI will reflect in the configuration JSON file, which can be downloaded on the configurations page.
Here is how this verse will be defined in our JSON config file:

{
  "stanzas": {
    "name_of_stanza": {
      "verses": [
        {
          "type": "AverageSuddenChange",
          "metrics": [
            "credit_score",
            "offered_amount",
            "approved_amount",
            "credit_label_delta",
            "offered_approved_delta_normalized"
          ],
          "segment_by": [
            "occupation",
            "purpose",
            "city"
          ],
          "baseline_segment": {
            "stage": [
              {
                "value": "inference"
              }
            ]
          },
          "min_anomaly_level": 2,
          "min_culprit_size": 5,
          "min_segment_size_fraction": 0.005,
          "time_series_points": 30
        }
      ]
    }
  }
}

With this verse configuration, we are overriding the default values of AverageSuddenChange (defined under "type"), and we are looking for segments from the segmentation fields (defined in "segment_by") (or any intersection of values) where the average of a metric has changed significantly in the last day, compared to the 29 days prior to it. (You can set the duration of the time series with the "time_series_points" param)

As we only want to track the above metrics in their inference runs, we added a "baseline_segment" param to state what is the baseline.

We are using "min_anomaly_level" to define that a segment is a sudden change when the difference in averages between the last day and the time series prior to it is at least the difference from the median in the benchmark period to a defined percentile, multiplied by "min_anomaly_level".

The "min_culprit_size" param will filter out segments that in the last day have less than 5 records.
Lastly, the "min_segment_size_fraction" param will filter out segments that are smaller than 0.5% of the data.

📘

Low thresholds

Note that we are using low thresholds such as min_anomaly_level and min_segment_size_fraction in order to get insights. After getting insights, users can raise the thresholds to get only relevant and significant insights.

Step 3. Save new verse and stanza.

Once all params have been defined, click on "Add verse", and then "Add stanza".

Once this is defined and saved in the config, Mona will start searching for anomalies that match these parameters. When done, new insights will be generated on the insights page.
You can configure Mona to send notifications on new insights via Email, Slack, PagerDuty, and more. We will go over this in the next chapters.

Step 4. Check new insights.

📘

Once a verse is added or edited, Mona's insight generator starts working immediately, but it might take a few minutes until insights are ready.

AverageSuddenChange insights that match these params will look like this:

This insight shows a sudden jump in the average of "approved_amount" in the last day, when looking at the segment "city: "Little Rock_Arkansas". The jump is from an average of 4821.17 to 14.9k. The anomaly level is 2.18.

When clicking on the insight card, you will open the single insight page which will show you additional information regarding this anomaly.

More information on how to read an insight can be found here

References:
Notifications