Common filter plugins

    You can use the filter to change the data type of a field. For example, you can use the mutate filter if you’re sending events to OpenSearch and you need to change the data type of a field to match any existing mappings.

    To convert the quantity field from a string type to an integer type:

    Sample output

    You can see that the type of the quantity field is changed from a string to an integer.

    1. {
    2.   "quantity" => 3,
    3.   "host" => "127.0.0.1",
    4.   "@timestamp" => 2021-05-23T19:02:08.026Z,
    5.   "amount" => 10,
    6.   "@version" => "1",
    7.   "headers" => {
    8.     "request_path" => "/",
    9.     "connection" => "keep-alive",
    10.     "content_length" => "41",
    11.     "request_method" => "PUT",
    12.     "cache_control" => "no-cache",
    14.     "content_type" => "application/json",
    15.     "http_version" => "HTTP/1.1",
    16.     "http_host" => "127.0.0.1:8080",
    17.     "accept_encoding" => "gzip, deflate, br",
    18.     "postman_token" => "ffd1cdcb-7a1d-4d63-90f8-0f2773069205"
    19.    }
    20. }

    Other data types you can convert to are float, string, and boolean values. If you pass in an array, the mutate filter converts all the elements in the array. If you pass a string like “world” to cast to an integer type, the result is 0 and Logstash continues processing events.

    For example, you can remove the host field from the event:

    grok

    With the grok filter, you can parse unstructured data and and structure it into fields. The grok filter uses text patterns to match text in your logs. You can think of text patterns as variables containing regular expressions.

    The format of a text pattern is as follows:

    SYNTAX is the format a piece of text should be in for the pattern to match. You can enter any of grok’s predefined patterns. For example, you can use the email identifier to match an email address from a given piece of text.

    The following request consists of the IP address of the visitor, name of the visitor, the timestamp of the request, the HTTP verb and URL, the HTTP status code, and the number of bytes:

    To split this request into different fields:

    1. filter {
    2.   grok {
    3.                              %{USER:auth} \[%{HTTPDATE:reg_ts}\]
    4.                              \"%{WORD:http_verb}
    5.                              %{URIPATHPARAM: req_path}
    6.                              \" %{INT:http_status:int}
    7.                              %{INT:num_bytes:int}"}
    8.   }
    9. }

    where:

    • IP: matches the IP address field.
    • USER: matches the user name.
    • WORD: matches the HTTP verb.
    • URIPATHPARAM: matches the URI path.
    • INT: matches the HTTP status field.
    • INT: matches the number of bytes.

    This is what the event looks like after the grok filter breaks it down into individual fields: