Parser

    The plugin supports the following configuration parameters:

    This is an example to parser a record {"data":"100 0.5 true This is example"}.

    The plugin needs parser file which defines how to parse field.

    The path of parser file should be written in configuration file at [SERVICE] section.

    1. [SERVICE]
    2.     Parsers_File /path/to/parsers.conf
    4. [INPUT]
    5.     Name dummy
    6.     Tag  dummy.data
    7.     Dummy {"data":"100 0.5 true This is example"}
    9. [FILTER]
    10.     Name parser
    11.     Match dummy.*
    12.     Key_Name data
    13.     Parser dummy_test
    15. [OUTPUT]
    16.     Name stdout
    17.     Match *
    1. $ fluent-bit -c dummy.conf
    2. Fluent Bit v1.x.x
    3. * Copyright (C) 2019-2020 The Fluent Bit Authors
    4. * Copyright (C) 2015-2018 Treasure Data
    5. * https://fluentbit.io
    7. [2017/07/06 22:33:12] [ info] [engine] started
    8. [0] dummy.data: [1499347993.001371317, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
    9. [1] dummy.data: [1499347994.001303118, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
    10. [2] dummy.data: [1499347995.001296133, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]

    You can see the record {"data":"100 0.5 true This is example"} are parsed.

    Preserve original fields

    By default, the parser plugin only keeps the parsed fields in its output.

    If you enable Reserve_Data, all other fields are preserved:

    1. [SERVICE]
    2.     Parsers_File /path/to/parsers.conf
    4. [INPUT]
    5.     Name dummy
    6.     Tag  dummy.data
    7.     Dummy {"data":"100 0.5 true This is example", "key1":"value1", "key2":"value2"}
    9. [FILTER]
    10.     Name parser
    11.     Match dummy.*
    12.     Key_Name data
    13.     Parser dummy_test
    14.     Reserve_Data On

    This will produce the output:

    1. $ fluent-bit -c dummy.conf
    2. Fluent-Bit v0.12.0
    3. Copyright (C) Treasure Data
    5. [2017/07/06 22:33:12] [ info] [engine] started
    6. [1] dummy.data: [1499347994.001303118, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}, "key1":"value1", "key2":"value2"]
    7. [2] dummy.data: [1499347995.001296133, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}, "key1":"value1", "key2":"value2"]
    8. [3] dummy.data: [1499347996.001320284, {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}, "key1":"value1", "key2":"value2"]
    1. [SERVICE]
    2.     Parsers_File /path/to/parsers.conf
    4. [INPUT]
    5.     Name dummy
    6.     Tag  dummy.data
    7.     Dummy {"data":"100 0.5 true This is example", "key1":"value1", "key2":"value2"}
    9. [FILTER]
    10.     Name parser
    11.     Match dummy.*
    12.     Key_Name data
    13.     Parser dummy_test
    14.     Reserve_Data On
    15.     Preserve_Key On

    This will produce the output:

    1. $ fluent-bit -c dummy.conf
    2. Fluent-Bit v0.12.0
    3. Copyright (C) Treasure Data
    5. [2017/07/06 22:33:12] [ info] [engine] started
    6. [0] dummy.data: [1499347993.001371317, {"data":"100 0.5 true This is example", "INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
    7. [1] dummy.data: [1499347994.001303118, {"data":"100 0.5 true This is example", "INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]
    8. [2] dummy.data: [1499347995.001296133, {"data":"100 0.5 true This is example", "INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}]

    Some timestamps might have fractional seconds, like 2017-05-17T15:44:31.187512963Z. The %L format option for Time_Format is provided as a way to indicate that content must be interpreted as fractional seconds. To parse the previous example, you could specify Time_Format %Y-%m-%dT%H:%M:%S.%LZ.

    {% hint style=”info” %} The option %L is only valid when used after seconds (%S) or seconds since the Epoch (%s), e.g: %S.%L or . {% endhint %}

    {% hint style=”info” %} Support for %L was added in Fluent Bit 0.12. {% endhint %}