Nest

    • - Take a set of records and place them in a map
    • lift - Take a map by key and lift its records up

    As an example using JSON notation, to nest keys matching the Wildcard value Key* under a new key NestKey the transformation becomes,

    Example (input)

    Example (output)

    1. {
    2.   "OtherKey" : "Value3"
    3.   "NestKey"  : {
    4.     "Key1"     : "Value1",
    5.     "Key2"     : "Value2",
    6.   }
    7. }

    Example usage (lift)

    As an example using JSON notation, to lift keys nested under the Nested_under value NestKey* the transformation becomes,

    1. {
    2.   "OtherKey" : "Value3"
    3.   "NestKey"  : {
    4.     "Key1"     : "Value1",
    5.     "Key2"     : "Value2",
    6.   }
    7. }

    Example (output)

    1. {
    2.   "Key1"     : "Value1",
    3.   "Key2"     : "Value2",
    4.   "OtherKey" : "Value3"

    The plugin supports the following configuration parameters:

    Getting Started

    In order to start filtering records, you can run the filter from the command line or through the configuration file. The following invokes the Memory Usage Input Plugin, which outputs the following (example),

    1. $ bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=Memstats' -p 'Remove_prefix=Mem.' -m '*' -o stdout

    Configuration File

    1. [INPUT]
    2.     Name mem
    5. [OUTPUT]
    6.     Name  stdout
    7.     Match *
    9. [FILTER]
    10.     Name nest
    11.     Match *
    12.     Operation nest
    13.     Wildcard Mem.*
    14.     Nest_under Memstats
    15.     Remove_prefix Mem.

    Result

    The output of both the command line and configuration invocations should be identical and result in the following output.

    1. [2018/04/06 01:35:13] [ info] [engine] started
    2. [0] mem.local: [1522978514.007359767, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Memstats"=>{"total"=>4050908, "used"=>714984, "free"=>3335924}}]

    Example #1 - nest and lift undo

    This example nests all Mem.* and Swap,* items under the Stats key and then reverses these actions with a lift operation. The output appears unchanged.

    Result

    1. [2018/06/21 17:42:37] [ info] [engine] started (pid=17285)
    2. [0] mem.local: [1529566958.000940636, {"Mem.total"=>8053656, "Mem.used"=>6940380, "Mem.free"=>1113276, "Swap.total"=>16532988, "Swap.used"=>1286772, "Swap.free"=>15246216}]

    This example takes the keys starting with Mem.* and nests them under LAYER1, which itself is then nested under LAYER2, which is nested under LAYER3.

    Configuration File

    1. [INPUT]
    2.     Name mem
    3.     Tag  mem.local
    5. [OUTPUT]
    6.     Name  stdout
    7.     Match *
    9. [FILTER]
    10.     Name nest
    11.     Match *
    12.     Operation nest
    13.     Wildcard Mem.*
    14.     Nest_under LAYER1
    17.     Name nest
    18.     Operation nest
    19.     Wildcard LAYER1*
    20.     Nest_under LAYER2
    22. [FILTER]
    23.     Name nest
    24.     Match *
    25.     Operation nest
    26.     Wildcard LAYER2*
    27.     Nest_under LAYER3
    1. [0] mem.local: [1524795923.009867831, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "LAYER3"=>{"LAYER2"=>{"LAYER1"=>{"Mem.total"=>4050908, "Mem.used"=>1112036, "Mem.free"=>2938872}}}}]
    4. {
    5.   "Swap.total"=>1046524,
    6.   "Swap.used"=>0,
    7.   "Swap.free"=>1046524,
    8.   "LAYER3"=>{
    9.     "LAYER2"=>{
    10.       "LAYER1"=>{
    11.         "Mem.total"=>4050908,
    12.         "Mem.used"=>1112036,
    13.         "Mem.free"=>2938872
    14.       }
    15.     }
    16.   }
    17. }

    Example #3 - multiple nest and lift filters with prefix

    This example starts with the 3-level deep nesting of Example 2 and applies the lift filter three times to reverse the operations. The end result is that all records are at the top level, without nesting, again. One prefix is added for each level that is lifted.

    Configuration file

    Result

    1. [0] mem.local: [1524862951.013414798, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Lifted3_Lifted2_Lifted1_Mem.total"=>4050908, "Lifted3_Lifted2_Lifted1_Mem.used"=>1253912, "Lifted3_Lifted2_Lifted1_Mem.free"=>2796996}]
    4. {
    5.   "Swap.total"=>1046524, 
    6.   "Swap.used"=>0, 
    7.   "Swap.free"=>1046524, 
    8.   "Lifted3_Lifted2_Lifted1_Mem.total"=>4050908, 
    9.   "Lifted3_Lifted2_Lifted1_Mem.used"=>1253912,