The data processing in AireInsights happens in modules, small services that are connected to the core AireInsights serivces. To build a custom module one of the SDKs can be used. In the pipeline definition a module is referenced and configured in a step.
Syntax
Yaml
Name: String
Type: StepType
InputTemplate: Liquid
ModuleName: String
ModuleContext: JsonString
ModuleInputFormat: DataFormat
ModuleOutputFormat: DataFormat
Json
{
"Name": String,
"Type": StepType,
"InputTemplate": Liquid,
"ModuleName": String,
"ModuleContext": JsonString,
"ModuleInputFormat": DataFormat,
"ModuleOutputFormat": DataFormat,
}
Properties
Name
The name of the step has to be unique (accross one pipeline) and is used to reference the step output in templates of other steps or the pipeline’s output.
Required: Yes
Type: String of alphanumeric characters and underscores, should always start with a letter, and not have any kind of leading sigil
Type
A ForAll step receives the data as arrays as is. In a ForEach step the variable specified in IterateOver is passed to the step element by element, the step is executed once for every element in that array.
Required: No
Type: StepType
Default: ForAll (0)
Options: ForAll (0), ForEach (1)
IterateOver
The name of a variable in the InputTemplate which will be passed to the step element by element, the step is executed once for every element in that array. In the InputTemplate this variable wont be an array anymore but just a single item.
Required: Only if Type is ForEach (1)
Type: String
InputTemplate
Liquid template to define the data sent to the module. All step outputs are available as a variable with the name of the step.
Required: Yes
Type: Liquid
ModuleName
The name of the module that should be used for this step.
Required: Yes
Type: String
ModuleContext
A String of a JSON. Allows to configure the module for this step.
Required: No
Type: JsonString
ModuleInputFormat
The data format of the module’s input data.
Required: Yes
Type: DataFormat
Options: Raw (0), Json (1), Xml (2), Csv (3), Text (4)
ModuleOutputFormat
The data format of the module’s output data.
Required: Yes
Type: DataFormat
Options: Raw (0), Json (1), Xml (2), Csv (3), Text (4)