A transformation reads data from a previous operation, processes the data according to the defined rules and passes the result on to the next operation. The rule options range from simply changing the attributes (e.g. “All in upper case”) to complex logical links (e.g. “Only stroke the cat if it is red or blue”) to self-defined tools that allow any changes to the table. You can find out more about creating these in the chapter ’Rules’.

Transformations are the core component of Chioro in which the processing/modification of attributes/data takes place.

The configuration includes the following aspects.

• Which data is passed on directly and which is processed?
• Which attributes are newly created by linking the existing data?
• Which attributes are copied/mapped directly to a new name?
• Should data from a data table be included in the processing?

#Tab Transformation

#Configure transformation/base

1. the name of the transformation. By default, Chioro simply numbers the transformations consecutively; the name can be customized as required.
2. if this switch is on (default), all attributes are transferred to the data set at the output. In the off position, only the attributes for which a rule exists are passed on.
3. by default, data records that are completely empty are discarded and deleted (default). If this switch is active, empty data records are continued.
4. parallel processing of the data can be prevented. This is useful if the data is interdependent. However, processing will be significantly slower.

#Configure transformation/properties

Metadata can be stored. These are available throughout the flow. See also Metadata.

#Configure transformation/AI Mapping

The AI Mapping tab is used to configure the behaviour of the AI-assisted attribute mapping (the “magic wand” next to a rule block) for this one transformation. Two prompts are sent to the AI:

• System prompt – the general background required for the task (e.g. “The role is that of a data-mapping assistant. Given a list of source and target attributes, the matching source must be selected for each target …”). The system prompt contains no variables and is sent verbatim. It represents the safe extension point: at this location project-specific context can be added without technical risk.
• User prompt – the actual per-call request. It is a Handlebars template containing the variables {{sourceAttributes}} and {{targetAttributes}}, into which the real source and target attributes are injected at runtime. This prompt should only be edited when the consequences are fully understood – if the variables are removed or renamed, the AI no longer receives the data and the magic wand stops producing correct results. For that reason the user prompt is hidden by default behind the Change user prompt link.

Default resolution order: if a field on this operation is blank, the value from the OPENAI_PROVIDER admin configuration is used. If that field is also blank, the built-in default applies. Customisations can therefore be made either per operation (locally) or organisation-wide in the admin configuration.

The Reset to default button (top right above each text field) reloads the corresponding built-in default into the field, after which it can be edited.

The option Include sample values from source columns (top 5) additionally transmits the five most-frequent values of every source column to the AI. This is particularly helpful for columns whose meaning cannot be inferred from the name but is evident from the contents (for example an ext_id column whose values are clearly EAN codes). In exchange, the token usage of the request increases.

#Example: adapting the system prompt for SAP source data

If the source data originates from an SAP system and the target attributes follow the BMEcat 1.2 standard, the system prompt can be extended with project-specific context by appending the following lines after the built-in default text:

Additional context:
- The source attributes come from an SAP export. Fields such as MATNR (material number),
  MAKTX (material short text) or MEINS (unit of measure) are SAP-typical identifiers.
- The target attributes follow the BMEcat 1.2 standard. Fields such as SUPPLIER_PID,
  DESCRIPTION_SHORT or ORDER_UNIT correspond to elements defined there.
- Semantic matches are to be preferred over pure name similarity
  (MATNR → SUPPLIER_PID, MAKTX → DESCRIPTION_SHORT, MEINS → ORDER_UNIT).

#Scope of changes

A change to the system or user prompt in this tab affects only this one transformation. Other transformations in the same flow or in other flows remain unaffected and continue to use either the admin-configuration value or the built-in default.

If a change is to apply organisation-wide to every transformation, it must instead be stored in the OPENAI_PROVIDER admin configuration in the Attribute-Mapping system prompt or Attribute-Mapping user prompt field. Operations whose own override remains blank then automatically pick up the value stored there.

#Configure transformation/comment

The comment stored here is displayed if the mouse pointer remains on an operation for longer than one second in the graphical flow overview.

#Execution

The manual execution of the transformation can be started here.

#History

Information about the previous executions of the transformation.

#Info Metrics

Basic information on the data in the result of the transformation. The data view below can be used for further analysis of the data.

#Tab Rules

#Assignment blocks

Blocks can be defined here to structure rules, insert attributes from a schema or create rules from a data table.

The exact description can be found under [Rule blocks](Rules/Rule Blocks).

#Attribute assignment

Attributes are assigned here. Tools can also be applied to the attributes.

The exact description can be found under Rules.

#Attributes

A list of the currently known attributes. These can be dragged to a rule for assignment.

It should be noted that the actual transformation does not take place in real time and may take some time depending on the number of rules and the number of rows. The preview takes place in real time, as only a very small data set is transformed and is visible on the screen. Our most recent benchmark of an actual transformation with 100,000 rows and 120 rules in a typical Chioro instance took about 15 minutes.