-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Description
Is your feature request related to a problem? Please describe.
Currently, output connectors always emit two additional metadata columns alongside user data:
time, which contains the number of computational minibatch in Pathway;diff, which contains either+1or-1, indicating whether the change corresponds to a row insertion or deletion.
In many use cases, this metadata is not required by the user and only adds noise to the output. At the moment, there is no way to disable these fields, even when they are irrelevant for downstream consumers.
Describe the solution you'd like
Introduce an additional configuration field for every output connector, and add an extra optional argument to each emitting method: special_fields_config.
This config should:
- Allow renaming each of the special metadata fields (
time,diff) to custom names; - Allow disabling the emission of each of these fields entirely.
This would give users fine-grained control over what exactly is included in the output.
Describe alternatives you've considered
Keeping the current behavior as-is, with special fields always included.
Additional context
Given the large number of output connectors in Pathway, this feature may be introduced gradually, prioritizing the most commonly used connectors first if necessary.
The change must be fully backward-compatible: the default value of the new entity should reproduce the current behavior exactly (i.e., both fields enabled with their existing names).