Skip to content

Add an option to disable or rename special fields in output connectors #162

@zxqfd555

Description

@zxqfd555

Is your feature request related to a problem? Please describe.
Currently, output connectors always emit two additional metadata columns alongside user data:

  • time, which contains the number of computational minibatch in Pathway;
  • diff, which contains either +1 or -1, indicating whether the change corresponds to a row insertion or deletion.

In many use cases, this metadata is not required by the user and only adds noise to the output. At the moment, there is no way to disable these fields, even when they are irrelevant for downstream consumers.

Describe the solution you'd like
Introduce an additional configuration field for every output connector, and add an extra optional argument to each emitting method: special_fields_config.

This config should:

  • Allow renaming each of the special metadata fields (time, diff) to custom names;
  • Allow disabling the emission of each of these fields entirely.

This would give users fine-grained control over what exactly is included in the output.

Describe alternatives you've considered
Keeping the current behavior as-is, with special fields always included.

Additional context
Given the large number of output connectors in Pathway, this feature may be introduced gradually, prioritizing the most commonly used connectors first if necessary.

The change must be fully backward-compatible: the default value of the new entity should reproduce the current behavior exactly (i.e., both fields enabled with their existing names).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions