Flesh out schema evolution support for variant

[voonhous]

Task Description

What needs to be done:

https://github.com/apache/hudi/pull/17833/changes#r2874744476

#17833 (comment)

Why this task is needed:

Comment from @vinothchandar
What kind of evolution are we able to support with Variant columns? Can we ensure there are tests around.

• Adding a Variant column to an existing table
• Removing a Variant column
• maxColumnId calculation when Variant fields (with negative IDs) coexist with regular fields
• Round-trip: HoodieSchema → InternalSchema → HoodieSchema for Variant types

Task Type

Code improvement/refactoring

Related Issues

Parent feature issue: (if applicable )
Related issues:
NOTE: Use Relationships button to add parent/blocking issues after issue is created.