Hi Spark community,
In our company we were planning to do some experiments on introducing table / operator stats from systems outside Spark. I didn't find a good injection point to do this in Spark public API.
I found PR #25015 could be useful but it didn't end up merging.
Another approach for us is to inject a logical optimizer rule to set stats for Spark logical plan. I found this API helpful but it's not exposed.
Can we expose the API or both of the 2 APIs to the extensions API? Or any suggestion is welcomed. Thanks!