How do teams validate downstream decision changes after feature-version changes?

[sauravsingla]

I’m interested in how Feast users validate one particular consequence of changing production features:

After a feature definition, transformation or feature set changes, how do you determine whether downstream operational decisions changed unexpectedly?

I’ve been working on an open-source project called DeciShift that compares two executable versions of an ML decision system at the final-action level.

It asks:

• Which records received a different final action?
• What action transitions occurred?
• Can the change be traced to a feature, model, threshold, policy or rule?
• Are changes concentrated in a particular cohort?
• Do they stay within an explicit release contract?

GitHub: https://github.com/sauravsingla/DeciShift

I’m considering a Feast-oriented experiment:

feature version A → model/decision flow A

versus

feature version B → model/decision flow B

followed by a DeciShift comparison of the resulting decisions.

For Feast users working with production feature pipelines: is this something you currently validate, and where in the feature/model release workflow do you perform it?

If there is interest, I can create a small reproducible Feast + DeciShift example using public data.