fix: python decompose model change and pipeline fix

[AngeloDanducci]

Misc PR

Type of PR

• Bug Fix
• New Feature
• Documentation
• Other

Description

• Link to Issue: fix: update python decompose example #566

Testing

• Tests added to the respective file if code was changed
• New code has 100% coverage if code as added
• Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

[github-actions]

The PR description has been updated. Please fill out the template for your PR to be reviewed.

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert|release)(?:\(.+\))?:

[AngeloDanducci]

@psschwei that issue you ran into should be solved with this update

@csbobby could use your feedback as well, the change in cli/decompose/pipeline.py assigns a validation strategy of llm if none exists - though I think the more correct long term change is probably to pass the validation strategy down when breaking up constraints. Prior to #445 a single constraint would not be broken down so this was a non issue.

It looks like this area is getting reworked in #556 so maybe this is fine in the interim.

[jakelorocco]

@csbobby, can you please confirm that you are okay with changing this and always having an LLM default?

[csbobby]

Hi @AngeloDanducci @jakelorocco Thank you for sharing this.
Yes, adding a fallback setup "llm" looks good point.

• We tested the extraction feature by using the mainstream models/APIs. Typically, it works well on both regular or larger sizes (7B or +). We don't recommend to use the weaker llms since the output quality will be potentially different or unpredictable. But if any user encounter it, the "llm" seems will helpful to provide a best try on meaningful constraint content instead of a program error. If you think this satisfy our intent Mellea usage for users, also looks good to me.

About the fix, the ideal position I think is the first position of the error occurred.

• The cons_key is the validation strategy, and the cons_str is the validation type (llm/code) for the following up execution. The extraction failure (if there is) of the validation type will be detected on the validation_decision.generate(m_session, cons_key).parse() at (L119 ).
• I would suggest to add the fallback to the extraction failure position at (L119 ) rather than other following steps (e.g., on the returned DecompPipelineResult (L145-147) in the original fix).

That would keep the consistency for the fallbacked cons_str.

Line 119

    constraint_validation_strategies: dict[str, Literal["code", "llm"]] = {
        cons_key: validation_decision.generate(m_session, cons_key).parse() or "llm"
        for cons_key in task_prompt_constraints
    }

[csbobby]

FYI, also completed #556.

[AngeloDanducci]

Thanks @csbobby - I've added the default value to line 119.

However I've also kept the update on 145.

I think the update on line 145 handles cases where a constraint in subtask_data.constraints doesn't exist as a key in constraint_validation_strategies at all (which happened when I ran this example).