fix Exponential blowup in Protocol-vs-Protocol structural subtyping #2595

[asukaminato0721]

Summary

Fixes #2595

Added a per-Subset memoization cache for protocol structural subtyping to break recursive protocol cycles and avoid repeated work across protocol members.
Kept the existing class-level recursive assumptions for cases involving quantified vars, and limited caching to stable class-type pairs without quantified vars.

Test Plan

add perf test

[Copilot]

Pull request overview

Adds memoization to protocol structural subtyping to prevent pathological recursion/exponential work when checking Protocol-to-Protocol conformance (as in issue #2595), and includes a regression perf test to exercise the cycle scenario.

Changes:

• Introduce a per-Subset protocol subtyping cache keyed by stable (Type, Type) pairs (excluding types containing Vars) to short-circuit repeated Protocol checks.
• Add ProtocolSubsetCacheEntry and wire cache initialization into Solver::subset.
• Add a perf regression testcase reproducing the cyclic Protocol scenario from #2595.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
pyrefly/lib/solver/subset.rs	Adds memoized fast-path for class-to-protocol structural subtyping checks to avoid repeated work/cycles.
pyrefly/lib/solver/solver.rs	Defines cache entry enum and initializes new per-Subset cache field.
pyrefly/lib/test/perf.rs	Adds a regression perf testcase that triggers Protocol-vs-Protocol cyclic structural subtyping.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[meta-codesync]

@grievejia has imported this pull request. If you are a Meta employee, you can view this in D94926986.

[meta-codesync]

This pull request has been merged in 0527e8c.

[yangdanny97]

There was a soundness issue in the original implementation of this PR that @grievejia patched in the merged version

If you're curious, refer to test_protocol_coinductive_cache_soundness for more details :)

[asukaminato0721]

Understand, I need to rollback the cached result if failed.