Rebuild the Salt-SSH thin archive when its configured contents change - #70288
Open
jpmckinney wants to merge 1 commit into
Open
jpmckinney wants to merge 1 commit into
jpmckinney wants to merge 1 commit into
Conversation
gen_thin only compared the Salt version and the Python major version, so an archive cached in the master cachedir was reused even after a Salt extension was installed or thin_extra_mods changed, and neither reached the target. Record an identity of what the archive was generated from, and compare against it. thin_sum calls gen_thin again, so it now forwards the same arguments: otherwise the regeneration this makes possible would drop the caller's configuration. Also fix thin_extra_mods for a module inside a namespace package, such as saltext.mysql: __import__ returns the namespace package, whose __file__ is None, which raised a TypeError instead of packing the extension. Document that thin_extra_mods takes a dotted path, along with the thin_exclude_saltexts, thin_saltext_allowlist and thin_saltext_blocklist options added in 3008.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Makes
gen_thin()treat the configured contents of the thin archive as part of its cache key, and letsthin_extra_modsname a module inside a namespace package.gen_thin()writes<cachedir>/thin/.thin-gen-config, recordingextra_mods,so_modsand the installed Salt extensions (name and version, read from entry points without importing them), and rebuilds the archive when that identity changes.thin_sum()takes**kwargsand forwards them togen_thin(), andSinglepasses the options it was given. Without this, the secondgen_thin()call could rebuild the archive with default options, droppingthin_extra_mods,ssh_ext_alternativesand thethin_saltext_*filters.get_tops()imports an extra module withimportlib.import_module()and resolves the directory to pack with the existing_get_package_root_mod()helper, so a dotted name in a namespace package is packed at the right path with its namespace.thin_extra_modstaking a dotted path, and adds the missing entries forthin_exclude_saltexts,thin_saltext_allowlistandthin_saltext_blocklist, including the fact that extension dependencies are not shipped.This targets 3008.x, the oldest supported branch where extensions are packed into the thin archive. The
thin_extra_modscrash is older than that — 3006.x and 3007.x have the sameos.path.split(__import__(mod).__file__)— but the fix here uses_get_package_root_mod(), which was added in 3008.0. Happy to write a separate backport for the older branches if you want one.What issues does this PR fix or reference?
Fixes #70287
References #70284
Previous Behavior
A cached archive was reused as long as the Salt version and the Python major version were unchanged, so newly installed Salt extensions and changes to
thin_extra_modsnever reached the target until--regen-thinwas used.thin_extra_mods: "saltext.mysql"raisedTypeError: expected str, bytes or os.PathLike object, not NoneTypewhen the archive was regenerated, because__import__()returns thesaltextnamespace package, whose__file__isNone.New Behavior
The archive is rebuilt when the extensions or the configured extra modules change, and is still reused when nothing changed.
thin_extra_mods: "saltext.mysql"packs the extension atpy3/saltext/mysql/.Verified with
salt==3008.2,saltext.mysql==1.1.0andpymysql, generating into a fresh cache directory: addingthin_extra_mods: pymysqlrebuilds the archive and packs 18py3/pymysql/files, where 3008.2 returns the archive it built before, unchanged.Two caveats worth naming:
gen_thin()after this change rebuilds the archive once, since an archive without.thin-gen-configis treated as outdated.thin_sum()gains a**kwargsparameter. Existing callers keep working, but a caller that passes no options can now regenerate an archive that was built with them, which is whySinglewas updated in the same commit.Merge requirements satisfied?
Commits signed with GPG?
Yes