Skip to content

AO3-6591: Move imported_from_url to a new table with different varian…#5585

Open
gmjimenezc wants to merge 10 commits intootwcode:masterfrom
gmjimenezc:AO3-6591
Open

AO3-6591: Move imported_from_url to a new table with different varian…#5585
gmjimenezc wants to merge 10 commits intootwcode:masterfrom
gmjimenezc:AO3-6591

Conversation

@gmjimenezc
Copy link
Contributor

…ts in different columns

Pull Request Checklist

Issue

https://otwarchive.atlassian.net/browse/AO3-6591

Purpose

  • Creates a new work_import_urls table to store import URLs separately from the works table, with pre-computed URL variant columns (minimal, minimal_no_protocol_no_www) for fast exact-match lookups
  • Eliminates the slow LIKE query in Work.find_by_url — the old 3-tier fuzzy matching (exact → variants → LIKE with Ruby filtering) is replaced by simple find_by calls against indexed variant columns
  • Moves all URL lookup logic into a new WorkImportUrl model, including caching, reducing complexity in the already-bloated Work model
  • Removes the imported_from_url column from the works table entirely, replacing it with a has_one :work_import_url association and a delegated imported_from_url reader method
  • Updates the Elasticsearch indexer to read imported_from_url via the association method instead of the column
  • Updates StoryParser to build a WorkImportUrl record when importing works
  • Updates the Open Doors tools controller and rake task to create/update WorkImportUrl records directly
  • Updates the API v2 works controller to query the new table for import URL searches
  • Adds a backfill rake task (work_import_urls:backfill) to populate the new table from existing data before the column is removed
  • Adds specs for the new model covering variant computation, validations, URL matching, and caching

Testing Instructions

How can the Archive's QA team verify that this is working as you intended?

References

Are there other relevant issues/pull requests/mailing list discussions? If not, you can remove this section.

Credit

warlockmel (she/her)

@github-actions github-actions bot added Has Migrations Contains migrations and therefore needs special attention when deploying Awaiting Review labels Feb 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Awaiting Review Has Migrations Contains migrations and therefore needs special attention when deploying

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant