Skip to content

feat: optionally look up disposable domains without building a Set of them - #326

Merged
phoet merged 3 commits into
micke:mainfrom
Spone:sorted-disposable-domains
Nov 10, 2026
Merged

phoet merged 3 commits into
micke:mainfrom
Spone:sorted-disposable-domains

Conversation

@Spone

@Spone Spone commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem

ValidEmail2.disposable_emails loads the ~180,000 disposable domains into a Set. Each domain becomes its own String plus a hash entry, so the first email checked grows the process by about 35 MB.

Change

The domains file is sorted and has no duplicates: pull_mailchecker_emails.rb writes it with .uniq.sort. So it can be searched without building a Set, as an opt-in trade of lookup speed for memory:

ValidEmail2.disposable_domains_lookup = :sorted_list # default: :set
  • ValidEmail2::SortedDomainList keeps the file as a single binary String (about 2.6 MB) and answers include? by bisection over its lines. Each step reads one line around the middle offset, so there's no index to build.

  • ValidEmail2.disposable_domains is what Address#disposable_domain? and #disposable_mx_server? now use:

    • with :set (the default), the Set of disposable_emails, as before;
    • with :sorted_list, the SortedDomainList.

    Any other value raises an ArgumentError.

  • disposable_emails still returns the same mutable Set, built on demand. With :sorted_list, domains added to it or removed from it are ignored by the validations; the README says so.

Measured

Against config/disposable_email_domains.txt (178,700 domains). Memory is the process's RSS growth after loading and one lookup, each case in a fresh process:

:set :sorted_list
Memory, Ruby 3.3.7 +33.8 MB +2.6 MB
Memory, Ruby 4.0.7 +36.2 MB +2.6 MB
Load time (Ruby 3.3.7) 80 ms 1.1 ms
Lookup (Ruby 3.3.7) 0.05 µs 4.2 µs

A sorted list lookup is slower than a Set's but stays well under the existing perform_under(0.0001) benchmark (100 µs).

Specs

  • spec/sorted_domain_list_spec.rb:
    • the file is sorted without duplicates, so a future generator change breaking the order fails here;
    • domains not in it aren't found.
  • spec/valid_email2_spec.rb:
    • :set is the default, and an unknown lookup is rejected;
    • with :sorted_list, a disposable domain is detected without loading the Set;
    • with :sorted_list, every entry of the Set is found in the sorted list.
  • spec/benchmark_spec.rb: a lookup benchmark with :sorted_list.

bundle exec rake: 148 examples, 0 failures. RuboCop reports the same 2 offenses as on main (Metrics/ClassLength on Address, Metrics/BlockLength on the main spec).

@Spone
Spone force-pushed the sorted-disposable-domains branch from 2c6cef4 to 9faaf2c Compare October 12, 2026 14:37
@Spone Spone changed the title perf: look up disposable domains without building a 14 MB Set perf: look up disposable domains without building a Set of them Oct 12, 2026
@phoet

phoet commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

this is a tradeoff that we should not decide on. if you would want to have a smaller footprint and be fine with slower lookup time, you should have an easy way to configure it.

so it would be ok for me if you add the additional bin format class and have an option to turn it on instead of using the set.

in addition, i would want to have a spec that makes sure every entry is found in both implementations, such as

  it "has all entries from the other list" do
    disposable_domains = ValidEmail2.disposable_domains
    disposable_emails = ValidEmail2.disposable_emails
    expect(disposable_domains).to be_a(ValidEmail2::SortedDomainList)
    expect(disposable_emails).to be_a(Set)
    expect(disposable_emails.reject { |domain| disposable_domains.include?(domain) }).to be_empty
  end

@Spone Spone changed the title perf: look up disposable domains without building a Set of them feat: optionally look up disposable domains without building a Set of them Nov 9, 2026
@Spone

Spone commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Makes sense, thanks! The sorted list is now opt-in, and the default is unchanged (the Set):

ValidEmail2.disposable_domains_lookup = :sorted_list
  • ValidEmail2.disposable_domains returns the SortedDomainList when that option is set, and disposable_emails (the Set) otherwise. Any other value raises an ArgumentError.
  • I added your spec checking that every entry of the Set is found in the sorted list.
  • The README documents the option and its trade-off: about 3 MB instead of about 35 MB, at a few microseconds per lookup.

Spone added 2 commits November 9, 2026 22:56
ValidEmail2.disposable_emails loads the ~180,000 disposable domains into
a Set: one String object per domain plus its hash entry, about 35 MB of
memory once the first email is checked. The file is sorted (see
pull_mailchecker_emails.rb), so lookups now go through
ValidEmail2.disposable_domains, a SortedDomainList keeping the file as a
single String (2.6 MB) searched by bisection.

disposable_emails still returns the same Set, built on demand. Once it
is loaded, disposable_domains returns it, so code adding domains to it
or removing them keeps working.
Lookups use the Set of disposable_emails by default again. Setting
ValidEmail2.disposable_domains_lookup = :sorted_list switches them to
the SortedDomainList, trading lookup speed for memory.
@Spone
Spone force-pushed the sorted-disposable-domains branch from bb450d8 to b854149 Compare November 9, 2026 21:56

@phoet phoet left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just small changes, works otherwise.

Comment thread lib/valid_email2.rb Resolved
Comment thread lib/valid_email2.rb Resolved
@Spone

Spone commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

@phoet changes done!

@phoet
phoet merged commit edb46ea into micke:main Nov 10, 2026
8 checks passed
@phoet

phoet commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

@Spone could you run main in production with your lookup on and report back before I craft a new version?

@Spone

Spone commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Yes of course! I'll do that and report back

@Spone
Spone deleted the sorted-disposable-domains branch November 16, 2026 09:01
@Spone

Spone commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Hi @phoet I've been running the main branch in production for 3 days, with the new ValidEmail2.disposable_domains_lookup = :sorted_list option, without any issue.

@phoet

phoet commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

👍

@Spone

Spone commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

@phoet can you push valid_email2 on RubyGems please? 🙏

@phoet

phoet commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

a new version should have been released, but nothing showed up on rubygems. have to investigate what went wrong. https://github.com/micke/valid_email2/actions/runs/43881741055/job/133349106539

@phoet

phoet commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

wrestled it to do something

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants