Skip to content

feat: Add SGD, Momentum, and Adam with same-net comparison - #6

Merged
ThomasHartDev merged 1 commit into
mainfrom
thomas/feat/optimizers
Jul 27, 2026
Merged

feat: Add SGD, Momentum, and Adam with same-net comparison#6
ThomasHartDev merged 1 commit into
mainfrom
thomas/feat/optimizers

Conversation

@ThomasHartDev

Copy link
Copy Markdown
Owner

Adds the three standard first-order optimizers as pure numpy update rules that sit next to the hand-written MLP backprop. Each class owns its own state (velocity for momentum, bias-corrected moments for Adam) and steps a flat list of parameter arrays in place, so the training loop can swap rules without touching the gradient math.

compare_optimizers retrains the same architecture on the same data with the same seed for each factory, which keeps init and minibatch order fixed. On XOR all three cut loss; Adam usually pulls ahead early because the adaptive rates absorb uneven gradient scale. Defaults match the common library settings so the curves are easy to reason about against PyTorch.

@ThomasHartDev
ThomasHartDev merged commit e6b2477 into main Jul 27, 2026
2 checks passed
@ThomasHartDev
ThomasHartDev deleted the thomas/feat/optimizers branch July 27, 2026 21:29
Comment thread src/optimizers.py
Comment thread src/mlp.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant