Skip to content

[Shopify] Migrate Shopify Connector to Expiring Offline Access Tokens#9327

Open
onbuyuka wants to merge 19 commits into
mainfrom
bugs/637954-migrate-shopify-expiring-tokens
Open

[Shopify] Migrate Shopify Connector to Expiring Offline Access Tokens#9327
onbuyuka wants to merge 19 commits into
mainfrom
bugs/637954-migrate-shopify-expiring-tokens

Conversation

@onbuyuka

@onbuyuka onbuyuka commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Background

As of December 2025, Shopify supports expiring offline access tokens with refresh-token rotation. Public apps created before 2026-04-01 must migrate to expiring tokens by 2027-01-01; after that date, REST/GraphQL Admin API requests made with non-expiring tokens are rejected. The Shopify Connector currently uses non-expiring offline tokens and must be migrated.

Token characteristics (Shopify):

  • Access token lifetime: 1 hour (expires_in: 3600)
  • Refresh token lifetime: 90 days (refresh_token_expires_in: 7776000)
  • Migration is a one-time, irreversible token exchange per shop (the original non-expiring token is revoked on success)
  • Only one refreshable expiring token per app + store

Refs: About offline access tokens - Migrating from non-expiring to expiring tokens

Approach

Token validity is centralized in a single orchestrator, EnsureValidAccessToken(Store), invoked from Shpfy Communication Mgt.GetAccessToken - so every API call (interactive and background) transparently migrates a legacy token or refreshes an expiring one before use. A scheduled backstop job additionally keeps idle shops access/refresh tokens alive and completes migrations for shops that never make an on-demand call.

Changes

Token storage - Shpfy Registered Store New (table 30138)

  • New fields Token Expires At / Refresh Token Expires At; refresh token kept in IsolatedStorage.

Acquisition, refresh & migration - Shpfy Authentication Mgt. (codeunit 30199)

  • New installs request expiring tokens (expiring=1).
  • EnsureValidAccessToken orchestrates migrate-if-legacy / refresh-if-near-expiry, serialized per shop.
  • RefreshAccessToken (grant_type=refresh_token) with transient retry and a terminal reconnect error.
  • MigrateToExpiringToken (one-time token exchange, best-effort - keeps the working token on failure).
  • ForceTokenRefresh, IsRefreshTokenExpired, and a Shop Card reconnect notification.

Consumption - Shpfy Communication Mgt.

  • GetAccessToken invokes the orchestrator; reactive 401 -> refresh -> retry in ExecuteWebRequest.

Scheduled backstop - Shpfy Token Refresh (30431) + Shpfy Token Refresh Shop (30432)

  • Dispatcher (TableNo = Job Queue Entry) iterates enabled shops and runs the per-shop worker (TableNo = Shpfy Shop) via Codeunit.Run for isolation; recurring Job Queue entry registered via installer + upgrade.

UX - Shpfy Shop Card

  • Reconnect notification when the refresh token has expired.

Docs & tests

  • Shpfy Token Refresh Test unit tests; living docs refreshed (business-logic, data-model, patterns, CLAUDE.md).

Telemetry

Emitted for migration success/failure, refresh success/transient/terminal and backstop per-shop failures, under event IDs 0000UIV-0000UJ1.

Test plan

  • Shopify App project compiles clean.
  • Shpfy Token Refresh Test unit tests for the expiry decisions.
  • Install / refresh / migration / backstop / notification scenarios validated locally.

Fixes AB#637954

Adds support for Shopify expiring offline access tokens (public apps must
migrate by 2027-01-01):

- Persist token/refresh expiry on "Shpfy Registered Store New" and store the
  refresh token in IsolatedStorage.
- Request expiring tokens on install (expiring=1), refresh before expiry, and
  migrate legacy non-expiring tokens via token exchange (best-effort).
- Orchestrate on-demand from "Shpfy Communication Mgt." (GetAccessToken) with a
  reactive 401 refresh-and-retry.
- Scheduled backstop job ("Shpfy Token Refresh" + per-shop worker) to keep
  tokens and 90-day refresh tokens alive; registered via installer/upgrade.
- Shop Card reconnect notification when the refresh token has expired.
- Tests for the refresh-token expiry decision logic.

NOTE: "Shpfy Token Dev Tools" (page 30440) is temporary test scaffolding and
must be removed before merge.

Fixes AB#637954

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
@github-actions github-actions Bot added the AL: Apps (W1) Add-on apps for W1 label Jul 11, 2026
@github-actions github-actions Bot added this to the Version 29.0 milestone Jul 11, 2026
onbuyuka and others added 4 commits July 11, 2026 17:06
- Remove the temporary "Shpfy Token Dev Tools" page (30440) and its permission-set entry.
- Clear the new telemetry event IDs (set to '') so they can be assigned by the tagging script.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Refresh the living docs (al-docs update) to cover slice 637954:
- business-logic.md: new "Authentication and token lifecycle" section.
- data-model.md: Shpfy Registered Store New token/expiry storage + ER entry.
- patterns.md: Job Queue dispatcher/worker per-shop isolation pattern.
- CLAUDE.md (app + Base): auth overview and "things to know".

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Remove the LogTokenTelemetry helper and inline Session.LogMessage('', ...) at
each call site so the telemetry tagging script can assign event IDs (it only
fills literal Session.LogMessage first-args). Matches the existing connector
convention. Tags left empty pending the script.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Fill the token lifecycle telemetry event IDs (0000UIV-0000UJ1) via the tagging script.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
- Mark SaveInstalledToken [NonDebuggable] (it holds the token-bearing response body).
- Give the token-refresh backstop its own Job Queue category (SHPFYAUTH) so it is
  not serialized behind long-running SHPFY sync jobs (pricing/inventory) and can
  refresh 1-hour access tokens promptly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
@onbuyuka
onbuyuka marked this pull request as ready for review July 13, 2026 06:57
@onbuyuka
onbuyuka requested a review from a team July 13, 2026 06:58
Comment thread src/Apps/W1/Shopify/App/src/Integration/Codeunits/ShpfyTokenRefresh.Codeunit.al Outdated
Comment thread src/Apps/W1/Shopify/App/src/Integration/Tables/ShpfyRegisteredStoreNew.Table.al Outdated
Comment thread src/Apps/W1/Shopify/App/src/Integration/Codeunits/ShpfyTokenRefresh.Codeunit.al Outdated
@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Copilot PR Review

Iteration 11 · Outcome: completed

Knowledge source: https://github.com/microsoft/BCQuality@186d8a131465475c79244d994acb872cd5c0d4bf

Findings by domain

Findings split into Knowledge-backed (cite a BCQuality article) and Agent (the agent's own judgement, no matching BCQuality rule).

Domain Findings Knowledge-backed Agent Inline Fallback
Error Handling 1 1 0 1 0
Performance 1 1 0 0 0

Totals: 2 knowledge-backed · 0 agent findings.

Orchestrator pre-filter (2 file(s) excluded)

  • layer-disabled (knowledge) : 2 file(s)

Findings produced by the AL review agent v1.7.3. Reply 👎 on any inline comment to flag false positives.

- Encrypt the refresh token at rest: IsolatedStorage.SetEncrypted instead of Set
  (90-day credential; Get transparently decrypts).
- Mark ResponseHasAccessToken [NonDebuggable] (handles the token-bearing body).
- Remove the redundant per-iteration Commit() in the token-refresh dispatcher
  loop; the per-shop worker (Codeunit.Run) already commits on success.
- Guard Codeunit.Run("Job Queue - Enqueue") in ScheduleRefreshJob so an enqueue
  failure is logged instead of aborting the install/upgrade.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Comment thread src/Apps/W1/Shopify/App/src/Base/Codeunits/ShpfyCommunicationMgt.Codeunit.al Outdated
@github-actions

Copy link
Copy Markdown
Contributor

$\textbf{🟠\ High\ Severity\ —\ Security} \quad \color{gray}{\texttt{\small Iteration\ 2}}$

SaveTokenResponse (new in this PR) stores the Shopify offline access token via RegisteredStoreNew.SetAccessToken(), which persists it with IsolatedStorage.Set (unencrypted at rest), while the refresh token introduced by this same PR is correctly stored via SetRefreshToken() using IsolatedStorage.SetEncrypted.

Both are OAuth credentials for the same store and belong in the same encrypted class. This inconsistency becomes load-bearing now: every store going forward will have an encrypted refresh token sitting next to an unencrypted, equally sensitive access token in the same record. Change SetAccessToken (Shpfy Registered Store New table) to use SetEncrypted for parity with the refresh token.

Knowledge:

Line mapping was unavailable, so this was posted as an issue comment.

👍 useful · ❤️ especially valuable · 👎 wrong - reply with why

onbuyuka and others added 2 commits July 13, 2026 10:50
Fill 0000UJ6 (enqueue-failure warning) via the tagging script.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
A legacy store has no refresh token, so EnsureValidAccessToken never hits the
fast path and re-attempts migration on every API call -- in a sync loop over
many records a failing migration means many redundant lock+HTTP+commit round
trips. Add a "Last Migration Attempt" timestamp on Shpfy Registered Store New
and only re-attempt migration once per hour (ShouldAttemptMigration/TryMigrate,
shared by EnsureValidAccessToken and ForceTokenRefresh). Docs updated.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Comment thread src/Apps/W1/Shopify/App/src/Integration/Codeunits/ShpfyTokenRefresh.Codeunit.al Outdated
Comment thread src/Apps/W1/Shopify/App/src/Integration/Codeunits/ShpfyTokenRefresh.Codeunit.al Outdated
Comment thread src/Apps/W1/Shopify/App/src/Integration/Codeunits/ShpfyTokenRefresh.Codeunit.al Outdated
onbuyuka and others added 3 commits July 13, 2026 11:12
- Do not interpolate GetLastErrorText() into the telemetry message string
  (LogRefreshFailure and the schedule-failure log). Use a generic message,
  move the error detail and shop code to custom dimensions, and classify the
  event as CustomerContent so the platform can handle it appropriately.
- Move the shared telemetry Labels to the codeunit object scope.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Codeunit 139635 is already used by "E-Doc. Receive Files" in the EDocument
test app; BCApps validates test object IDs globally. Move Shpfy Token Refresh
Test to 139613, which is free repo-wide within the Shopify test ID ranges.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
…riant API Test)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
@github-actions

Copy link
Copy Markdown
Contributor

$\textbf{🟡\ Medium\ Severity\ —\ Security} \quad \color{gray}{\texttt{\small Iteration\ 4}}$

The new SetRefreshToken procedure correctly uses IsolatedStorage.SetEncrypted for the 90-day refresh token, but the pre-existing SetAccessToken (line 68 of the same table) still uses the unencrypted IsolatedStorage.Set for the access token, sitting right next to the new encrypted call.

Both values are Shopify credentials of comparable sensitivity per the referenced guidance ('anything that would harm the tenant if leaked — API keys, tokens, connection strings, OAuth client secrets — uses SetEncrypted'). Since this PR is already touching this table's secret-handling code, it's a good opportunity to align SetAccessToken with SetEncrypted for consistency.

Knowledge:

Line mapping was unavailable, so this was posted as an issue comment.

👍 useful · ❤️ especially valuable · 👎 wrong - reply with why

onbuyuka added 2 commits July 13, 2026 13:45
The country builds compile the test app with CodeCop treating AA0137 as an error. DaysFromNow used exit() with a named return variable 'Result' left unassigned; drop the name.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
Correct the doc comment: the table lock serializes refresh/migration per company only (the store table is DataPerCompany), not across companies. Document that the lock is deliberately held across the token request and that cross-company safety relies on Shopify's idempotent-response window.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
@github-actions

Copy link
Copy Markdown
Contributor

$\textbf{🟡\ Medium\ Severity\ —\ Agent} \quad \color{gray}{\texttt{\small Iteration\ 7}}$

In ShpfyCommunicationMgt, after HandleUnauthorizedResponse forces a token refresh on a 401, the code Clears HttpResponseMessage and re-sends the request but discards the boolean return value of the retry's HttpClient.Send(HttpRequestMessage, HttpResponseMessage) call - unlike the original Send() a few lines above, whose return value is checked.

If the retry Send fails to obtain a response (e.g. a transient network failure right after the 401), HttpResponseMessage is left as the just-Cleared, empty default instance. The subsequent while-loop condition (IsBlockedByEnvironment / EvaluateResponse, both driven by HttpStatusCode()) then evaluates a status code of 0, which matches none of EvaluateResponse's retry branches (429, 500-599), so the loop silently exits and the caller receives a blank/default response instead of a clear signal that the retried request never completed.

Recommendation:

  • check the retry Send()'s return value the same way the first Send() is checked, and surface/handle the false case explicitly rather than falling through to evaluate an empty response.

Line mapping was unavailable, so this was posted as an issue comment.

👍 useful · ❤️ especially valuable · 👎 wrong - reply with why

Comment thread src/Apps/W1/Shopify/App/src/Integration/Codeunits/ShpfyTokenRefresh.Codeunit.al Outdated
onbuyuka added 3 commits July 13, 2026 14:18
Declare JobQueueCategoryLbl and JobDescriptionTxt in the codeunit's top-level var block so they are reliably extracted for XLIFF/translation and visible at object-level review.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
The access token is an OAuth credential of the same sensitivity as the refresh token, which this PR already stores encrypted. SetAccessToken now uses SetEncrypted for parity; existing unencrypted values remain readable via IsolatedStorage.Get, and future writes are encrypted.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
After a 401 forces a token refresh, the retried Send's boolean result was discarded; a transient failure then left an empty response that the retry loop would evaluate as status 0. Track the send result and gate the retry loop on it so a failed (re)send exits cleanly instead of evaluating an empty response.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
@onbuyuka

Copy link
Copy Markdown
Contributor Author

Addressed the latest review findings:

  • Encrypt access token at rest (Security, iters 2 & 4): SetAccessToken on Shpfy Registered Store New now uses IsolatedStorage.SetEncrypted, matching the refresh token. Existing unencrypted values remain readable via IsolatedStorage.Get; future writes are encrypted. (14e8487... / access-token commit)
  • Retry Send() result after 401 (Agent, iter 7): ExecuteWebRequest now tracks the send result and gates the retry loop on it, so a transient failure on the post-refresh re-send exits cleanly instead of evaluating an empty (status 0) response. Also hardens the pre-existing retry-loop resend.

The cross-company-same-store serialization finding remains intentionally open for reviewer input (a tenant-wide token store is a larger change than this migration slice).

Codeunit.Run cannot nest inside an open write transaction. On a fresh install AddRetentionPolicyAllowedTables (and on upgrade the earlier per-company steps) write before ScheduleRefreshJob runs Codeunit.Run(Job Queue - Enqueue), which would throw at runtime. Commit before the guarded enqueue; this is a one-time install/upgrade action, not a loop.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
gggdttt
gggdttt previously approved these changes Jul 15, 2026
…esh job

COMMIT (explicit or implicit) is not allowed inside install/upgrade triggers; the Commit added in 2260086 crashed OnInstallAppPerCompany (fresh install). Codeunit 'Job Queue - Enqueue' performs no implicit commit, so no prior commit is needed - matching the platform's own JobQueueEntry.EnqueueTask and the existing ShpfyBackgroundSyncs pattern. The guarded Codeunit.Run still prevents an enqueue failure from aborting the install/upgrade.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5fe0c18c-7f03-4f1f-880f-9adb9eb1dfcc
begin
if RefreshTokenExpired(RegisteredStoreNew) then begin
Session.LogMessage('0000UIY', TokenRefreshExpiredTxt, Verbosity::Normal, DataClassification::SystemMetadata, TelemetryScope::ExtensionPublisher, 'Category', CategoryTok);
Error(RefreshTokenExpiredErr, Store);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$\textbf{🟡\ Medium\ Severity\ —\ Error\ Handling} \quad \color{gray}{\texttt{\small Iteration\ 11}}$

RefreshAccessToken raises a plain Error(RefreshTokenExpiredErr, Store) whose message explicitly names the fix ('Open the Shopify Shop card and reconnect the store to continue') but offers no way to get there.

The PR already built the pieces for a Show-it/Fix-it action for this exact scenario (Shpfy Authentication Mgt.ReconnectFromNotification and the Shop Card notification action), so when this error surfaces in an interactive context it dead-ends the user instead of using ErrorInfo with AddNavigationAction (Page::"Shpfy Shop Card") or AddAction pointing at the existing reconnect handler. Wrapping the three Error(RefreshTokenExpiredErr, Store) call sites in an ErrorInfo with a navigation/fix-it action would make the error self-service instead of a dead end.

Knowledge:

👍 useful · ❤️ especially valuable · 👎 wrong - reply with why

@JesperSchulz JesperSchulz added the Integration GitHub request for Integration area label Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AL: Apps (W1) Add-on apps for W1 Integration GitHub request for Integration area

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants