diff --git a/domain-skills/x/articles.md b/domain-skills/x/articles.md new file mode 100644 index 00000000..16c59e5c --- /dev/null +++ b/domain-skills/x/articles.md @@ -0,0 +1,131 @@ +# X (Twitter) — Reading Articles & Tweets + +Field-tested 2026-06-30 against an X **Article** (long-form post) using a +**logged-in** browser profile. A *"See what's happening"* login modal still +rendered on top of the page, but the full article text was in the DOM behind +it — the modal blocked clicks, not reads. **Logged-out behavior is +unverified**: X removed guest access years ago, so a truly anonymous browser +may hit a real wall. If you land on a login modal, try the DOM read below +before concluding you need auth — and never type credentials yourself +(auth wall → stop and ask the user). + +--- + +## TL;DR + +X **Articles** (long-form posts) live at the **same URL shape as a tweet** — +`https://x.com/{handle}/status/{id}` — there is no `/article/` path. Read the +body via `document.body.innerText`; the login modal that may overlay the page +does not block DOM reads. + +The one trap that wastes time: **navigate and extract in the same +`browser-harness -c` invocation.** A second, separate call can re-attach to a +stale or different tab and you read the wrong page (see Gotchas). + +--- + +## Approach (Recommended): innerText read + +```python +# ONE invocation — navigate + hydrate + extract together. +new_tab('https://x.com/mvanhorn/status/2070966613994795489') +wait_for_load() +wait(4) # JS hydration; article body is lazy +title = js('document.title') # ' on X: "" / X' +body = js('return document.body.innerText') +open('/tmp/x_article.txt', 'w').write(body) +print(title) +print(len(body)) # full article ~18K chars +``` + +- `document.title` carries the headline cleanly: + `Matt Van Horn on X: "Your AI's Memory Is Quietly Making It Dumber (...)" / X`. + Strip the ` on X: "` prefix and `" / X` suffix to get the title. + ⚠️ If the title starts with `🟢 ` that is the **harness's own tab marker** + (added by `new_tab()`), not part of X's title — strip it first. +- `document.body.innerText` contains the whole page. Layout, top → bottom: + 1. **Left-nav chrome** — `Home / Explore / Notifications / ... / Profile / More / Post` + (when logged in, followed by the account's name/handle). + 2. **Article header** — `Article`, author name + `@handle`, `Subscribe`, + then the **headline**, then bare engagement counts. + 3. **Article body** — the actual prose (this is what you want). + 4. **Footer** — the timestamp line (`2:24 PM · Jun 27, 2026`), view count, + reply/repost/like counts, `Relevant people`, `Trending now`, `Terms · Privacy …`. + +### Slicing the body out of the chrome + +The body sits between the headline and the timestamp line. Cheap, robust cut: + +```python +import re +def x_article_body(full_text, headline): + # start just after the headline's first occurrence in the content area + i = full_text.find(headline) + start = full_text.find('\n', i) if i != -1 else 0 + # end at the post timestamp ("H:MM AM/PM · Mon DD, YYYY") — search AFTER + # start, or a timestamp-shaped string earlier in the page truncates the body + m = re.search(r'\d{1,2}:\d{2}\s*[AP]M\s*·\s*\w+ \d{1,2}, \d{4}', + full_text[start:]) + end = start + m.start() if m else len(full_text) + return full_text[start:end].strip() +``` + +Note: a timestamp-shaped string *inside the article prose* would still end the +slice early — if the result looks short vs. `len(full_text)`, fall back to the +footer anchors (`Relevant people` / `Trending now`) as the end marker. + +--- + +## Regular tweets (not Articles) + +For an ordinary tweet, the body is in `[data-testid="tweetText"]`: + +```python +js('var e=document.querySelectorAll("[data-testid=tweetText]");' + 'var a=[]; for(var i=0;i