Google not indexing the Makeability Lab website

[jonfroehlich]

For some reason, the Makeability Lab website is not being indexed correctly by Google. I noticed that if you search for a project like "HandSight Makeability Lab" that none of the results include the actual HandSight project page.

Perhaps relatedly, I recently received this notification from Google.

Google has validated your fix of Page indexing issues on site https://makeabilitylab.cs.washington.edu/. The specific issue validated was: Duplicate without user-selected canonical.

I then followed the link and it brought me here

More investigation is needed.

[jonfroehlich]

OK, if you use this Search Console Inspect tool on Makeabiltiy Lab urls like HandSight, you can see that robots.txt is not available and so not indexing our website

[jonfroehlich]

In fact, the entire Makeability Lab website does not appear on Google atm:

it appears to be on Bing:

[jonfroehlich]

Matt got back to me and said he tried to fix this. Something appears to now be working as I can at least submit the Makeability Lab website to be indexed:

[jonfroehlich]

This continues to be a problem. I can seem to get content indexed if I specifically add the page; however, as of now, we have 606 pages that are not indexed on the Makeability Lab website: 513 of these are marked as "crawled by not currently indexed"

You can see that we started to lose our indexing around Nov 2023:

[jonfroehlich]

This continues to be an issue for some pages:

For example, I searched "makeabilitylab handsight" and it didn't come up correctly in Google (interestingly, some newer projects are fine like "makeabilitylab mobiprint")

So, now trying to add handsight project manually

And other manually:

But many projects I spot checked were already on there...

[jonfroehlich]

FWIW, it does seem like we're now better indexed by Google. And the original example (HandSight) is now indexed well:

[SH20RAJ]

If Google Search Console is throwing 'Discovered - currently not indexed', it usually means Google is just rationing crawl budget for your domain right now. Instead of waiting weeks for Googlebot to naturally return, you can try forcing a universal ping via the IndexNow API. If you don't want to set up the API handshake manually, running the sitemap through something like IndexFast will blast the ping to Bing/IndexNow and hundreds of other engines simultaneously, which often forces Google to cross-reference and wake up their crawler faster.

[jonfroehlich]

We now have a sitemap.xml file in Google Search Console:

[jonfroehlich]

Progress — shipped in 2.12.0 (PRs #1330, #1332). These target the "Crawled – currently not indexed" signal class directly:

• Distinct per-page <meta name="description"> on every page type (was the same generic description site-wide — a classic thin/duplicate-content driver).
• Self-referential canonical URLs, pinned to the singular /project/<short_name>/ form (matches the sitemap), collapsing the /project/ vs /projects/ duplication to one indexable URL.
• schema.org JSON-LD: Organization (home), Person w/ sameAs (member pages), NewsArticle (news).
• Fixed http→https for canonical/OG behind the proxy (Project page social media (opengraph) not working perfectly #1236) so we no longer advertise non-canonical http:// URLs.

This is a gradual, GSC-driven recovery — keeping this open to monitor the Pages → "Crawled – currently not indexed" bucket over the coming weeks now that prod is on 2.12.0. The robots.txt/sitemap hard-blockers were already resolved earlier. Durable proxy fix: #1329.