Discovered, Not Indexed: Googlebot Skipped 5 of 13 Posts

Search Console has a status called Discovered – currently not indexed. It means Google knows a URL exists and has chosen not to fetch it yet. The report tells you which URLs are in that state and nothing else: not when Google learned about them, not what it fetched instead, not whether it is coming back. If you want those answers you have to read the one record Google cannot edit, which is your own server’s access log.

I read this site’s. It runs WordPress 7.1 behind Cloudflare on ordinary cPanel hosting, and it published 42 posts between 1 August and 12 September. For every one of them the log says when the post went live and when the first request from a verified Googlebot address arrived. For 29 of them, the August posts, that gap has a median of 15 hours. For the 13 posts published from 30 August onward, five had never been requested at all when I read the log on 13 September, at ages of 5, 7, 9, 10 and 14 days, and the eight that were requested took a median of three days. In the same 15 days Googlebot fetched the XML sitemap that lists all five 476 times, roughly once every 45 minutes.

Bing, for comparison, fetched every one of the 42 posts within a median of 18 seconds of publication, because the SEO plugin tells it to. Google has no equivalent for ordinary pages, and this post is about what that looks like from the server side, how to read it on your own site, and what actually moves a URL out of the discovered pile.

What the status means, and what the log adds

Google’s own documentation is short on this one. Discovered – currently not indexed: the page was found, typically through a sitemap or a link, and not crawled yet, and the documented reason is that Google wanted to crawl it but expected doing so would overload the site, so it rescheduled. On a site whose origin answered 2,465 Googlebot requests in the same fortnight with 2,304 of them 200 and not one 5xx, overload is not what happened; the URL simply never reached the front of the queue. The sibling status, Crawled – currently not indexed, means the fetch happened and Google chose not to index what it got. The two need different fixes, and Search Console will move a URL from one to the other without saying when.

The access log resolves both questions with one grep. A request from a Googlebot address for the URL means the page has left the discovered state; the timestamp says when. No such request, days after the sitemap listed the URL and while Googlebot is visibly fetching that sitemap, is the discovered state observed directly. And the log adds the one thing Search Console never shows: what Googlebot fetched instead, which on this site turned out to be the more useful number.

There is a precondition. The log has to carry the visitor’s real address. This site sits behind Cloudflare, which would normally leave Cloudflare’s own addresses in the log; the host restores the originating IP, so 66.249.x.x appears as itself. If your log is full of 104.16.x.x and 172.71.x.x entries, that restoration is off and you need to turn it on before any of this works. The other precondition is that Cloudflare is not answering Googlebot from its cache: on this site HTML is served cf-cache-status: DYNAMIC, so every page request Google makes reaches the origin and lands in the log. Static files can be cached at the edge, which is why the asset counts below are a floor.

The August baseline: 29 posts, 29 fetched, median 15 hours

The first thing to establish is what normal looked like, because a five-out-of-thirteen figure means nothing without it. The log I had covers 31 July onward, so August gives a clean month.

Every one of the 29 posts published between 1 and 29 August was requested by a verified Googlebot address. The median wait was 14.6 hours; 19 of the 29 were fetched inside a day; the mean was dragged to 46 hours by a handful of slow ones. The slowest was a REST API security audit published on 8 August that Googlebot did not touch for 381 hours, sixteen days. I already knew that post as the one page on the site that has never recorded a Search Console impression in its life. The log now says why the first month of that life was quiet: for sixteen of its days there was nothing to show.

Twenty-eight of the 29 August posts have impressions in the current 28-day window. The one that does not is the one that waited sixteen days. That is one data point, not a law, but it is the right shape.

30 August onward: five posts Google never requested

The break is sharp. Here is every post from 30 August to 12 September, with the first verified Googlebot request, the first verified Bingbot request, and the Search Console impressions for the page through 10 September, the newest day the API had.

Three things in that table are worth separating, because they are three different problems.

The five never rows are Discovered – currently not indexed, observed from the server. Every one of those URLs was in the post sitemap within minutes of publishing, and I verified each day that it was listed exactly once. Googlebot read those sitemap files 476 times across the window. It knew the URLs. It did not want them.

The DeepSeek row is the other state. Googlebot fetched it on 5 September, twice within twenty seconds, and five days of Search Console data later the page has no impressions. That is what Crawled – currently not indexed looks like before the report says so, and it is the harder one: Google saw the page and passed. The four rows fetched between 40 and 245 hours after publishing are too young to call either way; a page fetched on 10 September cannot show impressions in data that ends on 10 September.

The last two rows are the encouraging part. The 11 and 12 September posts were fetched in 9 and 10 hours, back on the August pace. Whatever changed around 30 August is not a permanent setting, and the last two days suggest it has partly reversed. I do not know what it was. Nothing on the site changed on that date: no plugin update, no robots.txt edit, no Cloudflare rule, no change to how posts are created. The posts from both eras are made the same way, with the same template, no category and no tags. The change is on Google’s side of the connection, which is where crawl scheduling lives.

Why Bing takes 18 seconds

The Bing column is not Bing being clever. Rank Math’s Instant Indexing module submits every published post and page to IndexNow, the push protocol Bing, Yandex, Naver and Seznam share, the moment WordPress fires the publish hook. Bingbot arrives 10 to 20 seconds later, from an address that reverse-resolves to search.msn.com, every single time; the two outliers at 132 and 176 seconds are still under three minutes. All 42 posts, no exceptions, no sitemap involved.

Google does not participate in IndexNow. Google’s Indexing API exists but is restricted to pages carrying JobPosting or BroadcastEvent structured data, and plugins that push ordinary posts through it are operating outside its terms. For a normal blog post the only push Google offers is the Request indexing button in Search Console’s URL Inspection tool, one URL at a time, by hand, with a daily quota. Everything else is pull: Google reads your sitemap and your links and decides.

Which brings up the number that surprised me most. Google read this site’s sitemap files 476 times in 15 days and read the five URLs it was skipping zero times. Discovery was never the problem. Sitemap pings, sitemap resubmission, sitemap plugins, sitemap anything: the URLs were discovered within the hour, and they were discovered again 475 times after that. Being discovered is the state the report is describing.

Where the 2,465 requests went instead

The fetches Googlebot did make are the useful half of the story. Across the same 15 days the log holds 2,465 requests from verified Googlebot addresses, about 164 a day. Classified by what they asked for:

Three lines in that chart matter for a site owner.

879 page requests across 303 distinct URLs. The site has 413 published posts and a few dozen pages. In 15 days Google re-read 179 of the posts, 43% of the catalogue, plus most of the pages, some of them repeatedly: the homepage 45 times, the privacy policy 41 times, the main chatbot-plugin comparison 28 times, a hotel-booking-plugin listicle from 2024 fifteen times, the documentation hub and its two busiest add-on pages eight to ten times each. This is not a site Google is rationing. It is a site Google is happy to re-crawl, while declining to crawl the newest 5 of 13 posts once.

247 archive requests, 234 of them tag pages. The fifth most-fetched URL on the whole site is /tag/best-free-backup-plugin-for-wordpress/, at 13 fetches. It is a tag archive, it is set to noindex, and it currently contains no published posts at all. The site has 1,305 tags, 438 of them empty; Google fetched 149 tag archives 234 times in the window, 57 of those fetches on ten archives that are empty. That is more than twenty times the total it spent on the 13 new posts combined. Crawl demand follows what Google already has, not what you have just added.

476 sitemap requests and 223 admin-ajax POSTs. The sitemap figure is Google’s polling rate on this site, one read per 45 minutes, and it is the reason I am confident the five posts were discovered. The 223 POSTs to admin-ajax.php are Googlebot rendering pages: the chatbot widget and the theme make AJAX calls when the page loads, and the rendering service makes them too. They are not wasted crawl, but they are a fifth of the non-asset total, and they are a reminder that every JavaScript call on your page is a request Google makes.

The inbound links were there, from pages Google does not visit

The standard advice for Discovered – currently not indexed is internal links. The five skipped posts had them. Between them they carry 27 inbound links from 17 distinct published pages, added on the day each was published, in context, with descriptive anchors. Here is how many times Googlebot fetched each linking page in the same 15 days.

Eight of the 17 linking pages were themselves not fetched once in the window, and the best of them managed six. The pricing posts in particular link to each other in a chain: each new one got links from the previous three, which were themselves new, which Google was not fetching either. Twenty-seven links inside a closed cluster of unfetched pages are, for discovery purposes, zero links. The homepage, fetched 45 times, links to no blog posts at all; the blog index, which does list the newest ten, was fetched three times in 15 days, thirteen counting its paginated pages. So the crawl paths that Google actually walks on this site lead to 2024 content and tag archives, and the paths to the new posts run through other new posts.

That is the actionable finding, and it is not specific to this site. If your internal links to a new page come from pages Google fetches every few weeks, the link is real but the discovery signal arrives every few weeks. A link from a page Google fetches every day is a different object.

Two states, two fixes

Because the log separates the states, the fixes separate too.

Discovered, not fetched is a priority problem. Google has the URL and has ranked it below everything else it wants to re-read. The levers are the ones that raise that priority: a link from a page in Google’s frequent-crawl set, a manual Request indexing in URL Inspection, and, honestly, fewer new URLs per day if you are adding them faster than the site’s crawl demand is growing. This site added 13 posts in 14 days, on a domain where Google was already re-crawling 300 pages a fortnight. The five skipped ones were not the worst five; one of them was an accessibility piece sitting in a run of pricing pieces. Priority is not a quality verdict on a single URL. It is a queue.

Crawled, not indexed is a content decision, and the DeepSeek post is the specimen. Google fetched it twice and did not index it. Nothing about the URL’s priority helps here; Google would fetch it again tomorrow and reach the same conclusion. That state answers to the page itself: whether it is substantively different from what Google already indexes, on this site and on the web. Ten posts on model API pricing in a fortnight, each built on the same measured-cost table, is a pattern a search engine is entitled to call one page’s worth of information, however carefully each was written. I am not going to pretend the log does not say that.

The earlier finding on this site was that pages Google does index and does show on page one still earn almost no clicks, because the queries are AI-shaped rather than human-shaped. This is the layer underneath it: before a page can fail to earn a click, it has to be fetched, and that is not guaranteed either.

How to read this on your own site

You need shell access to the origin, the raw access log, and about ten minutes. On cPanel the live log is ~/access-logs/example.com-ssl_log and the monthly archives are ~/logs/example.com-ssl_log-Sep-2026.gz; other hosts keep them under /var/log/apache2/ or /var/log/nginx/.

First, has verified Googlebot ever requested the URL:

( zcat ~/logs/example.com-ssl_log-Sep-2026.gz; cat ~/access-logs/example.com-ssl_log ) \
  | grep '^66\.249\.' | grep -i googlebot | grep 'GET /your-new-post/ '

An empty result, on a URL your sitemap has listed for days, is Discovered – currently not indexed observed directly. A result with a timestamp is the moment the page left that state; if Search Console still shows no impressions a week later, you are looking at the crawled-not-indexed case instead.

Second, filter by address, not by user agent. In the same 15 days this log holds 1,244 requests carrying a Googlebot user agent from addresses that are not Google’s, a third of all Googlebot-labelled traffic. The largest single source, 281 requests, has no reverse DNS at all; one fake even visited a post from a Google Cloud VM, which is Google’s network and not Googlebot. Restricting to 66.249.0.0/16 catches nearly all real Googlebot traffic; to be exact, resolve the address:

host 66.249.72.70
# 70.72.249.66.in-addr.arpa domain name pointer crawl-66-249-72-70.googlebot.com.

and check that the name resolves back to the same address. I wrote up the full verification method, with the ranges for the AI crawlers too, when this site turned out to be serving nearly half its “Googlebot” traffic to impostors.

Third, count where the real requests go. A one-liner over the same filtered log, cutting the path and counting, gives you the chart above for your own site in a few seconds. What you are looking for is the ratio between fetches of the pages you care about and fetches of pages you do not: tag archives, author pages, paginated lists, date archives. If the second group dominates, your new posts are competing for crawl against pages you could noindex tomorrow.

Fourth, check the posts’ first-fetch latency against your own baseline before deciding anything is wrong. One slow post is noise. A month of posts fetched within a day followed by a fortnight in which a third are never fetched is a signal, and the only way to see it is to have the earlier month to compare against, which is a reason to run this before you need it.

What I am changing

Three things, each of which I can measure against the same log next week.

The five skipped posts each got one contextual link today from a page in Googlebot’s frequent-crawl set: the pages it fetched seven to ten times in the window, which on this site are older setup tutorials, a small-business chatbot listicle and the chatbot use-case guide. If those links do what a link from a fetched page should, the five URLs appear in the log within days, and the number of days is the result.

The publishing pattern changes. New posts will get at least one inbound link from a page Google actually visits, not only from the last three posts. And the run of near-identical model-pricing pieces stops, because the log has now said the same thing Search Console was going to say in a month, only sooner.

URL Inspection’s Request indexing is the one push Google offers, and it is a hand operation in the Search Console UI. The plan is to use it on two of the five and leave the other three to the links alone, so that next week’s log can say whether the button or the link did the work. I will report what it says either way.

The chatbot on this site, for what it is worth, has been answering questions from all thirteen posts since the minute each was published, because it indexes the site’s own content and does not wait for anyone’s crawler. That is not a substitute for being in Google. It is a reminder that being in Google is a decision Google makes, on its own schedule, from a queue you cannot see, and that the only instrument you own that watches that queue is the log file your host has been writing the whole time.

Frequently asked questions

How long should a new WordPress post take to be crawled by Google?

On this site, with an XML sitemap Google polls every 45 minutes, the August median was 15 hours and two thirds of posts were fetched within a day. There is no published Google figure. Establish your own baseline from the log; the useful signal is a change from it, not the absolute number.

Does resubmitting the sitemap fix Discovered – currently not indexed?

Not if the log shows Google already reading the sitemap. Here it read the files 476 times in 15 days while skipping five of the URLs in them. The status describes a page Google has discovered and deprioritised; discovering it again does not change the priority.

Is Discovered – currently not indexed a quality problem?

Sometimes, indirectly. The status itself is about crawl priority, and priority is set at the site level as much as the page level. Crawled – currently not indexed is the status that speaks to an individual page’s content. The access log tells you which of the two you have.

Can I push a post to Google the way IndexNow pushes it to Bing?

No. Google does not accept IndexNow, and its Indexing API is limited to job postings and livestream events. The URL Inspection tool’s Request indexing button is the only per-URL push for ordinary pages, and it is manual and quota-limited.

Why does my log show Googlebot requests from addresses outside 66.249?

Most are not Googlebot. A third of the Googlebot-labelled requests on this site in 15 days came from unrelated addresses. Verify by reverse DNS to googlebot.com or google.com and a matching forward lookup before counting a request as Google’s.