Modern search engines, increasingly powered by artificial intelligence, demand a sophisticated approach to search engine optimization. While fundamental technical configurations remain essential, the ability of AI to understand, process, and rank content hinges on how effectively websites present information, manage their technical infrastructure, and provide a superior user experience. Analyzing trends in web development reveals both progress and persistent challenges in meeting these evolving demands.
The Evolving Field of Search Indexing
For search engines to rank content, they must first discover and understand it. This process, known as crawlability and indexability, relies on foundational elements like robots.txt files and canonical tags. Data from 2020 shows a slight improvement in robots.txt usage, with 75% of mobile sites having a valid file, up from 72.16% in 2019. This indicates better control over what parts of a site search engines can access.
However, complexities arise with dynamic content and JavaScript rendering. While meta robot tags appear on 28% of desktop and mobile pages, a small percentage, 0.16%, saw these tags changed by JavaScript during rendering. This can be problematic because if a page loads new indexing instructions via a meta robot tag after the initial HTML is processed, search engines might not execute the JavaScript to see the updated directive. This means a page intended to be indexed might be missed, or vice versa, if the initial HTML contains a noindex tag that is later removed by JavaScript.
Canonical tags, which tell search engines the preferred version of a page, also show mixed setup. In 2020, 53.6% of mobile pages used canonical tags, an increase from 48.3% in 2019. On mobile, 45% were self-referencing, while 8.5% pointed to a different URL. For desktop, 52% of pages had canonical tags, with 48% self-referencing and 4% pointing elsewhere. A notable issue is the conflict between canonical tags specified in the original HTML and those implemented or changed by client-side JavaScript. For instance, 0.7% of mobile pages and 0.5% of desktop pages included canonical tags only in the rendered DOM, not the original HTML. This reliance on JavaScript for a critical indexing signal can create mixed messages for search engines, potentially leading to indexing issues. Similarly, 0.15% of mobile pages and 0.17% of desktop pages showed conflicts between canonical tags in HTTP headers and the HTML head. These discrepancies highlight the need for consistent and clear signals to AI-powered indexing systems.
Content Quality and Presentation in an AI-Driven Era
The substance and presentation of content are paramount for engaging users and informing search algorithms. In 2020, the median word count for desktop pages was 402 words, a 16.2% increase from 346 words in 2019. Mobile pages also saw growth, with a median of 348 words, up 13.7% from 306 words. While content length has increased, the emphasis remains on providing complete and useful information that directly addresses user search needs, rather than simply adding more words.
The way content is delivered also affects its visibility. Desktop websites showed a 12% increase in median word count when rendered compared to their original HTML. Mobile sites displayed 11.5% more words when rendered than in their raw HTML, yet they still had 13% less content overall than desktop sites. This reliance on client-side JavaScript for content display means that while search engines are improving their ability to render and index JavaScript, some websites might miss opportunities for organic search visibility if their content isn’t consistently available and indexable in its initial form.
Title tags are nearly universal, with 99% of both desktop and mobile pages including them, a slight improvement from 97% of mobile pages in 2019. The median page title was six words and 38 characters on both mobile and desktop. Meta descriptions, which offer a brief summary for search results, were present on 68.6% of desktop pages and 68.2% of mobile pages, a modest rise from 64% of mobile pages in 2019. The median length was 19 words, with 138 characters on desktop and 136 on mobile. These lengths are generally below the commonly referenced 160-character guideline, suggesting room for more descriptive summaries.
Strategic Linking for Discovery and Authority
Links are fundamental to how search engines discover new content and assess its authority. In 2020, desktop pages had a median of 76 links, while mobile pages had 67. A concerning trend is the decrease in internal links, which help search engines crawl a site and distribute “link equity.” The median number of internal links on desktop pages dropped to 61, a 12.8% decrease from 2019, and on mobile pages to 54, a 10% decrease. And, 5.6% of desktop pages and 6% of mobile pages contained no internal links at all. This suggests many sites are not fully optimizing their internal linking structures, potentially hindering crawlability and the flow of authority.
External links, which point to other websites, also saw a decline. Desktop pages linked to external sites a median of 7 times, down from 10 in 2019. Mobile pages had a median of 6 external links, down from 8. This reduction might indicate increased caution from websites about passing on link popularity or recommending external resources.
A notable disparity exists between mobile and desktop linking, with mobile pages having a median of 62 links compared to 68 on desktop. Given search engines’ shift towards mobile-first indexing, these differences can negatively impact a site’s performance. The adoption of rel="nofollow" attributes was seen on 28.6% of desktop pages and 30.7% of mobile pages. However, newer attributes like rel="ugc" (user-generated content) and rel="sponsored" were used on less than 0.3% of pages. While these newer attributes add semantic information that AI algorithms can use, their slow adoption suggests publishers may not yet see major additional value over nofollow.
On a positive note, the discoverability of links within major JavaScript frameworks for single-page applications greatly improved. Testing mobile navigation links with hash URLs showed a 53% reduction in unhashable links for React, a 58% reduction for Vue.js, and a 91% reduction for Angular-driven sites compared to the previous year. This improvement is important as more websites rely on JavaScript for dynamic content and navigation.
Using Structured Data for Enhanced Visibility
Structured data provides search engines with explicit cues about the meaning of content, which is increasingly vital for AI-driven understanding and rich search results. JSON-LD has emerged as the preferred format, appearing on 29.8% of mobile pages and 30.6% of desktop pages. Overall, 38.6% of desktop pages and 39.3% of mobile pages presented structured data in their original HTML, while 40.1% of both desktop and mobile pages showed it in the rendered DOM.
However, JavaScript’s role in structured data can also introduce complications. For instance, 1.5% of desktop pages and 1.8% of mobile pages only presented structured data in the rendered DOM, relying entirely on JavaScript execution. And, 4.5% of desktop pages and 4.6% of mobile pages had structured data in their original HTML that was then altered by JavaScript during rendering. These scenarios can generate mixed signals for search engines, potentially leading to misinterpretation or missed opportunities for rich snippets.
Specific types of structured data have seen remarkable growth, often driven by search engines offering enhanced visibility features. VideoObject usage, for example, grew by 30.11% on desktop and 27.7% on mobile. Even more dramatic increases were observed for schema types like FAQPage, HowTo, and QAPage, which gained major visibility in search results. FAQPage markup grew by 3,261% on desktop and 3,000% on mobile. HowTo markup increased by 605% on desktop and 623% on mobile, while QAPage grew by 167% on desktop and 192% on mobile. While these figures primarily reflect growth on internal pages rather than homepages, they underscore the impact of using structured data to enhance content presentation in search results.
Page Experience and Core Web Vitals
User experience signals are increasingly integrated into search engine ranking algorithms, especially with the introduction of Core Web Vitals. Analysis shows that desktop remains the higher-performing platform for user experience, with 33.1% of desktop sites achieving good Core Web Vitals scores, compared to only 20% of mobile sites. This disparity highlights a major challenge for mobile optimization, particularly given the prevalence of mobile browsing.
Security is one area of improvement, with HTTPS adoption reaching 77.4% on desktop pages and 73.2% on mobile pages, representing a 10.4% increase from 2019. This rise is partly due to aggressive browser warnings for non-HTTPS pages and the requirement of HTTPS for high-performance protocols like HTTP/2.
Mobile-friendliness, a direct factor in mobile-first indexing, still presents issues. While 42% of both mobile and desktop pages had a correctly configured viewport meta tag, a large 11% of mobile pages and 16.2% of desktop pages lacked this tag entirely. This suggests many sites are not yet fully optimized for mobile users. On the positive side, 80.3% of desktop pages and 83% of mobile pages used CSS configurations for height, width, or aspect ratio, indicating a high degree of responsive design setup. However, for desktop sites with separate mobile versions, only 0.64% used the rel="alternate" tag with a specified media value, an important signal for search engines to understand the relationship between these versions.
In summary, while many websites are adopting basic SEO configurations and showing improvements in areas like HTTPS and structured data usage, major opportunities for search visibility and growth are still being missed. The disparities between desktop and mobile performance, the reliance on client-side JavaScript for content, and inconsistencies in technical setups all point to an ongoing need for organizations to prioritize and adapt their SEO strategies to the dynamic, AI-driven search environment.
