Are Your Web Pages Spider Friendly?

Unlike human visitors, who view the page in a Web browser, search engine spiders analyze a page by reading the HTML in the page source. Pages that are filled with images, videos, or Flash may provide a great user experience, but search engines may not be able to index the content on your page if that's all it contains. For search engine optimization purposes, it is best to give the engines simple text so that they can accurately determine the relevance of your pages' content. It is extremely important that you consider how a search engine spider would respond to any site changes you plan on making.

Google actually provides a useful set of guidelines that will help you build search engine friendly Web pages. You can view all of Google's suggestions in their Webmaster Guidelines, but the most important guideline for landing page SEO is as follows:

Use a text browser such as Lynx to examine your site, because most search engine spiders see your site much as Lynx would. If fancy features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling your site.

SEO Browser is one of the better Web-based text browsers that we have found. However, it does not consider everything that Google does. If you are evaluating the SEO friendliness of an existing page on your site, then we recommend viewing Google's cached text version of that page.

How to view Google's cached text

If you are evaluating the spiderability of a page that has already been indexed by Google, then determining what is and is not spiderable is a relatively simple process if you view Google's cached text version of the page. There are a few ways you can get to the cached text of a page, which we will now describe.

Method 1: Search in Google for something that will return the page you would like to analyze. You can copy text from the page and search for it or or simply search for the URL of the page. If you are going to view the cached text of your home page, querying your domain or company name should work well. Once you are in a SERP that shows your page, then you will want to click on the link that says "Cached" in your listing. Here is an example showing the SEO Cipher home page:

Screenshot of link to cache of SEO Cipher

Once you click on the link shown in the above screenshot, you will be viewing Google's full cache of your page. You will then want to click on the "Text-only version" link, which we have circled in the screenshot below.

Screenshot of full cached version of SEO Cipher

If you have followed along with the previous steps, then you should now be viewing the cached text of your page. For comparison, here is the cached text of the SEO Cipher home page:

Cached text of spider friendly SEO Cipher home page

Method 2: It is also possible to query Google so that you are taken directly to the full cached version of your page. You can do this by using the cache: command. To use this command you you would simply enter the following query in the Google search bar, replacing the bolded portion with your page's URL:

Assuming that the URL you enter is a valid page that has been indexed, this command will take you directly to the full cache of your page. You would then simply click on the "Text-only version" link as described in Method 1 to view the page's cached text.

Method 3: Finally, you could also just copy and paste the following URL into your browser's address bar, replacing the bolded portion with your page's URL:

This method takes you directly to Google's Text-only version of your page.

Analyzing the Cached Text

We recommend comparing Google's text-only version of your page with the full page. Is there any content that appears on your page that does not appear in the cached text? If so, then you will have to investigate what may be causing the indexing problem by viewing the HTML source of the page.

Remember that if your page content is contained within images, frames, JavaScript or Flash, then it is unlikely to get indexed. We always recommend displaying content as simple text for search engine optimization purposes. Even though Google has made some significant improvements in their spidering ability, we do not believe that you should rely completely on that ability. The odds are good that if your page's content does not appear in Google's Text-only version, then you are not getting benefit from it as an on-page SEO ranking factor.

Home | Services | SEO Basics | Quote Request | Ask your Questions | Step-by-Step SEO Guide

Offering search engine optimization in the Los Angeles Area - Call us at 805.558.3542 for a free consultation or request a free quote.

© 2015 SEO Cipher. All Rights Reserved | Privacy Statement