SEO Tools : Search Engine Spider Simulator

(Search Engine Spider Simulator) A lot of Content and Links displayed on a webpage may not actually be visible to the Search Engines, eg. Flash-based content, the content generated through javascript, the content displayed as images, etc.

SEO Tools : Search Engine Spider Simulator

How it Works

(Search Engine Spider Simulator) A lot of Content and Links displayed on a webpage may not actually be visible to the Search Engines, eg. Flash-based content, the content generated through javascript, the content displayed as images, etc.

This tool Simulates a Search Engine by displaying the contents of a webpage exactly how a Search Engine would see it.

It also displays the hyperlinks that will be followed (crawled) by a Search Engine when it visits the particular webpage.


Search Engine Spider Simulator

Enter URL to Spider


See Your Site With the Eyes of a Spider

The article explains how Search Engines view a Webpage.

Making efforts to optimize a site is great but what counts is how search engines see your efforts. While even the most careful optimization does not guarantee tops position in search results.

If your site does not follow basic search engine optimization truths, then it is more than certain that this site will not score well with search engines. One way to check in advance how your SEO efforts are seen by search engines is to use a search engine simulator.

Spiders Explained

Basically, all search engine spiders crawl the Web and index pages, which are stored in a database, and later use various algorithms to determine page ranking, relevancy, etc of the collected pages.

While the algorithms of calculating ranking and relevancy widely differ among search engines, the way they index sites are more or less uniform and it is very important that you know what spiders are interested in and what they neglect.

Search engine spiders are robots and they do not read your pages the way a human does. Instead, they tend to see only particular stuff and are blind to many extras (Flash, JavaScript) that are intended for humans. Since spiders determine if humans will find your site, it is worth considering what spiders like and what don’t.

Flash, JavaScript, Image Text, or Frames?!

Flash, JavaScript, and image text are NOT visible to search engines. Frames are a real disaster in terms of SEO ranking. All of them might be great in terms of design and usability but for search engines, they are absolutely wrong.

An incredible mistake one can make is to have a Flash intro page (frames or no frames, this will hardly make the situation worse) with the keywords buried in the animation. Check with the Search Engine Spider Simulator tool a page with Flash and images (and preferably no text or inbound or outbound hyperlinks) and you will see that to search engines this page appears almost blank.

Running your site through this simulator will show you more than the fact that Flash and JavaScript are not SEO favorites. In a way, spiders are like text browsers and they don’t see anything that is not a piece of text.

So having an image with text in it means nothing to a spider and it will ignore it. A workaround (recommended as an SEO best practice) is to include a meaningful description of the image in the alt attribute of the tag but be careful not to use too many keywords in it because you risk penalties for keyword stuffing.

The ALT attribute is especially essential when you use links rather than text for links. You can use ALT text for describing what a Flash movie is about but again, be careful not to trespass the line between optimization and over-optimization.

Are Your Hyperlinks Spiderable?

The search engine spider simulator can be of great help when trying to figure out if the hyperlinks lead to the right place. For instance, link exchange websites often put fake links to your site with _javascript (using mouseover events and stuff to make the link look genuine) but actually, this is not a link that search engines will see and follow. Since the spider simulator would not display such links, you’ll know that something with the link is wrong.

It is highly recommended to use the 

If you are interested must visit:

The solution to this problem is to put all menu item links in the

If you happen to have tons of hyperlinks on your pages (although it is highly recommended to have less than 100 hyperlinks on a page), then you might have a hard time checking if they are OK.

For instance, if you have pages that display “403 Forbidden”, “404 Page Not Found” or similar errors that prevent the spider from accessing the page, then it is certain that this page will not be indexed.

It is necessary to mention that a spider simulator does not deal with 403 and 404 errors because it is checking where links lead to not if the target of the link is in place, so you need to use other tools for checking if the targets of hyperlinks are the intended ones.

Looking for Your Keywords

While there are specific tools, like the Keyword Playground or the Website Keyword Suggestions, which deal with keywords in more detail, search engine spider simulators also help to see with the eyes of a spider where keywords are located among the text of the page.

Why is this important? Because keywords in the first paragraphs of a page weigh more than keywords in the middle or at the end. And if keywords visually appear to us to be on the top, this may not be the way spiders see them. Consider a standard Web page with tables.

Read More: SEO Tools for Your Search Engine Optimization

In this case, chronologically the code that describes the page layout (like navigation links or separate cells with text that are the same sitewide) might come first and what is worse, can be so long that the actual page-specific content will screen away from the top of the page.

When we look at the page in a browser, to us everything is fine – the page-specific content is on top but since in the HTML code this is just the opposite, the page will not be noticed as keyword-rich.

Are Dynamic Pages Too Dynamic to be Seen At All

Dynamic pages (especially ones with question marks in the URL) are also an extra that spiders do not love, although many search engines do index dynamic pages as well.

Running the spider simulator will give you an idea of how well your dynamic pages are accepted by search engines. Useful suggestions on how to deal with search engines and dynamic URLs can be found in the Dynamic URLs vs. Static URLs article.

Meta Keywords and Meta Description

Meta keywords and meta description, as the name implies, are to be found in the tag of an HTML page. Once meta keywords and meta descriptions were the single most important criterion for determining the relevance of a page but now search engines employ alternative mechanisms for determining relevancy.

So you can safely skip listing keywords and descriptions in Meta tags (unless you want to add their instructions for the spider what to index and what not but apart from that meta tags are not very useful anymore).