Googlebots Secret Way Of Crawling Pages
AKA: “View The Internet In Matrix Mode”
The reason why I titled this article this way was because earlier I tried to publish this article and couldn’t do it. I tried everything I could to publish the article and after about an hour of investigating I discovered a glitch in either Wordpress or my server? Either way, this post is about a Text Based Only Web browser. Unfortunately, the glitch is that I can’t mention the name of the browser. If the name of this browser is mentioned twice in the post, for some reason I can’t publish the article. Again, there is some kind of glitch that prevents an article from posting if this word is mentioned twice. And unfortunately, the word is used as part of the download URL, which forces me to not to use the name in the post! Anyway…
Earlier today, I was reading quite a bit at Google Webmaster Tools . And from what I was reading, I learned quite a bit about the Googlebots and how they see your pages compared to how humans do. In part of my reading I discovered a Text Based only web browser that Google suggests people download. They say that this Text Based web browser provides webmasters a very close representation as to how Googlebots interpret and see your page.
As I mentioned above, I titled this article the way I did for a few reasons actually. One is the problem I had when attempting to publish the post using this *special word more than once. Two, was after leaving Google Webmaster Tools and pursuing a search for this Text Based browser and finding it… I had a heck of a time trying to install it. Finally, after giving up on the program, I decided to give Google one last shot and attempted to search for a different source, version, etc… of this browser.
By refining my search and I finally found the Text Based browser and it had been updated to run on Windows XP! I was totally happy…
Installation went flawless, and I was able to finally surf the Web using this Text Based Web browser.
Now, if you are one bit interested in seeing what your sites look like in a Googlebot’s eye, I would strongly recommend that you download this Text Based browser and view for yourself.
What amazes me, is things that look good for humans, such as a navigation menu or menu links bar, look terrible in a Googlebot’s eyes. The things that you value and put on the top fold of your page appear on the bottom or sometimes the middle of the page when a Googlebot views it. I learned quite a bit about the global picture of what my site really looks like today. And I have to say, I was fairly impressed! For the most part my site looks exactly how I would want the Googlebots to see it.
I spent quite a bit of time surfing the web using this Text Based browser and there are quite a few sites that I visit on a regular basis that need tons of work! I am not mentioning any names, and to rule out thinking that your site might be one of the ones I am talking about, don’t worry the odds are very slim, because I visit what seems like thousands of sites a day!
However, if I were you… I would download, install, and use this Text Based browser… if you have seen the old movie, The Matrix… you can finally see the World Wide Web in what I would like to call and coin: “View The Internet In Matrix Mode”.
Download Now!
Also remember the famous Matrix quote: “Why or why didn’t I take the BLUE PILL!”
Good luck!
| 3.1 |
13 Comments! Join The Discussion by Leaving Your Comment.
What do you have to say about this post? Leave a comment!
_20081119103701.gif)
Date/Time: 1-22-2007 01:05:47 Comment #364
I downloaded the red pill. I knew I had 50/50 chance, but you threw me a little on adding the quote. But then I saw the blue pill was firefox and I knew that firefox wasn’t webbased.
Date/Time: 1-22-2007 01:13:14 Comment #365
Yes, you got it right… Take the Red Pill to discover the Matrix! lol…
Date/Time: 1-22-2007 01:22:42 Comment #366
Hi. I downloaded L Y N X and it seems like the google bot just reads and views the page files exactly how they are written line for line in the source code. It doesn’t understand CSS styling and things like that.
Thats kinda weird how you cant say that word. I had to recomment to get it to submit.
Date/Time: 1-22-2007 01:32:41 Comment #367
White,
Thanks for the comment, and yes I believe that you are correct. It seems that the Googlebots read straight down the file, disregarding any rich format that we see live using a normal Web browser.
What I am learning is that you have to really write some Good HTML in order to make your links appear in the exact order you want the Googlebots to see them in. Out of the box, Wordpress does pretty good with this. But what I see in other sites is the various plugins and added stuff in the sidebars messing with things. Also it seems that the feed urls are usually the very first link the Googlebots see… It’s pretty interesting stuff, and I think I personally have a new tool for improving my site’s optimization for Google.
Date/Time: 1-22-2007 01:51:53 Comment #368
Okay, I got Links installed and I figured out how to use it, but I don’t see what you are talking about. How does this help me improve my SEO? I don’t get it?
Date/Time: 1-22-2007 01:59:43 Comment #369
Chris,
The program itself doesn’t help improve your site’s SEO. The program is a tool for you to use to improve your site’s SEO. Such as a wrench is used as a tool to loosen a bolt. Viewing your site using this Text Based Web browser allows you to see your site in a way that is very similar to how a Googlebot would see it.
Why is this important? You might ask…
The reason why this is important is because you can see which links get crawled first on your pages. If you are trying to flood indexing on a certain section of your site, you would want to be assured that this section would be one of the first links that the bot finds on first on your page. Now, you can place a link on your page where you think is best, but that may differ from where a Googlebots actually sees the link… for all you know, the link could be one of the last pages the Googlebot crawls to and indexes. I hope you are with me now.
Best Regards,
Garry Conn
Date/Time: 1-22-2007 02:02:48 Comment #370
So, why do I have to change my links around? what is the right way to put them and how is the right way to let the google robot see it?
Date/Time: 1-22-2007 02:14:51 Comment #371
Heather,
You don’t have to change the order of your links on your page at all unless you are unhappy with the order in which the Googlebot sees them in. There isn’t a right or wrong way to order your links… It all depends on which sections, links, pages, etc… on your site you consider to be most valuable. In other words, which pages, links, or sections of your site do you want to work on getting the most crawler traffic to? Again, the more crawler traffic to certain sections of your site mean a more thorough indexing which means a deeper crawl into your deep linked pages. There are no rules, the Text Based Web browser is just a tool that you can use to examine how pages look to Googlebots. You have to use your own judgement with which links the Googlebot should see first. You might be like me… happy with how things already are.
Thanks for the comment.
Garry
Date/Time: 1-22-2007 02:25:45 Comment #372
Yeah, I get it now. Thanks for telling about this. I still need to learn how to use the browser more but i see how to use it as a seo tool.
Date/Time: 1-22-2007 02:29:22 Comment #373
this program is really cool because you cao do just about everythign you can normally do on a regular browser like firefox but there are no pics and it does like sorta like the matrix.
This is really cool thanks.