Oh Dear, I am used to watching out for black widows and brown recluse spiders because of where I live- but not I have to watch for search spiders??? What do they look like AND do they bite??? Amy

Amy,

This should make things as clear as mud for you.

About Search Indexing Robots and Spiders

Many search engines use programs called robots to locate web pages for indexing. These programs are not limited to a pre-defined list of web pages, instead they follow links on pages they find, which makes them a form of intelligent agent. The process of following links is called spidering, wandering, or gathering. Once they have a page or document, the parsing and indexing of the page begins.

Jerry

Can they be tamed?
then what do I feed them?
Googles and Yahoos?

Whoopee ! Now I understand --NOT--!
Thanks, Jerry. I think I am of the age to want my forms of intelligent agents to have flesh and bone- you as an example- we can wander together and gather flowers on a sunny day, celebrate life and that would be just fine with me. I'm just too old for some of this. Fondly, Amy

I just read Jerry's "About search indexing" twice more in an serious effort to be part of the "in crowd". But I find that Don's questions are more to the point for me. Amy

The web now has many trillions of pages on it. Some of it utter garbage, some of it amazingly useful. The addition of zillions of blogs to the zilliions of web sties has compounded this, and is growing at rates no one ever thought possible. Besides commercial sites and more, people blog about the nonsensical to the profound. Even oral cancer opinion blogs and patient's chronicalling their journey are part of it all. So how do you make sense of these trillions of pages of information? How do you find something that comes close to what you are looking for? Search engines like Google (who by the way has been a fantastic partner of OCF's by giving us the search engine that runs our site search function) make it possible. But cataloging all this stuff is a task that boggles the mind. So the first thing to do is send out ittle self directing programs (spiders), robots (bots), to see what's out there and add it to some kind of structured list/catalog which of course is out of date every second of every day, with the rate of new additions to the www. So these spiders wander from link to link, page to page, cataloging every word and image they come across. What they bring back is sorted by a proprietary algorithm (no one knows how Google decides if something is valuable or not, but they have a means that makes sense to them) and when you type in oral cancer in a search engine -cancer stuff comes up.

Your ranking as a site, blog, or whatever in the Goggle system cannot be bought, not for love nor money. People have speculated on what this criteria for ranking is, since they would like to manipulate it for higher ranking in the list, but only Goggle knows for sure. Anyway, OCF has been in the top three finds for oral cancer since our first year out here, and we have spent more than 80% of our 7.5 years out here in the number one spot. When we get beaten out it is by the NIH or someone that has hundreds of thousands of pages of quality information. Tonight's search in Google for "oral cancer' finds OCF in the number two position, behind only Medline, the world's largest database of medical articles. Not a bad showing considering that there are 5,690,000 finds right now for oral cancer.... and yes, that is the correct number of zeros.

Anyway, all this is made possible by spiders that crawl aimlessly around the web cataloging what they find. They are crawling on our messages now besides our web site. What that means is that information in our threads will be found by people in need and hopefully help them or bring them to us.

So spiders are a good thing? Like our "dock spiders"??

Brian,

Thanks for the explanation and now not only I, but I'm sure, Amy, will understand spiders.

Now would you give a lesson on those "spyders" you can buy on Wall Street?

Jerry

I'm assuming that you are referring to exchange traded funds. They ended up with this weird acronym because the first one that that was allowed tracked the S&P (500) Depository Receipts Index (SPDR pronounced spider) and that became the vernacular for these kinds of funds. Although State Street, an investment advisory firm has started using the word spyder as a branding mechanism for their particular family of these kind of funds. Least any of you think that my understanding of all this is surprising or whatever, I tend to have a very good level of knowledge about things that have hurt me.... cancer being one, and exchange traded funds another. For those that care, we might add to the list, my wealth of knowledge (or at least opinions) about wars and why we get into them, what keeps helicopters airplanes flying (surviving several crashes) and ex-wives.... Most of these topics I have developed a pretty good understanding of, except perhaps that last one.

Thanks Brian- one of the spiders just crawled across my screen- cute little thing- I'm naming him BH