Amal Nagm

What is "The Invisible Web"?

Many untrained users have the naive expectation that they can locate anything on the Net by using Google or Yahoo or Ask.com. No, as powerful as these search engines are, they do not index everything on the Internet. In fact, search engines index less than 10% of the entire Internet! That remaining 90% is called the "Invisible Web", or in other words, "The Cloaked Web" or "The Deep Web". This is the massive content that is beyond the capabilities of spider indexing.

To better visualize this concept, let's start with some size estimates from Google.com, Yahoo.com, Cyberatlas, and MIT. These stats are current to July 2006:

56 billion static web pages are publicly-available on the World Wide Web.

Another estimated 6 billion static pages are available within private intranet sites

200+ billion database-driven pages are available as dynamic database reports ("invisible web" pages)

Google.com indexes 9.75 billion web pages.

Google, considered by experts to be the best search database today, is capable of indexing only a fraction of this monstrous content! Even by using electronic spiders to read and catalog millions of web pages each week, Google current indexes only 9.75 billion web pages of the 200+ billion pages out there, or less than 4% of all available Internet content.

So if Google only catalogs 4% of the World Wide Web, and other search engines catalog even less, then where is the remaining 90%of web content hidden?

That's where the "Invisible Web" begins

The "Invisible Web" (aka "Deep Web" or "Cloaked Web") is content that is closed-off to search engines. To be specific: the Invisible Web is about 200+ billion web pages that are stored not as static web pages, but as on-demand database content. As of December 2005, robot spiders are not advanced enough to read these private databases. Only a human reader can see these "invisible pages" by directly visiting these sites and making direct database requests.

Technical terminlogy:

"Spider": an artifical intelligence program, or robot, that is sent out weekly to scour the public Internet and read millions of static web pages.

The spider reports back to its mother database with its results, and those results get collated into search engine catalogs for public use.

"Database-Driven Web Content": web pages that exist only temporarily, and are generated only when readers request answers from a large database. These temporary web pages are dynamic, usually cannot be bookmarked, and commonly have extremely long URL addresses. Examples of databased web content: Today's job postings in Honolulu, apartments available in Singapore, today's weather report for Dublin, flights available to Istanbul, stock quotes for the NYSE, houses for sale in Winnipeg, reviews on the movie "King Kong", leather jackets for sale on eBay, hard drives for sale at Best Buy, your current savings account balance. These temporary web pages, once displayed to the reader, cease to exist moments later. Minute-by-minute, these databased web pages are re-created to reflect updated information on the database.

"OK. So I think I understand this now. 'Invisible Web Pages' are really 'Dynamic Web Pages'. That's when a database builds me a temporary page to answer my database question! Neat! So how do I find these thousands of "Invisible Web" databases?

Finding and reading "Invisible Web" pages is not difficult nor prohibited, but it is time-consuming. Because Invisible Web ("Deep Web") content is very specialized and unique to a specific topic of changing content, you will need to search twice:

First: you need to use primary search engines (Google, Yahoo, MSN, A9, Vivisimo, Dogpile, Ask) to locate the database you want. Expect this to be at least 30 minutes of searching (e.g. search for "jobs in honolulu", "weather reports dublin", "houses for sale winnipeg", "hard drive sales at best buy")

Secondly: once you find the specific database you want, you then need to search within that database.

For example: you may find www.monster.com helpful for searching databased jobs in Honolulu. Perhaps you will read www.theweathernetwork.com to find your weather reports, or www.expedia.com to search for available flights. Note: expect this "invisible web" database searching to take hours; these databases overflow with interesting and useful content!