Exploring the Hidden Web

Google, one of the most popular search engines, at best can index and search about 4 billion to 5 billion Web pages, representing only 1 percent of the World Wide Web.


But officials from Connotate Technologies, said they have developed technology that can mine and extract data from the Deep Web, which contains an estimated 500 billion Web pages, and deliver it in any format and through any delivery mechanism. The Deep Web refers to content in databases that rarely shows up in Web searches.


Through the use of intelligence-based software modules called information agents, corporate and government organizations can quickly and easily target specific unstructured data from intranets and password-protected Web sites on a continual basis.


Molloy said information agents can go to complex Web sites and databases, extract information — such as dates, names or contract identification numbers — and automatically deliver that data in any format. Pricing starts at a little more than $100,000. [Going where no search engine has gone before]