The Deep or the Dark Web?
When I speak about pages not indexed by Google, maybe your first idea is about the Dark Web. While the Dark Web, also called Darknet, fits the description of sites/pages not indexed by Google, the Dark Web and the Deep Web are not the same thing. An example of a Deep Web page is a closed group on Facebook. Since the page is accessible after a login only, and Googlebot can’t log in to access it, the page is not indexed. However, when you log in to this group, you can see the page. Similarly, if the page requires payment to gain access, Googlebot can’t index it, but you can view it after you pay. No-follow or broken links, or dynamic pages generated on the go after a search query from a user, also stop search engines from indexing pages, but you as a human can access this information. Info in the form of an image/video or other formats search engines don’t understand but humans do is another example of Deep Web content. These pages are accessible with a simple browser, and generally they use the http (or https) protocols. On the other hand, the Dark Web uses a different routing protocol with built-in encryption. Two popular protocols are TOR and I2P. The Dark Web contains lots of illegal resources, too, and search engines by no means will index these, even if they could. Now, after I explained the difference between the Deep and the Dark Web, let’s see what you can do to find stuff in the Deep Web.
1. Try Other Search Engines
Sometimes a page is not accessible by Google (for one reason or another) but is indexed by other search engines. Technically speaking, in this case the page isn’t in the Deep Web (because it’s accessible via a search engine), but for anybody whose search starts and ends with Google, the page is not there. If you get in the habit of occasionally using other search engines, such as DuckDuckGo, a local search engine, or even Bing, in addition to Google, you might be surprised by the amount of good stuff you can find with them.
2. Find the Main Page with Google and Go on Your Own
In other cases Google has the main page of a site only. This happens with sites that require login or payment or that have no-follow links Google didn’t index. If this is the case, it’s easy – find the main page with Google, and then explore the site on your own. If the site has a search functionality, your task is even simpler.
3. Try Google Books or Go to a Library
For scholarly information, try Google Books. If you know a document exists, but you can’t find it with Google Books, you can go to a library, especially an academic one. Many libraries at colleges and universities subscribe to paid databases, and chances are you will be able to use these, maybe even for free.
4. Try the Deep Web Search Engines
As surprising as it sounds, the Deep Web has search engines of its own. In many aspects, such as user interface or functionality, these search engines are light years behind Google, but don’t judge a book by its cover. There are really a lot of these engines, and it might take quite a lot of time to find what you want. Even worse, sometimes you might spend days or weeks to no avail. Check this huge list of Deep Web search engines and locate the ones in your area of interest. If you are looking for older versions of current pages and stuff that used to be online but isn’t anymore, try the The Internet Archive project.
It might be a bit harder to find stuff in the Deep Web, but if you are looking for highly specialized stuff, you might have more luck there than with general search engines.
