hidden universes of information on the internet

Hidden Universes of Information on the Internet

Rev. 04/2013

Russ Haynal Internet

Instructor, Speaker, and Paradigm Shaker

21015 Forest Highlands CtAshburn, VA 20147

http://navigators.com

Phone : [email protected]

Note: If you send me an email, put “internet training” in the e-mail's subject

Russ Haynal/navigators.com/

Internet Instructor & Speaker

http:/Course Outline

• Introduction to Internet Architecture

• Preparing for a search

• “Persona” Issues

• Search Tools - In Depth

• Advanced Features

• Specialized Resources

• Source Evaluation

• Demo Sites and Summary

specific_page.html

Online Web page = http://navigators.com/opensource.html



http:/Disclaimer

• This session illustrates a wide variety of search tools, techniques and research methods

• Consult your organization’s policies to verify if these methods are approved for your type of Internet connection



http:/Internet Definition

Internet represents a once per thousand year event

Last such event = Gutenberg printing press

Are You Literate in Today’s Online World?

“A large collection of Inter-connected networks and computers”

“A new fundamental form of communication that will absorb other communication channels”



http:/Internet’s Growth stats.html



http:/Number of hosts in each Domain

net 296,984,286

com 155,221,495

edu 12,471,151

mil 2,933,061

gov 2,470,274

org 2,131,705

jp 63,199,811

it 24,923,735

br 22,212,190

de 20,148,634

cn 17,446,117

au 16,952,338

fr 16,254,627

mx 14,467,988

nl 13,507,022

ru 12,550,558

pl 12,391,453

ar 8,628,736

ca 8,608,728

uk 8,363,257

in 6,777,881

tw 6,144,613

ch 5,232,713

be 5,041,680

se 4,919,057

fi 4,603,467

es 4,266,121

dk 4,242,576

za 4,162,786

tr 4,034,284

cz 3,988,136

pt 3,558,417

no 3,523,575

at 3,389,616

co, gr, th,

hu, nz, ro

us 2,151,694

Source: www.isc.org as of July 2011

Top Level Domains

stats.html



http:/Example Backbone Maps

Level 3

AT&T

isp.html

Sprint

C&W

Verizon



http:/Backbones Connecting

regionalISP #1

For a complete picture, initiate traceroutes from within several different backbones

Backbone ISP- A

Backbone ISP

Regional ISP

Exchange Point

Client(PC)

Server

Private Peering

Backbone ISP- B

regionalISP #2

Web hosting center

Enterprise LAN/Wan

Large organization

Destination

traceroute.html



http:/Exchange Point Traffic

• Notice the daily fluctuations - Analysts may want to “schedule” their research

• Traffic continues to grow rapidly in many locations

isp.html

Source: http://www.hkix.net



http:/How Does it Work?• Internet started as “Packet Switching Networks” using TCP/IP

(Transmission Control Protocol - Internet Protocol)

• Every Internet connection has a unique IP Address consisting of 4 numbers, each number has a range of 0-255 (ie. 198.211.16.134)

• Internet IP numbers are allocated through a hierarchy

– IANA --> ARIN/RIPE/APNIC/LACNIC/AFRINIC --> ISP’s /Company/Country

• Routers direct your packets of information along the “preferred” path

Router Router

Router

Router

Router

Router

RouterRouter

Note: The next generation of IP Address space (IPV6) is quite LARGE3,911,873,538,269,506,102 IP #’s per square meter of the Earth's surface 4,500,000,000,000,000 IP#’s for every observable star in the known universe

traceroute.html



http:/Domain Name System• The Domain Name System (DNS) associates

alpha-numeric names with IP addresses

• Names are registered with commercial registrars such as Go Daddy or country-specific registrars

• DNS Servers are distributed throughout the Internet - They act as a set of inter-linked phone books

• You enter “www.navigators.com”, and the DNS servers match it to “198.171.173.51”

• Historical meaning for domain names– .com=commercial .net= Internet Provider .org = non-profit

– .uk = United kingdom .pk= Pakistan .ru = Russia

• Reality…. Many country domain names are now for sale to ANYONE from ANYWHERE

domain_name.html



http:/Web Server / Web Site

Web site is the content

The web server is a computer loaded with server software and a reliable Internet connection.

Web pages= name.html

Graphics=name.gif=name.jpg



http:/A more complex environment

DataBase

•Internet users interact with web server•Web server query is passed along to data base.•The content of the database is only displayed

TEMPORARILY in a web page that is created in response to USER-actions.

•Most database content is unreachable by search engines

Web Browser

Online Hosting

typed form

Web server

page data

Application server



http:/Accessing a Web PageA series of communications is involved to view a web page

company.com

Sales

gadget.html

1. Browser requests URL:http://www.company.com/sales/gadget.html

2. Connect via Internetto web server company.com

3. Server sends gadget.htmlfrom its sales directory

4. Browser displays gadget.html, requests graphics,

and eventually terminates connection to the server

(background communications:cookies, browser info, etc)

“Document not found”? - Try shortening the URL!

logo.gif






• “Persona” issues






specific_page.html



http:/Introduction to “Persona”

• While viewing a web page (URL1), You click on a hyperlink to visit another web page (URL2)

• Your web browser sends “environment variables” to the web server

• Webmaster’s use this information to determine information about you and your organization (physical location, your interests, software, etc.)

ReportsAccesslogs

Analyst Webmaster

Web ServerURL1

URL2

InternetAccess

As you surf the Internet, you give-off a certain persona

You should always know what websites know about you

persona.html



http:/Persona Details•Your persona is communicated to every web server that you visit

•You should be explicitly aware of your persona before you visit any website. For example, should you visit:– badguy.com from agency.gov?

Your persona is communicated via “environment variables” such as:

•REMOTE_HOST = This is the name associated with your IP Number.

•REMOTE_ADDR= This is the IP number of your computer, or proxy. A webmaster could do a traceroute to see how you are connected.

•HTTP_REFERER = This is the URL of the page you were previously viewing. Be careful on how you create web pages. For example, do you want to reveal the following?:

– http://badguy.com is listed on http://intranet.agency.gov/joe_smith/investigation_targets.html?

• Your persona may also be transmitted via Java Applets such as ga.js and urchin.js (google analytics)

persona.html



http:/A Typical Scenario...

searchtool.com webmaster knows your “search terms”

destination.com webmaster knows what “search terms” you used to find them

Persona:- agency.gov OR- town.ninja.com

Analyst

searchtool.com

destination.com

“search terms”

http://searchtool.com/query=searchterms

hits

page

webmaster

webmaster

persona.html



http:/Always check your Persona

• You can also search for: proxify who am I

This is a key paragraph to look for… If this is missing, then no referring URL is being passed

Important Note: This testing page is most accurate when you click on a link to arrive at this page.

persona.html



http:/Think before you click...• Does your connection method leak a Referring URL?

• IF IT DOES... do NOT “Click” on your search results

• A click on this search result will tell the webmaster at orgnet.com that you are searching for “terrorist”

Referring URL

Hover over the link to see its URL

persona.html

http://www.google.com/query=terrorist_&start=110



http:/

ninja.com

Agency_portal.com/page_namestarget.comagency.gov

Analyst #1

Analyst #2

Persona=agency.gov + referrer = portal

Persona=ninja.com + referrer = portal

The “portal” Problem...

Exposing a “less recognizable” persona

Analyst #1: uses agency.gov persona to visit “targets”

Analyst #2: uses “ninja.com” persona to visit “targets”

Now “ninja” persona may be recognized as “agency.gov” visitor

The “parallel visit” Problem...

ninja.com

target.comagency.govAnalyst #1

Analyst #2

Even with no http_referer, a webmaster can still make the association due to high volume hits or similar usage patterns.



http:/Plan out your Internet Research

• Spell it Out - Define the topic, spell it out, key words, acronyms, “what” and “who”

• Strategize - Choose your approach, which online resources, search tools

• Search - Get online, execute, stay focused, use advanced search features

• Sift - Filter the results, follow the leads

• Save – Make bookmarks, take notes, organize results, share with co-workers.

search_methodology.html



http:/Spell out the topic...1. Name of topic, and what do you want to learn about the topic:

__________________________________________________________________

__________________________________________________________________

2. Spell out the topic (words, acronyms, abbreviations)

_______________________________

_______________________________

_______________________________

_______________________________

_______________________________

_______________________________

_______________________________

_______________________________

3. Make a list of “who” might publish such information (industry association, government agency, NGO’s, user group etc.)

__________________________________________________________________

__________________________________________________________________

generic, simple terms obscure, specific terms

search_methodology.html



http:/Follow All Good Leads in Parallel

linkAlinkBlinkC

Results

linkAlinkBlinkC

Results

link1link2link3

Page AlinkXlinkYlinkZ

Page 1linkMlinkNlinkO

Page Y

link1link2link3

Page A

link1link2link3

Page B

link1link2link3

Page C

Valuable links B&C never get explored...

Many users only follow one good lead at a time

linkXlinkYlinkZ

Page 1linkMlinkNlinkO

Page Y

• Right-click to open each link in its own browser window (or tab)• Switch between open windows using “ALT-TAB” or “ALT-TAB-TAB-TAB”• Note: http_referrer is still transmitted• Do NOT launch multiple browsers from desktop or start-menu.

multiple_browsers.html












specific_page.html



http:/Overview of Internet Search Tools

• Search Engines (Google, search.yahoo.com)– Large database – text from billions of clickable pages

• Directories (dir.yahoo.com, www.dmoz.org)– Manually built subject trees–links to millions of web sites

•“User Pages” (Joe’s guide to widgets)– Built by subject experts- hundreds of topic-related links

Each Tool has strengths and weaknesses

Pick the right tool...

search_tool_intro.html



http:/Internet Directory

(i.e. dir.yahoo.com, www.dmoz.org)

Directory

Filer may not be a subject-expert

URL’s & Descriptions (submitted by Users)

• Good for early stages of search, general subjects

• Links are grouped by topic

• Pages are manually built




http:/Typical Format of a Directory

Subject category:use to navigate

Discover “user pages” on this subject

Number of listings in this sub-category

Alias to another branch




http:/Searching a directory...

Main Menu“top”

Topics

subtopics

Content of subject tree

website

Links to external web pages

• Searches the text within the directory’s own web pages.

• Use search terms that would appear in:

– category titles

– web site titles

– web site’s brief description

• You are NOT searching the websites – just their brief description

wireless




http:/Search Engines

(ie. Google.com, bing.com)

• Search Engine’s “robot” explores Internet, and copies web pages into its database

• Supports very detailed keyword searches

• Take the time to learn about the features & options of the search engine

Web ServersSearch Engine Site

RobotIndexerSearch

Interface

Your PC

Indexed Database

cachedWeb pages

copied Web page


You must envision what the target page will look like. “Use your imagination”

Try adding the words “resume” or “curriculum vitae” to your search terms



http:/Search Engine Comparison

• http://ranking.thumbshots.com – Compares the first sixty hits from two search engines you select

• Notice on this search for “jihad’, only 12 out of 60 hits appeared in both Google and Yahoo… Most hits are unique to each search engine.

search_tools.html

• News, forums and analysis of search engines.



http:/Pay Attention to search results

Clustering – Google showing a maximum of 2 hits per domain

Cached = Google’s local text copy of the page. Graphics will still be downloaded from the remote website, unless you get to a “text only” version of cached

See hits from only that domain

Indentation = clustering

Problem: Getting a “text only” version of Google’s Cached has become a complicated process. For a detailed set of options see: navigators.com/cached.html

cached.html



http:/“User Pages”

Potential weblink

• Usually focused on a specific subject

• Developed by “experts” in that field(or just a person with passion for subject)

Info Expert

• Often contain “the best” online resources




http:/Finding “User pages”• Announced to Dmoz and other directories

• Linked at wikipedia, wikimapia

• Groups of users at forums, blogs and mailing lists

• Watch for sites labeled:“Joe’s ultimate guide to widgets”

• “User pages” often point to other “user pages”

• “Surfing Upstream” from several related sites (covered in Hidden Universes part 2)

• Ask other researchers – there are several sites that everyone knows as “the best”

• Interactive, live communication (Chat, telephony, virtual worlds)

search_universes.html



http:/Wiki ____• A Wiki allows immediate creation and editing of pages by “anyone”

• Wikipedia.org – Encyclopedia that can be instantly edited by ANY Internet user.

• Good starting point for many subjects to gain an overview of the topic

• Page can be biased from the most recent editor

• Some entries get “locked-down” due to vandalism

• old.wikimapia.org – same concept applied to google maps

• “map type” google map: zoom to the right location

• “map type” “wikimapia classic” : to see comments

• To learn about the authors: Click on the comment box menu history the user’s name stats then clicking on the stat numbers listed shows every place that user has added

old.wikimapia.org

Anybody can add labels & comments onto map

“map type” “wikimapia classic”



http:/Mailing List search techniquessearch_universes.html

• Many mailing lists are mentioned on web pages

• Use google or bing to search:

• Your_Searchterms “mailing list”– will find some mailing lists that talk about your search terms

• Your_searchterms “mailing list” reply thread index – will find postings from the mailing list that have been archived onto the web

Websites that host mailing lists tend to have

great content



http:/Blogs and Forums• A Web Log (blog) is usually owned by one person.

• Owner can post a log of their daily activities, or post ongoing comments about a topic.

• Others may also be allowed to add comments onto the blog

• Wordpress and blogger are popular sites

• Forum – An online discussion site focused on a particular topic

• Many users can participate by posting messages.

• Moderators may “police” comments that are considered off-topic

• Try searching for:

• Searchterms forum post - to find a forum that discusses your topic

• Searchterms forum post replies views – to find individual threads and messages that discuss your topic

• Membership requirements are a barrier to search engine robots

• Vbulletin is a popular program used on many websites












specific_page.html



http:/

Mailing Lists

“The web” is TINY

• Many detailed searches are a two-step process

– first find the specialized database

– then type a very specific query into the database.

World Wide Web (pages.html)

ChatVOIP

BlogsForums

Multi-media

Search engines

Total Online material

SpecializedDatabases

Email

Closed systems

1. Initial Search 2. Detailed Search

search_universes.html

( 1000X larger than the web )



http:/Lists of Search Engines

• For specific information, use a specialized search tool – Get “deeper” results than a general search engine

• Thousands of search engines are listed

• Search engines are grouped according to the subject they cover

search_tool_specialized.html

70,000 databases

55,000 public record

databases

.com

.net



http:/Specialized Search Engines

• A phone book for the entire U.S. Includes reverse look-ups

• Federal Register and much more

• List of manufactures and companies from around the world

Specialized search tools contain content that search engines can’t reach

search_tool_specialized.html

• Real-time tracking of ships from around the world



http:/Country-Specific Resources

• The online training page links to many additional resources such as news, language translators, and country-specific resources. (Hidden Universes part 2)

universes_part2.html

Phone books

20,000 news sources












specific_page.html



http:/Source evaluation

• Pick apart the URL:

• Determine where “ownership” of the web page begins– www.facebook.com/joesmith/info.html

– www.joesmith.com/stuff/info.html

• Browse the directories (shorten URL if necessary)

• Look at domain’s home page - Is it a web hosting site? Is “pathname” a user account?

• Search for “links to” the page, shows who links to that specific web page. (command = link: )

• IF the domain home page looks like the “owner” of the content, then you can move forward with whois and traceroute

sesseval.html



http:/Source Evaluation - Using WHOIS• Domain names are “registered” at Internet registrars

(global, country-specific)

• Each registrar develops its own policies– may authenticate requester of domain name (.gov, .mil)

– may accept anyone (with money) (.com, .org, .net, .tv, .pk, etc)

• Registrants provide “point of contact” information, for at least invoicing purposes

• Domain “point of contact” information is often available from the registrars’ database via a “WHOIS” query

• WHOIS contents may be inaccurate, although usually the email, or postal address will be correct to receive renewal invoice

whois.html



http:/Performing a “Whois Query”• “whois” reveals the “owner” of a domain (searchenginewatch.com)

whois.html

Administrative contact: Ron Doobay HAYMARKET HOUSE 28-29 HAYMARKET LONDON SW1Y 4RX UK +44.2074849700 +44.2079302238 [email protected]

Technical contact: Domain Administrator 3rd Floor Prospero House 241 Borough High Street Borough London SE1 1GA UK +44.2070159370 +44.2070159375 [email protected]

Created on: 1998-03-20 Expires on: 2015-03-19

Domain name servers: NS3.INCBASE.NET 85.133.68.200 NS2.INCBASE.NET 62.140.213.136 NS1.INCBASE.NET 62.140.213.135

• Spam concerns has lead to many domain names being registered via “privacy enhanced” options



http:/Whois Complications

• Intense competition between registrars makes WHOIS more difficult

• Some registrars block whois queries originating from competing registrars, or from popular third-party whois sites

• Queries may also get blocked if too many queries originate from the same IP# (i.e. one proxy for many researchers )

Third-Party Whois SitesRegistrars

= Integrated database of: .com, net, org, biz, coop. museum, edu, info, aero, arpa, int

= permitted WHOIS Query= query may be blocked

whois.html



http:/Traceroute• Shows a network path between 2 machines

• Traceroute designed to help de-bug network connections

• Can initiate traceroute from your workstation, or from public “traceroute servers” located throughout the Internet

• Each Internet provider has their own naming convention for their infrastructure

– Location labels: City names or 3-letter airport codes

– Exchange points (LINX, HKIX, AMS-IX)

– Infrastructure Topology (T3, FDDI, GE)

• Recognize that a website can be hosted anywhere

– Could be at organizations’ own site, but may be hosted at a well-connected web hosting facility

traceroute.html



http:/

traceroute Output from WWW.Telcom.Arizona.EDU to www.nsa.gov: traceroute to www.nsa.gov (65.213.217.241), 30 hops max, 40 byte packets 1 128.196.128.253 (128.196.128.253) 1 ms 1 ms 1 ms 2 192.80.43.25 (192.80.43.25) 1 ms 1 ms 1 ms 3 192.80.43.58 (192.80.43.58) 1 ms 1 ms 1 ms 4 207.250.65.133 (207.250.65.133) 5 ms 5 ms 5 ms 5 core-01-ge-2-2-0.phnx.twtelecom.net (209.234.146.45) 5 ms 5 ms 5 ms 6 core-02-so-0-0-0-0.lsag.twtelecom.net (168.215.53.73) 17 ms 17 ms 17 ms 7 tran-01-ge-0-3-0-0.lsag.twtelecom.net (168.215.54.98) 17 ms 17 ms 17 ms 8 500.POS1-1.GW3.LAX1.ALTER.NET (208.222.8.245) 17 ms 17 ms 17 ms 9 122.at-0-0-0.CL2.LAX4.ALTER.NET (152.63.52.246) 18 ms 17 ms 18 ms 10 0.so-0-0-0.TL2.LAX9.ALTER.NET (152.63.115.146) 20 ms 20 ms 0.so-7-0-0.TL2.LAX2.ALTER.NET (152.63.2.82) 18 ms 11 0.so-6-0-0.TL2.DCA8.ALTER.NET (152.63.3.193) 74 ms 75 ms 74 ms 12 0.so-5-0-0.XL2.DCA8.ALTER.NET (152.63.35.250) 74 ms 74 ms 74 ms 13 188.ATM6-0.GW3.BWI1.ALTER.NET (152.63.39.41) 75 ms 75 ms 76 ms 14 * * * 15 * * *

Results of Traceroute

Traceroute and other online resources help reveal the dynamic

architecture of the Internet

Indicates that Time-Warner and UUNET (Alternet) peer at Los Angeles

BWI Airport code

traceroute.html



http:/A Foreign Newspaper ???URL = http://www.eldia.com.ar

• “.ar” implies that the site is from Argentina?

• Traceroute reveals that this website is physically hosted in the US.

traceroute from WWW.Telcom.Arizona.EDU to www.eldia.com.ar: 1 woody-netops.telcom.Arizona.EDU (128.196.128.1) 1 ms 1 ms 1 ms …..8 peer-01-ge-0-3-0.chcg.twtelecom.net (168.215.53.194) 46 ms 46 ms 46 ms ….10 p4-6-0-0.r01.chcgil01.us.bb.verio.net (129.250.2.254) 48 ms 47 ms 47 ms 11 p16-2-0-0.r02.stngva01.us.bb.verio.net (129.250.5.103) 83 ms 80 ms 80 ms 12 ge-1-1.r0728.stngva01.us.wh.verio.net (129.250.27.219) 81 ms 83 ms 80 ms 13 ge-26.a0702.stngva01.us.wh.verio.net (161.58.129.13) 81 ms 80 ms 80 ms 14 noticiasargentinas.com (161.58.165.155) 80 ms 80 ms 81 ms

• Chicago,illinois• Sterling, Virginia

• wh= web host



http:/10 minute web page example• Launch microsoft word

• Type, type, type (be creative)

• File save as web page

• Make a hyperlink:

– Highlight some text : “insert menu” -> Hyperlink”

– Type complete URL (i.e. http:/ /www.cnn.com )

• Test the page --> file preview in Browser

• Borrow a graphic -> “right-click” CNN logo -> “Save image” (C/temp)

• Inserting a graphic ->

– “insert menu” --> “insert image” from file

• Upload the finished page: --> announce to Yahoo, google, etc

Every webmaster only requires this 10 minutes worth of knowledge

developer.html












specific_page.html



http:/Each search tool is different

• Each search tool has it’s own unique set of defaults and options

• Take the time to learn the options of each tool

– Don’t assume anything

• These tools are competing, trying to be unique

• Read the help



http:/Search - Review

• Stay organized in your search

– (spell, strategize, search , sift, save)

• Always be conscious of the type of tool you are using (and read its help)

• The “right” search terms, placed correctly into the “right” search tool, should quickly yield “good” results

• Better Yet: Discover the best “user pages” and online communities for your topic - follow their leads(They have already weeded through the junk.)

• Stay organized in saving your discoveries



http:/Search Scenario

• Create bookmark folder

• Explore topic areas at directories or wikipedia

• Watch for “user pages”

• Are there databases or forums for the topic

• Surf Upstream to find additional “user pages”

• Save search engines for specific, obscure search terms - Use advanced queries



http:/An open Paradigm

Content Consumerof content

• Types of content

– Information, entertainment, business, leisure

• Content origins

– corporations, Hollywood, other people

• Content formats

– text, audio, video, Interactive reality

• Transport Mechanism

– Phone line (copper/fiber), coaxial cable, Wireless, Direct Satellite, Electric Lines

transport

Mergers and acquisitions are occurring horizontally and vertically



http:/Summary• Internet contains a large, fragmented information space

• Search engines are limited to only billions of “Clickable” pages

• The best content is organized by “people without lives”

• The Internet’s “critical mass” will transcend all other communication technologies

• Change is the only constant

The Future is Clear...Master the Information Superhighway

orBecome Roadkill

hidden universes of information on the internet

Documents