principle components analysis (pca)rcs46/lectures_2017/10-unsupervise/... · 2017-10-19 ·...
TRANSCRIPT
Principle Components Analysis (PCA)
Rebecca C. Steorts, Duke University
STA 325, Chapter 10 ISL
1 / 26
Agenda
I Recap of information retrievalI Relation to PCAI Idea behind PCAI Application to NYTimes data setI Multidimensional scalingI Scree plots
2 / 26
Information retrieval
I Recall that information retrieval systems often representdocuments as what are called bags of words.
I Such documents are represented as vectorsI Each component counts how many times each word in the
dictionary appears in the text.I This throws away information about word order.I Part of the representation of one document might look like:
a abandoned abc ability able about above abroad absorbed absorbing abstract43 0 0 0 0 10 0 0 0 0 1
and so on through to zebra'',zoology’‘, “zygote”, etc. to the endof the dictionary.
3 / 26
Information retrieval
These bag-of-word vectors have three outstanding properties:
1. Most words do not appear in most documents; thebag-of-words vectors are very sparse (most entries are zero).
2. Small number of words appear many times in almost alldocuments.
I (Examples: “the”, “is”, “of”, “for”, “at”, “a”, “and”, “here”,“was”, etc.)
3. Many words’ counts are correlated with some but not all otherwords.
4 / 26
Takeaway
I Don’t get much value from keeping around all the words.I Better off projecting down to a smaller number of new
variables.I Project partially since the words mean slightly different things.I This is exactly what principal components analysis does and
it’s very useful as an exploratory data analysis exercise fordimension reduction.
5 / 26
Principal components analysis (PCA)
I PCA is a tool for exploratory data analaysis and dimensionreduction.
I Take large set of correlated variables and replace with smallernumber that collectively explain most of the variability.
I The PC directions are directions in the feature space alongwhich the original data are high variable.
I PCA is an unsupervised learning tool – since it involves thefeatures X and no response Y
6 / 26
Why is it useful?
Suppose that we wish to visualize n observations withmeasurements on a set of p features, X1, . . . ,Xp as part of anexploratory data analysis.
We could do this by examining two-dimensional scatterplots of thedata, each of which contains the n observations’ measurements ontwo of the features.
However, there are p = p(p − 1)/2 such scatterplots
For example, with p = 10 there are 45 plots!
PCA seeks to to find a low-dimensional representation of the datathat captures as much of the information as possible.
7 / 26
PCA
PCA seeks a small number of dimensions that are as interesting aspossible, where the concept of interesting is measured by theamount that the observations vary along each dimension.
Each of the dimensions found by PCA is a linear combination of thep features.
8 / 26
How many principal components are there?
There are a total number of min(n − 1, p) principal components.
For some reason R prints out the n principal component (and I’mnot sure why it does this)!
9 / 26
PCA
The first principal components of a set of features X1, . . . ,Xp is thenormalized linear combination of the features
zi1 = φ11X1 + φ21X2 + · · · + φp1Xp =p∑
j=1φi1Xi
that has the largest variance.
By normalized, we mean that∑p
i=1 φ2i1 = 1.
I φ11, . . . , φp1 are the loadings of the first principal component.I Together the loadings make up the the principle components
loading vector φ1 = (φ11, . . . , φp1)T
10 / 26
Finding the first PC
Given a data set Xn×p, how do we compute the first principlecomponent?
Since we’re only interested in the variance, let us assume that eachof the variables in Xn×p has been centered to have mean zero.
We then look for the linear combination of the sample feature valuesof the form
zi1 = φ11xi1 + φ21xi2 + . . .+ φp1xip =p∑
j=1φi1xi1. (1)
that has the largest sample variance subject to the constraint∑pj=1 φ
2j1 = 1.
11 / 26
Finding the first PCFinding the first principle component loading vector solves theoptimatization problem:
maxφ11,...,φp1
1n
n∑i=1
(p∑
j=1φj1xij)2
subject top∑
j=1φ2j1 = 1 (2)
Using equation 1, we can write the objective in equation 2 as
1n
n∑i=1
z2i1
Note: Since 1n
∑ni=1 xij = 0, the average of z11, . . . , zn1 is also zero.
Thus, the objective that we are maximizing in equation 2 is just thesample variance of the n values of zi1
where z11, . . . , zn1 are the scores of the first principal component.12 / 26
PC
The optimization problem in equation 2 is beyond the scope of thiscourse, but it can be solved by what is called aneigen-decomposition (it requires linear algebra).
13 / 26
Geometric Interpreation
There is a very nice geometric interpretation for the first principalcomponent.
The loading vector φ1 with elements φ11, . . . , φp1 defines a directionin the feature space along which the data vary the most.
If we project the n data points onto this direction, then theprojected values are the principle component scores z11, . . . , zn1.
14 / 26
New York Times Example
I Have news stories taken from the New York Times AnnotatedCorpus
I Consists of about 1.8 million stories from the Times, from 1987to 2007.
I Stories have been hand-annotated by humans with standardizedmachine-readable information about their contents.
I From this corpus, have randomly selected 57 stories about artand 45 stories about music.1
1Turned them into a bag-of-words data frame, one row per story, one columnper word; plus an indicator in the first column of whether the story is one aboutart or one about music.
15 / 26
PCA for NYTimesload("pca-examples.Rdata")
# The workspace now contains:# nyt.frame.raw: a data frame#with counts of words (columns) in stories (rows)# first column, "class.labels",# is a factor indicating "art"# or "music"# nyt.frame: the same, with word
# counts suitably normalized and weighted# art: vector where each row is itself a
# vector of words giving the# actual stories about art, with
# punctuation removed, etc.# music: ditto# Some miscellaneous functions used to
# create the data sets (see end of# this file for gory details)
16 / 26
PCA for NYTimes# How big is it?dim(nyt.frame)
## [1] 102 4432
# Remember: rows = stories, columns = words (except the first column, which# is the type of story)
# What are some typical words?colnames(nyt.frame)[sample(ncol(nyt.frame),30)]
## [1] "penchant" "brought" "structure" "willing" "yielding"## [6] "bare" "school" "halls" "challenge" "step"## [11] "largest" "lovers" "intense" "borders" "mall"## [16] "classic" "conducted" "mirrors" "hole" "location"## [21] "desperate" "published" "head" "paints" "another"## [26] "starts" "familiar" "window" "thats" "broker"
# A little bit of the datasignif(nyt.frame[sample(nrow(nyt.frame),5),sample(ncol(nyt.frame),10)],3)
## jacket patch tapes want ford failed condemn intentional confined## 24 0 0 0 0.0000 0.0000 0.0000 0 0 0## 2 0 0 0 0.0275 0.0704 0.0000 0 0 0## 85 0 0 0 0.0482 0.0000 0.0000 0 0 0## 59 0 0 0 0.0000 0.0000 0.0000 0 0 0## 76 0 0 0 0.0000 0.0000 0.0215 0 0 0## destroyed## 24 0## 2 0## 85 0## 59 0## 76 0
# What you should see: lots of zeroes! Most words do not appear in most stories.# Harder to see: strong correlations between words which do appear.
17 / 26
PCA for NYTimes
I Need to omit the first column in the first command because itcontains categorical variables, and PCA doesn’t apply to them.
I The second command just picks out the matrix of projectionsof the variables on to the components.
I Called rotation because it can be thought of as rotating thecoordinate axes in feature-vector space.
# Do PCAnyt.pca = prcomp(nyt.frame[,-1])
# Omit the first column of class labels# Extract the actual component directions/weights for ease of referencenyt.latent.sem = nyt.pca$rotation
18 / 26
Bi-plot
−0.4 −0.3 −0.2 −0.1 0.0 0.1 0.2
−0.
4−
0.3
−0.
2−
0.1
0.0
0.1
0.2
PC1
PC
2
1
23
4
5
6
7
8
910
111213
1415
1617
18 19
20
2122
23
24
252627
28
29 30
3132
33
34
35
36
3738
39
40
41
42
4344
45
4647
48
4950
51
52
53
54
55
56
57
5859
60
61
6263
64
6566
67
68
69
70
71
72
73
74
75
76
77
787980
81
82
83
84
85
86
87
88
89
90
9192
93
94
95
96
97
9899
100
101102
−0.2 −0.1 0.0 0.1
−0.
2−
0.1
0.0
0.1
X.X.dX.nd
X.s
X.thX.thisaabandonedabcabilityable
about
aboveabroadabsorbedabsorbingabstractabstractionabstractions
abundanceacademicacademyaccentsacceptaccessaccessible
acclaimedaccommodateaccompaniedaccompanyingaccordingaccordinglyaccountaccountedaccusedachievedachievement
acknowledgeacknowledgedacquiredacquisitionacquisitions
acreacrossacrylics
act
acted
actingaction
actionsactiveactivities
actoractors
actressacts
actuallyadamadamsadamssadaptationaddaddedadding
additionadditionaladdressaddressesaddsadheringadjacent
administrationadmiredadmissionadmitsadoptedadsadults
advanceadvancedadvantage
adventureadventurousadvertisementsadvertising
adviceadvisedadviseradvisingadvocatesaestheticaffair
affairsaffectaffectedaffectionaffordafraidafrica
africanafricasafter
afternoonafterwardagain
againstage
agentaggressiveagingagoagreeagreedagreementahead
aimair
airy
alain
alanalbeitalbert
albumalertalfredalicealienalienationalignedalive
allallen
alliancealliances
allowallowed
allowingallows
alludeallusionsalmost
alonealongalready
alsoalteralternate
alternatesalternativealthoughalwaysam
ambiguousambiguouslyambitionambitionsambitious
america
americanamericansamid
amongamountamsterdamamusementamyananalysisanchored
ancient
andandrea
andrew
andy
angeles
angelsangerangryanimalanimals
animatedannanneanniversary
announcedannual
anonymousanotheranselansweranthonyantiantonioanxietyanyanyoneanythinganywayapart
apartmentapparent
apparently
appealingappear
appearanceappearancesappearedappearing
appears
applause
appleappliesappreciateappreciationapprehensionapproachapproachedapproachesappropriate
approvalapproveapproved
april
archetypalarchitectarchitectsarchitecturalarchitecture
areareasarent
aria
armarmsarosearoundarranged
arrangementarrangementsarrestedarrivearrived
art
arteartfularthurarticle
artifactsartificial
artist
artistic
artists
arts
artworkartworks
asasideask
asked
aspectsassemblagesassembledassertionassistantsassociateassociated
association
associationsassumed
assuranceatatheneumatlanticatmosphereattemptattemptsattendedattendingattentionattitude
attitudesattorneyattractattractedauauctionaudacity
audience
audiencesauditoriumaugaugust
austin
austriaauthenticauthor
authoritiesauthorityautobiographicalautobiographyavailable
avantavenueavoid
award
awards
awareawarenessawayaweawkwardaxisbbabies
baby
backbackdropbackersbackgroundbackgroundsbackingbackstagebackwardbaconbad
badlybakerbalancebalancedballballadballet
balloonbamboobanalbanality
band
bands
bankbanks barbare
barefootbarely
baritone
baroquebarrelbarriersbased
basicbasisbassbathed
battlebaybebeachbear
beatbeating
beautifulbeautifully
beautybeauxbecamebecause
becomebecomesbed
beenbeethovensbefore
beganbegin
beginningbegins
begunbehaviorbehestbehind
beigebeingbelievebelievedbelievesbelieving
belowbenbeneathbergmanberlinbernard
bernstein
besidesbestbetterbetween
beverly
beyondbigbigger
biggestbill
billboardsbillionbillsbilly
biographerbirdbirds
birthbitbits
bizarreblackblackness
blacks
blakeblankblanketblastbleak
blendblended
blessedblockblocks
bloodbloomblow
blue
blues
bluntblurringboard
bob
bodiesbodily
body
bohemiabonusboogie
book
books
boom
boostbooth
borderborders
bored
born
bostonboth
bottombought
boulderboulevardbouncedboundboundariesbounds
bourgeoisbowl
box
boxesboy
brahmsbrancusibrandbrandywinebrant
brassbravadobravebread
break.breaking
breakthroughbreathedbreathing
brianbricebrickbridgebridgesbriefbriefingbrieflybrightbrilliantbringbringsbritainbritishbroad
broadcastbroadcasting
broadway
broke
brokerbronzebronzesbroodingbrookbrooklyn
brother
brothers
broughtbrownbruce
brushedbrushesbubblebuddingbudget
build
buildingbuildingsbuilt
bullbunchburiedburleighburnedburtonbusbushbushybusiestbusinessbusinessmanbustbusy
butbutchbutter
buybuyingbuzzbyc
cabaretcablecafecalif
californiacallcalledcallingcallscalmcalvin
cambridgecame
cameracampaigncampuscan
canadacanadiancanal
canceled
cannotcanscant
canvascanvases
capcapacitycapitalcaptaincaptivatedcapturecapturedcapturescarcare
career
careerscarlo
carnegiecarolcarouselcarriedcarries
carrycarryingcars
cartooncartoons
carvedcase
castcasting
castscasual
cat
catalogcatalogue
catastrophecatchcategoricallycategorycathedralcatherinecatholiccaughtcause
causedcautious
cbs
cd
ceaseceiling
celebratecelebratedcelebratescelebrationcelebratory
cellist
cementcenter
centerpiececenterscentralcenturies
century
cerebralceremony
certaincertainlycertificatechaddschairchairmanchallengechamberchampionshipschance
change
changedchangeschangingchannelchanting
chapinchaptercharactercharacteristiccharacterscharcoalchargecharlescharmcheapercheckcheckscheeredcheerfulchef
chelsea
chemicalchestchic
chicagochiefchild
childrenchildrens
childschinachinese
chipschoicechoiceschooses
choosingchoralchords
choreographedchoreographerchoreographychoruschoruses
chosechosen
chrischristianchristopher
churchchurchscinematic
circacirclecirclescitecitedcities
citizencitizenscitycityscivic
clarinetistclarityclark
classclassesclassic
classical
classics
clay
clearclearlyclichclientscliff
climateclimbclinical
clintonclips
close
closedcloselycloserclosesclosing
clothesclottedcloyingclubclubsclueco
coast
coat
cocacoincidecolacoldcolecollaboratedcollaborationcollagecolleaguecolleaguescollectingcollection
collections
collectivecollectorcollectors
college
color
coloredcolorfulcoloring
colors
columbiacolumncomcombinationcombined
combiningcome
comedy
comescomfortcomic
comingcommandcommentcommercecommercialcommercials
commissioncommissionedcommissionercommissionscommitmentcommittedcommitteecommoncommunity
compactcompagniecompanies
companycompanys
comparecomparedcomparisoncompellingcompetecompetition
complainedcompletecompleted
completely
complexcomplexitycomplicatedcomponentcomposed
composercomposers
compositioncompositions
computercomputers
conceivedconceptconceptual
concernconcernedconcerns
concertconcerts
concludedconcludesconclusionconcoction
concretecondemnconditionconductconductedconductor
conferenceconfidenceconfidentconfined
confirmscongressconjunction
connconnectedconnectionconnectionsconscious
consecutiveconservativeconsiderconsiderableconsideredconsists
constantly
constructedconstructionconstructionsconsumerconsumersconsumingcontainedcontemplativecontemporaries
contemporary
contentcontextcontinue
continuedcontinues
continuingcontourscontractcontractscontrastcontributed
contributescontributioncontributionscontrolcontrolledcontroversialconventionalconventionsconversationconveyedconvincedconvincingcool
cooper
coordinatedcopiedcopyrightcorcoran
corkcornercorporatecorrectioncorridor
costcostscostumecostumes
cotterartcottoncould
couldnt
councilcountcountercountless
countries
country
countryscountycouplecoupledcouplescoursecourtcourting
cousincovercoveredcoveringcovers
cowboycozycrabcrackscradlecraftcraftycramercrane
crawlcrazycream
creamycreatecreated
createscreatingcreationcreationscreativecreativelycreativitycreaturescreditcrewcriticcritical
criticizedcritics
crosscrowd
crowdedcrowdscrucialcrudecrycrystalcubist
culturalculture
cummingcummingscunningham
curatorcurators
curiouscuriouslycurrentcurrentlycurtaincustomers
cutcutscuttingcy
cyclecynical
ddaddadsdailydallasdamagedamaged
dancedanced
dancer
dancersdances
dancingdangerdarkdarkened
datedatesdaughter
daughtersdaumier
daviddavisdawndaydaysdaytime
dazzlingdedeaddeadlinedeadlydeadpandealdealer
dealers
dealing
deathdeborahdebt
debutdebutsdecdecade
decadesdecemberdecidedecideddecidedlydecidingdecisiondecorated
decorative
dedicateddeepdeeper
deeply
defiant
defineddefinesdefiningdefinitiondeftdegreedegrees
deldelayeddelicate
delightfuldemanddemanded
demandsdemonstratedenis
departdepartment
depictdepicteddepictingdepictionsdepictsdepressiondepthderiveddesdescendingdescribeddescription
designdesignateddesigned
designersdesignsdesiredesperatedespite
destroyeddestructiondetacheddetaildetailsdetermineddevelopdeveloped
developerdevelopers
developingdevelopment
deviseddevoteddi
diamond
dianadickdiddidnt
diedieddifference
differentdifferentlydifficult
digital
dimensiondinner
direct
directed
directingdirection
directly
director
directorsdirtydisappeardisasterdiscoverdiscovereddiscussdiscusseddiscussing
discussiondisk
displaydisplayed
dissolveddissonances
distantdistinctdistinctiondistinctivedistributiondistrict
divadive
diversedivideddivision
dodocumentary
doesdoesntdog
doingdollardollarsdolphin
domesticdominated
don
donald
donations
donedontdoordoorsdotdotsdoubledoublingdoubtdoug
douglasdown
dozendozensdrdrafteddrama
dramatic
drawdrawingdrawingsdrawndreamdreamily
dreamingdreamlike
dreamsdressdressed
dressesdrew
driftdrifted
drinkingdrivedrivendrivingdrummer
drydudueduetduke
dumpyduringdutch
dynamicdynastyeeach
eagereagerlyeagerness
earearlier
earlyearnedearnest
earth
earthly
easeleasily
easteasterneasyebullience
eccentricechoeseconomiced
edgeedges
editoreditorseducationedwards
effecteffectivelyeffectseffortefforts
eighteightheitherelelaborateelder
elderlyelectedelectronicelegant
elegantlyelementalelementaryelementselizabeth
else.elsewhereelvis
embeddedembody
emergingemilyemmyemotionally
emphasisemphasizedemphasizesempireempty
emulsionenable
encore
encounterencounteredencouragedend
endedendingendless
endsenergyengaged
engagement
engagingengineeringengineersengland
englishengravedenhancedenjoyenjoyedenjoymentenlargedenormous
enough
ensemble
enterenterpriseenthusiasmenthusiasticentireentirelyentrance
entrancesentrepreneurentries
entryenvelopingenvironmentenvisionedepisodesequallyequateequipped
eraericerroreruptedescapeespecially
essenceessentialessentiallyestablishestablished
establishmentestateestheticestheticallyestimate
estimated
et
etchings
europeeuropeaneuropeans
even
evening
eveningsevent
events
eventually
ever
everyeverydayeveryoneeverythingeverywhereevidenceevidentlyevokeevokedevokesevoking
evolutionexexactlyexaminationexampleexceptexceptionexceptional
exceptionallyexchangeexcitementexcitingexecutive
exhibitexhibitedexhibiting
exhibitionexhibitions
existexistenceexistingexitsexoticexpandexpansion
expectexpectedexperienceexperiencedexperiencesexperimentalexperimentingexpert
experts
explainexplained
explainingexplains
expletiveexploitsexplore
exploringexplosiveexposureexpressexpressedexpressingexpressionexpressionismexpressionistexpressionistic
exquisiteextendedextendsextensiveextentexterior
extraextraordinaryexultationeye
eyedeyesffabric
fabulous
facadefacefacesfact
fadefail
failedfailurefairfaithfallfalling
falls
falsefamefamiliarfamilies
familyfamilysfamous
fan
fancifulfantasyfar
farefascinating
fascination
fashion
fashionablefashionedfasterfatefatherfathersfavored
favoritefearfeaturefeaturesfeaturing
febfebruary
feelfeelingfeelingsfeels
feet
feldmanfeldmansfell
fellowfellowshipfelt
femalefeminist
festival
festivals
fewfewerfieldsfifthfightfightingfigurative
figure
figuresfiled
fillfilledfilmfilms
filter
finalfinallyfinancedfinances
financialfinancing
findfindingfinds
finefingerfingers
finishfinishedfirefiresfirm
firstfishfishingfitfitsfiveflagflairflamesflashesflat
flavoredfledflemish
flesh
flewflightflippedfloatfloating
floor
floorsflorence
flowflowerflowersfoam
focusfocusedfoldsfolk
followfollowedfollowersfollowingfollowsfood
foot
footballfootlightsfor.forbiddingforceforcedforcesfordforegroundforeignforest
foreverforget
forgotten
formformalformatformedformer
formsforwardfoundfoundationfoundationsfoundedfourfourth
fragmentationfragments
framefrancefrancescofrancis
franciscofrank
fredfrederic
frederickfree
freedom
frenchfrescoesfresh
fridayfriendfriends
friendshipfringesfromfront
frozenfrustrationfueledfull
fullyfultonfun
function.functionalfunctions
fund
fundamentallyfunny
furniturefurtherfusionfussfuturefuturisticg
gabrielgag
gagosiangaingained
gaining
galleries
gallery
gallerysgamegames
gapgaragegarbage
garde
garden
gardensgarygate
gathergatheringgave
gaygeldergeneralgenerallygeneratedgenerationgenerationsgenre
genresgentlegentlygeometric
george
geraldgerman
germanygertrudegesturegestures
get
getsgettinggiantgiantsgiftgildedgilman
girlgirls
giuliogiuseppe
givegivengivesgivingglamorous
glassglennglimpseglobalglobegloryglue
glueckart
gogoalgodgodsgoesgoghgoinggold
goldengonegoodgoodmangoods
gospelgotgothic
governmentgovernmentsgowngracegraduallygraffitigrahamgrammygrandgrandiosegranitegrantsgrasp
grass
graygraysgreatgreatergreatest
greekgreen
greenwichgrew
gridgridsgroom
ground
groundsgroupgroupsgrow
growinggrowsgrowthguarded
guards
guestguggenheim
guideguidelinesguilty
guitarguitarist
gun
guyguysh
had
hairhalf
hallhalls
hallucinatoryhallway
hand
handedhandlehandledhandlinghandshandsomehanghanging
hans
happenhappenedhappening
happenshappyharborhard
harderharlemharmonicharmonies
haroldharrisonsharshhartford
harvard
harvestedharvey
hashasnthat
hauntinghavehavenhavinghaydnhe
headheadache
headedheadingheads
healthy
hearheard
hearinghearingsheartheavenheavily
heavyheeledheldhelen
helphelpedhelpinghencehenry
her
hereheritageheroheroesheroic
herselfhes
hicksville
hidden
hide
high
higherhighest
highlyhillhills
himhimself
hintedhintshiphipshiredhishistorianshistoric
historicalhistory
hithitshockeyhoffman
holdholdingholdings
holdsholeholland
hollywood
homagehome
homelesshomerhomes
honghonor
honoringhop
hopehopeshopinghorsehospitalhosthot
hotel
hour
hourshouse
householdhouseshouston
how
howardhowever
howshugehugohuman
humanshumidityhumor
hundredshungryhunthunter
hurthusbandhustle
hybrid
i
icons
id
ideaidealisticidealizeidealsideasidenticalidentifiedidentifyidentityideological
if. ignatiusignoreignores
ii
illillegalilluminatedillusionsillustrateillustrational
im
image
images
imaginationimagine
imitation
immediate
immediatelyimpendingimplicationsimplied
impliesimportance
importantimpossibleimpresarioimpressimpression
impressionismimpressionist
impressiveimprovedimprovementimprovisationimprovisations
in.inchinclude
includedincludes
includingincomeincorporatedincorrectlyincrease
increasingincreasinglyindeedindependentindianindicatedindividualindoorsindustrialindustryinfamousinfatuation
inflate
inflated
influenceinfluencedinfluentialinformalinformationinformed
infraredinhabitedinherentinherited
initiallyinjuriesinner
innocenceinside
insistedinsists
inspirationinspiredinstallinstallationinstallationsinstalledinstanceinstantinstead
instituteinstitutionalinstitutionsinstrumentalinstrumentsinsuranceintact
intellectualintellectuallyintendedintenseintent
intentionalintentionallyinteractinterestinterested
interestinginterestsinteriorinteriorsinternational
internetinterpretationinterviewinterviewsintimacyintimateinto
introduced
introducingintroduction
invasioninventioninventionsinvestment
invisibleinvitationinvitedinvokeinvolve
involved
involvementinvolvesinvolvingiris
iron
ironyirvingisisland
islip
isnt
israeliissueissuedissuesit
italianitalians
italy
italysitemsits
itself
ive j
jackjacketjacksonjail
jamesjan
janejanisjanuaryjapan
japanesejasper
jaunty
jazzjeffjenniferjeremyjerry
jersey
jetsonsjewelryjimmyjobjoejohnjohnny
johnsjohnsonjohnsonart
johnssjoined
jointjokejokesjon
jonathan
jonesjosephjoyjrjuan
juliejuly
junejunglejunior
justjustice
juxtapositionkkane
keenkeepkeepingkenkennedy
kennethkeptkevin
key
kidskileykilled
killingkindkinds
kingkingskiosk
kisskneeknewknockedknowknowingknowledgeknown
knows
kong
kyl
la
lablabellabeled
labelslaborlacklackinglacks
land
landed
landscapelandscapes
lane
languagelarge
largelylargerlargest
larklarry
last
lastedlastinglatelaterlatestlatinolatter
laughlaughedlaughinglauralaurencelavish
law
lawnlawrence
lawyerslayers
leleadleaderleading
leagueleaguesleanleapslearn
learnedlearningleastleave
leavesleaving
ledleeleftleftover
legal
legendlegendarylegislatorslegitimatelegs
leighlendlengthlengthyleno
lentleonleonard
leslielessletlets
letterlettermanletters
lettinglevellevelsliamlibertylibrary
librettistlibrettolichtenstein
lieslifelifeslifetime
light
lighterlightinglights
like likedlikelylikes
limitedlimousinelincolnlindalinelined
lineslinklinks
lionliquid
lisa
listlisten
listenedlisteninglistsliterallyliterary
lithographslittle
livelived
liveslivinglloydloanloanslobbylocal
location
lockedloeilloftlondonlonelonelylonglonger
longtimelook
lookedlookinglookslooseloosely
lordloslosing
losslostlotlotsloudlouislounge
love
lovedlovelylover
loversloveslowlowerloyallucasluckluminous
lureslustluxurylyinglynnlynne
lyriclyricism
lyrics m
mamache
machine
machomacys
mademadisonmagazinemagazinesmagicmainmainstream
majesticmajor
makemakesmakingmalemallmanmanagedmanages
manet
manhattan
manholemanipulatemanipulatedmannermansmansion
mantegnamanuelmanymapmarble
marc
march
mariemarkmarkedmarket
marketingmarriagemarriedmarthamarvelousmary
maskmasksmasonmassmassesmassivemastermasterpiecemasterpiecesmastersmatchmatched
matchingmaterial
materials
matinee
matissemattermatters
mauricemax
maymaybemayormcdonald
me
meadows
mean
meaning
meansmeantmeanwhilemeasure
measuringmeat
media
medicalmeditationmediummeetmeeting
meetingsmeldsmelodramaticmelodymembermembersmemoirsmemory
menmenacing
menilmensmention
mentionedmentionsmerce
merely
merge
mermaidmess
messagemessages
met metalmethodmethodsmetropolismetropolitanmexicomezzanine
mezzo
michaelmicroscopicmid
middlemidnightmidtownmightmikemiles
militarymillenniummillermillionmillions
mindmindedmineminesminiatureminimalminimalistminimumministerminneapolisminor
minorityminuteminutesmiracle
mirrormirrorsmismatched
miss
missedmissingmissionmisspelledmisstated
mistressmitchellmixmixedmixesmixingmock
modemodelmodelsmodern
modernismmodernistmodestmogulmomentmomentarilymoments
momentummonday
mondriansmonetmoney
monochromesmonolithicmontages
monthmonthsmontrealmonumentalmood
moon
mooremoremoreovermorning
morris
moscowmostmostly
mother
motifmotifsmotionmountmountain
mountainsmountedmouthmove
movedmovementmovesmovie
moviesmoving
mozartsmr
mrs
ms
mtvmuchmugmultimediamultiple
mundanemuralmuralsmurray
museum
museums
musicmusical
musicalsmusicianmusiciansmusingsmust
mutedmutual
my
myers
mysteriousmysterymythological
nna
nails
naked
namenamednames
nancynarrativenarrativesnarrow
nashvillenassaunational
naturalnaturally
nature
nbcneal
nearnearbynearlyneatlynecessarilynecessarynecessity
neck nedneedneedednegotiatednegotiationsneighborneighborhoodneil
neithernelsonneo
nervousnetwork
networks
neverneverthelessnewnewestnewly
newman
newsnewspaper
next.
nicenicelynicholasnicknicole
night
nightclubnightsnine
no
nobodynoisenominatednominationsnondescript
nonenonethelessnonprofitnornormnormallynorthnorthernnostalgia
nostalgicnotnotablenotably
notenotednotes
nothingnoticenoticednotingnotion
novnovelnovembernow
nowherenudenumbernumbers
numerousnurtures
oobject
objects
obliterateobsessed
obstaclesobtainedobviousobviouslyoccasionoccasionaloccasionallyoccuroclock
octoctober
odds
ofoffoffensiveoffer
offeredofferingoffersofficeoffices
officialofficialsoftenoh
oiloilsold
olderoldestoleolsonominousononce
oneonesongoingonlineonly
onstage
onto opopenopened
openingopens
opera
operasoperatic
operatingoperationopportunitiesopportunityopposedoppositeoptimismoptimisticoptions
opulent
ororange
orchard
orchestra
orchestralorchestras
order
organorganization
organizations
organizedoriginaloriginality
originallyososcar
otherothersotherwise
ourourselvesout
outdooroutputoutrightoutsideoutsider
outstandingovaloveroverall
overcrowdedoverlappingoverlookingoversightowed
ownownedownerownersowns
ppapablopaces
pacific
packpackedpage
pagespaidpain
paint
painted
painter
painterlypainters
paintingpaintings
paintspairpairing
palace
palettepampanpandemonium
panel
panelspanicpantspaolopaper
papierpapp
parents
paris
parisianparkparks
parsonspartparticularparticularlypartiespartlypartnerpartnerspartnershippartnershipsparts
partypass
passedpassingpassion
passionatepast
pastichepatpatch
patentlypathpathspatrick
patronpatronspattern
patterns
pattipaulpavilion
pay
payingpeace
penchantpentagonpeopleperpercentpercentageperceptionperchedpercussionpercussionists
perfectperfectly
perform
performanceperformances
performedperformersperforming
perhapsperiodperiodspermanent
permissionperson
personal
personalitiespersonalityperspectivepeterphallusphase
phenomenonphiladelphia
philharmonic
philipphilippines
phillipsphilosophicalphone
photophotograph
photographedphotographerphotographersphotographicphotographsphotography
photos
physical
pianissimopianistpiano
picasso
pickpicked
picking
pictorialpicturepictures piecepiecespierre
pilepiles
pills
pinkpioneerpiratedpitplaceplaced
placesplagued
plainplanplanet
plannedplanning
plansplantedplants
plasterplastic
play
playedplayer
playersplayful
playingplaysplaywright
playwrightsplaywritingpleasepleasurepleasures
plentyplucky
plumplusplush
plywoodpoetpoetrypointpointedpoints
polaroidpolepolicepolished
politicalpoliticspolkepoolpoor
pop
popepopularpopularityportporter
portionportraitportraits
portrayspositionpositionedpositionspossesspossession
possibilitiespossibility
possiblepossiblypostposterpostersposturingpotentialpower
powerfulpracticalpracticepraisedprepreacherprecisely
precision
precocious
prejudice
premiere
premierespremisepreparationpreparatorypreparedpreparingpresence
presentpresentedpresenterspresentingpresentspreservationpreservepresidentpresspressingprettypreview
previews
previouspreviouslypricepricesprimalprimarilyprimaryprimeprince
principal
printprintedprintingprintmaking
printspristineprivateprivately
prizeprobably
problem
problemsprocessprocessesproduceproduced
producerproducers
producesproducingproduct
production
productions
productsprofession
professionalprofessionalsprofessor
profileprofitsprofoundly
programprogrammingprograms
progressprojectprojectionprojectionsprojectspromise
promisedpromisingpromotepromoting
prompted
proof
propaganda
propelledpropertyproportionproportionsproposals
proposedpropsprosceniumprospectprot
protectprotectedprotectionproud
proved
provesprovideprovidedprovider
providesproviding
provocativepruningpsychedelicpsychologicalpub
public
publicationspublished
puccinipulledpullspulsepunishment
punspurchase
purchasespurepurely
puritypurplepurposepurposespursued
pursuingpushputputsputtingpuzzlingqualitiesquality
quantitiesquarterquartet
queen
queens
questquestionquestionedquestioningquestionsquick
quietquietlyquilts quintetquite
r
rabbit
radicalradicallyradio
ragerailingsrainraiseraised
raisingralph
ranrandallrange
rangingraphael
rapidrapperrare
rarelyrarity
ratrate
ratherratingrauschenbergraw
rayrazed
rcarereachreachedreadreadingreadyreal
realismrealisticrealityrealizerealizedreally
realmreasonreasonsrebeccarecallrecalled
recalling
recapturereceive
receivedreceives
recentrecently
recitalrecitalsrecognitionreconstructs
recordrecordedrecordingrecordingsrecords
recoverrecoveredrecreaterectanglerectangularrecurring
red
reducedreductionreference
referencesreferred
referringreflectreflectedreflectingreflectsrefuserefused
regional
registerregistryregular
regulations
rehearsal
reinforcerejectedrejectionrelate
relatedrelationsrelationship
relativerelativelyrelease
releasedrelevant
reliefsreligion
religious
relyremainremainedremainingremainsremarkableremarksrembrandtrememberremembersremindreminder
removedremoving
renaissancerenderedrenderingrenderings
reneerenovatedrenovatingrenovationrent
rentedreopen
reopenedreopeningrepeatedrepeatedly
repertoryrepetitionreplacereplacedrepliedreport
reportedreportersreports
representrepresentationrepresentativerepresentativesrepresentedrepresents
repriserepublicanreputationreputationsrequirerequired
requiresresearchresemble
resembles
reservationsresidenceresidentresinresistance
resolvedresourcesrespectablerespectivelyresponseresponsible
restrestaurantrestaurantsrestingrestless
restorationrestoredresultresultedresultsretainedretiredretitledretreatretrospective
returnreturnedreturningreturnsrevealed
revealingrevealsrevenuereverencereversereview
revisionsrevisitrevival
revivals
revolutionrevolutionary
rewardrhythm
ricerichrichardrichardsonrichnessrid
riderifle
rightringingrippleriserisesrisingrisk
ritaritual
riverriversroadroadsrobrobertrobertarobesrobinson
robust
robustlyrochelle
rock
rocketrocks
rocky
roleroles
rollrollingromanromanceromanticromeronald
room
roomsrootroots
roserossrothenberg
rothkoroughroughlyround
routineroutinely
rowroyroyal
ruckusruderuinsrules
runrunning
runs
ruscharushrushingrussiarussianryder
s
sagasaidsaint
saints
salarysale
salessallesalutesamsamaras
samesamplingsamuelsan
sang
sarahsardonicsargentsatsatisfaction
saturday
saulsavedsawsawssaxophonesay
saying
says
scale
scandalscaryscattered
scenarioscenesceneryscenes
schedulescheduledscholarscholarsscholarship
schoolschoolsschwartz
science
scorescoresscottscramblescreaming
screen
screeningsscreens
sculptorsculptural
sculpturesculptures
seasearch
searching
season
seasonsseatseatingseatsseattlesecond
secretsection
sectionssecurityseductiveseeseeing
seekingseemseemed
seeminglyseems
seenseesselected
selectionselfsellsellingsemisenatesenatorsendsendingsenior
sensesensualsentsentimentsentimental
separate
septseptember
sequenceserene
seriesserious
seriousness
serveserved
service
servicesset setssetting
settingssettlesettledseven
severalsevereseverelysexsexual
shadesshadow
shakespeareshakespeares
shapeshapedshapesshapingsharesharedsharessharonsharp
sharply
she
sheershell
shes
shimmeringshineshinyshipsshirt
shirtsshockedshoesshootingshop
shoppingshops
short
shortlyshotshotsshould
shouldersshout
shouting
showshowedshowingshown
showsshrill
shutside
sidessightsightingssigmarsignsignaled
signaturesignedsignificantsigningsignssilhouettedsilhouettessilk
silver
silverysimilarsimilarlysimplesimplersimplicitysimply
simpsonssimultaneously
since
sing
singersingerssinging
single
singssinksinking
sirsistersitcomsitcomssite
sitessitssitting
sixsixthsize
sizedsizesskepticalsketch
sketchesskinskirtskyskylightskylineskyscrapers
slashsleeksleepslide
slightlyslow
slowly
small
smallersmartsmearingsmellsmith
smithtownsmokesmoothsnakesnakessnappy
snoutsnowso soapsocial
societiessociety
softsofter
softwaresoho
soldsoldiersolidsolosoloists
solomonsomber
somesomebodysomehowsomeonesomething
sometimessomewhat
sonsonatasondheim
song
songs
songwriterssonic
sonssoon
sophisticated
soprano
sortsought
soulsoundsoundedsounds
sourcesouth
southern
sovietspace
spacesspaceyspainspanishsparespatialspeakspeakingspecialspecializesspecialtyspecificspecificallyspectacularspeculationspendspendingspent
spherespinal
spirit
splashspokespokesmansponsoredsponsorssports
spotlight
spreadspring
spur
squaresqueezed
squid
st
stablestaff
stage
stagedstages
staging
stainedstaircasestandstandardstandardsstandingstands
stanley
star
starestarkstarredstarringstars
startstartedstarting
startlingstartsstatestatement
statementsstates
staticstationstatuary
statue
statuettesstatus
staysteadysteelsteepedstefanostein
steinbergsteinway
stentorian
stepstephen
steppedsteppingsteps
stevestevenstickstill
stillsstimulating
stint
stitched
stone
stonesstoodstopstopped
storestoresstoriesstory
straightstrangestreamstreetstreetsstrengthstrengthsstressesstretchstretchedstrictlystrikestriking
stringstringsstripstrippedstripsstrokesstrollstrongstrongeststronglystructurestructures
strugglestuckstudded
studentstudentsstudied
studiesstudio
studios
studystuffsturdystyle
stylesstylish
stylistically
subjectsubjectssubsequentsubstancesubstantialsubtlesubtletysuburb
suburbansubversivesucceed
succeededsuccesssuccessfulsuccessor
suchsuddensuddenly
sue
sufferedsufficientsuffolksuggestsuggestedsuggesting
suggestivesuggests
suitsuitssummer
summerssun
sundaysung
sunlightsunny
sunset
supersuperimposedsuperintendentsupport
supported
supportingsupposedsuresurely
surfacesurfaces
surgesurgeonssurprisesurprisessurprisingsurprisingly
surreal
surrealistsurroundedsurroundingsurroundingssurroundssurvey
survivesusan
suspectsuspended
sweetsweetlyswingswingingswirlingswisssymbolsymmetry
symphony
synonymoussystem
ttaaffetabletacticstailtaiwantake
takentakestaking
tale
talenttalestalk
talkingtalkstall
taptapestask
taught
tax
taylortchaikovskyteamteamsteartearstechtechnically
techniques
technology
teenstelephonetelevision
televisions
tell
tellingtellstemporarytendtendedtendencytender
tendstennessee
tenor
tentativelytenuretermtermsterritorytextexas
texttextstexturethanthankthat
thats
the
theater
theaters
theatricaltheirthemthematicthemethemesthemselvesthen
theretheres
these
they theyrethickthievesthinthingthingsthinkthinkingthinksthirdthis
thomasthosethough
thought
thoughtfulthoughts
thousandsthreatthreateningthreethrewthrift
throughthroughoutthrowthrown
thursday
thusticketticketmaster
tickets
tietiestimbre
time
timedtimelesstimestimingtimpanitintiny
title
titledtitlestotoccatatodaytodaystogethertold
tomtomkinss
tomorrowtonalitytonetoned
tonight
tonightstony
tootooktoolstoptopped
topstorsototaltotemtouchtouchestouchingtough
tourtouredtouring
tourists
tours
towardtowardstowertowers
towntownstoytracetraces
tracktractortradetradedtradition
traditionaltraditionallytraffic
tragedytrainingtransformtransformationstransformedtransformingtransgressionstransparenttrashtravel
traveledtravelerstraveling
treasurestreasurytreat
treatedtreatmenttreetreestriangle
tribute
tricktricky
triedtries trio
triptrompetroubletrucktruetrulytrust
trytryingtucked
tuesday
tuesdaystullytune
tunes
turn
turnedturner
turningturnstvtwice
twisttwotwomblytype
u
ultimateultimatelyunableuncannyuncertaintyunclearunderunderstanding
undoubtedlyundulatingunfinished
unfortunateuniformunionuniqueunited
universaluniversity
unknownunlikeunlikelyunnoticed
unrelated
untiluntitledunusual
unwelcomeup
uponupperupsideurbanurge
ususeused
useful
usesusingusualusuallyutterv
vavaguely
vainvalley
valuablevaluevaluesvanvaried
varietyvarious
varyingvastvaudeville
vevegasvenicevenue
venus
verdisvernonversesversionversionsveryveterans
vivibrant
vicevictimsvictorvictoriavictory
videovienna
vienneseviewviewed
viewerviewers
viewingviews
village
vincentvintageviolenceviolinviolinistviolins
violistvirtuallyvirtuosity
visiblevisiblyvisionvisit
visitedvisitingvisitor
visitors
vistavisualvivid
vocal
vogue
voice
voicesvolcanic
volumevonvulnerablew
wadingwadsworthwagners
waitingwaivedwalk
walkedwalker
walkingwalks
wallwalls
walterwaltzwantwantedwants
war
warholwarholswarm
warmthwarnerwarrenwars
was
washwashingtonwasntwatchwatcheswatchingwaterwatercolor
watercolorswatsonwavewaxwaywayswe wealthweaponwear
wearingwearsweather
webwebb
wednesday
week
weekendweeklyweeksweiweightweillwelcome
wellwentwere
westwesternwetwhat
whatswheelswhen
wheneverwherewhetherwhich
while.
white
whitney
who
wholewhomwhosewhywichita
widewidelywifewildwildly
will
williamwilliams
williamsburgwillingwilson
wind
windowwindows
wingwingswinning
winslowwinterwipedwirewisely
wish
witwitchwith
withinwithoutwitnesswitnesses
wittywolf
wolfgang
womanwomanswomen
womens
won
wonderwonderfulwonderingwont
woodwoodcutswooden
woogiewoosterwordwordswore
workworked
workersworking
works
workshopworkshopsworldworlds
worldwideworried
worryworseworstworthwouldwouldnt
wound
writerwriters
writes
writing
written
wrong
wrote
wrywww
xavieryyale
yangyard
yearyearsyellow
yesyesterday
yetyieldingyogiyorkyorkersyorks
you
youngyounger
youryoureyouth
youthfulyu
I This shows the two first PC for the NYtimes data.I Blue scores represent the scores for the first two PC’s.I The red arrows represent the first two PC loading vectors.I Here, a bi-plot is not useful at all since the data is too high dimensional. (See Figure 10.1 for where a
bi-plot is useful.)19 / 26
Look at the leading components# What are the components?# Show the 30 words with the biggest positive loading on PC1signif(sort(nyt.latent.sem[,1],decreasing=TRUE)[1:30],2)
## music trio theater orchestra composers opera## 0.110 0.084 0.083 0.067 0.059 0.058## theaters m festival east program y## 0.055 0.054 0.051 0.049 0.048 0.048## jersey players committee sunday june concert## 0.047 0.047 0.046 0.045 0.045 0.045## symphony organ matinee misstated instruments p## 0.044 0.044 0.043 0.042 0.041 0.041## X.d april samuel jazz pianist society## 0.041 0.040 0.040 0.039 0.038 0.038
# biggest negative loading on PC1, the other end of that scalesignif(sort(nyt.latent.sem[,1],decreasing=FALSE)[1:30],2)
## she her ms i said mother cooper## -0.260 -0.240 -0.200 -0.150 -0.130 -0.110 -0.100## my painting process paintings im he mrs## -0.094 -0.088 -0.071 -0.070 -0.068 -0.065 -0.065## me gagosian was picasso image sculpture baby## -0.063 -0.062 -0.058 -0.057 -0.056 -0.056 -0.055## artists work photos you nature studio out## -0.055 -0.054 -0.051 -0.051 -0.050 -0.050 -0.050## says like## -0.050 -0.049
# Ditto for PC 2signif(sort(nyt.latent.sem[,2],decreasing=TRUE)[1:30],2)
## art museum images artists donations museums## 0.150 0.120 0.095 0.092 0.075 0.073## painting tax paintings sculpture gallery sculptures## 0.073 0.070 0.065 0.060 0.055 0.051## painted white patterns artist nature service## 0.050 0.050 0.047 0.047 0.046 0.046## decorative feet digital statue color computer## 0.043 0.043 0.043 0.042 0.042 0.041## paris war collections diamond stone dealers## 0.041 0.041 0.041 0.041 0.041 0.040
signif(sort(nyt.latent.sem[,2],decreasing=FALSE)[1:30],2)
## her she theater opera ms## -0.220 -0.220 -0.160 -0.130 -0.130## i hour production sang festival## -0.083 -0.081 -0.075 -0.075 -0.074## music musical songs vocal orchestra## -0.070 -0.070 -0.068 -0.067 -0.067## la singing matinee performance band## -0.065 -0.065 -0.061 -0.061 -0.060## awards composers says my im## -0.058 -0.058 -0.058 -0.056 -0.056## play broadway singer cooper performances## -0.056 -0.055 -0.052 -0.051 -0.051
20 / 26
Look at the leading components
I These are the thirty words with the largest positive andnegative projections on to the first component2.
I Words with positive projections mostly associated with musicI Those with negative components with visual arts.I Why do we see women and mothers here?
2Which direction is positive and which is negative depend on internal choicesin the PCA algorithm
21 / 26
PCA for NYTimes# Plot the projection of the stories on to the first 2 components
# Establish the plot windowplot(nyt.pca$x[,1:2],type="n")
# Arts stories with red Aspoints(nyt.pca$x[nyt.frame[,"class.labels"]=="art",1:2],pch="A",col="red")
# Music stories with blue Mspoints(nyt.pca$x[nyt.frame[,"class.labels"]=="music",1:2],pch="M",col="blue")
−0.4 −0.3 −0.2 −0.1 0.0 0.1 0.2
−0.
3−
0.2
−0.
10.
00.
10.
2
PC1
PC
2
A
A
A
A
A
A
A
A
A
A
AAA
A
A
AA
AA
A
AA
A
A
AA
AA
AA
A
A
A
A
A
A
AA
A
A
A
A
A
AA
AA
A
AA
A
A
A
A
A
A
A
MM
M
M
MM
M
MM
M
M
M
M
M
M
M
M
M
M
M
MM
MM
M
M
M
M
M
M
M
M
M
MM
M
M
M
M
M
M
M
M
MM
# The separation is very good, even with only two components.
22 / 26
How well is PCA doing?
Even though we have gone from 4431 dimensions to 2, and thrownaway a lot of information, we could draw a line across this plot andhave most of the art stories on one side of it and all the musicstories on the other.
If we let ourselves use the first four or five principal components,we’d still have a thousand-fold savings in dimensions, but we’d beable to get almost-perfect separation between the two classes.
This is a sign that PCA is really doing a good job at summarizingthe information in the word-count vectors, and in turn that the bagsof words give us a lot of information about the meaning of thestories.
The figure also illustrates the idea of multidimensional scaling —finding low-dimensional points to represent high-dimensional data bypreserving the distances between the points.
23 / 26
Scree Plot
I We can figure out the number of principal components byfitting what’s called a scree plot.
I Choose the smallest number of principal components that arerequired such that an adequate amount of variability isexplained.
I We look for the point at which the proportion of varianceexplained by each subsequent principal drops off.
I This is called the elbow of the scree plot.I These plots are application specific and ad-hoc.
24 / 26
Scree Plot
plot(nyt.pca,type="l", main="")V
aria
nces
0.01
20.
014
0.01
60.
018
0.02
0
1 2 3 4 5 6 7 8 9 10
25 / 26
Summary
I Visual analysis of PCA is adhoc (such as the scree plot).I In fact, the question of how many PC to choose is very
ill-defined in practice and depends on a specific application.I In practice, we tend to look at the first few PC in to find
interesting patterns in the data.I If not interesting patterns are found, then further PC are
unlikely to be of interest.I We could use PC in the context of regression (see Section 6.3.1
for further details and this is more principled).
26 / 26