top secret ii si tl rel to usa, aus, can, gbr, nzl

IIII4V I n *

1 f*

I ( « 1 1 • M K t

TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL

December 2012

t i l l *

TOP SECRET if SI tt REL TO USA, AUS, CAN, GBR, NZL

TOP SECRET // SI I! REL TO USA, AUS, CAN, GBR, NZL 2

m u m p —

1 u :

TOP SECRET // SI f! REL TO USA, AUS, CAN, GBR, NZL

TOP SECRET II SI tf REL TO USA, AUS, CAN, GBR, NZL

Lesson Objectives

VIntroduction to XKEYSCORE ^Purpose and Capabilities VData Flow

VWhat is a Cluster? VXKEYSCORE Databases

TOP SECRET // SI ft REL TO USA, AUS, CAN, GBR, NZL

• XKEYSCORE performs filtering and selection to enable analysts to quickly find information they need based on what they already know.


• XKEYSCORE also performs SIGDEV functions such as target development to allow analysts to discover new sources of information.



• XKEYSCORE processes data at field sites, where it is collected, and allows analysts from all over the world to query it.

• At field sites, the XKEYSCORE software can run in clusters of few or many servers, giving it the ability to scale in both processing power and storage.

• All processing is plugin or fingerprint based, which allows new capabilities to be quickly deployed to support operational needs.

TOP SECRET // SI fl REL TO USA, AUS, CAN, GBR, NZL 5

• XKEYSCORE is a Computer to Computer (C2C) exploitation system.

• It is a fully distributed processing and query system.

• XKEYSCORE can run on multiple servers.

• Plugin and fingerprint architecture allows new capabilities to be quickly deployed.


. XKEYSCORE is typically installed with Red Hat AS5u8 operating system. The suggested disk set up is: • Set up separate partitions for / (root), /var,

/tmp, and /export/data • XKEYSCORE clusters can be composed of three

different functionalities, which are: • One host acts as the web server/user interface, etc... • Another host normally runs as the real-time processing unit • Other host acts as the search or query system.

• Hybrid system can perform multiple roles on one server, which enables efficient registration. • process_data_parent • 1 queryproc



• The backend is where the raw data for XKEYSCORE is processed; that is, we receive information from our sources (e.g. WEALTHYCLUSTER2), process it, and store it into a database.

[sessions] [processing engine] (database) > (user queries)

L phone numbers

email addresses

[ or log ins

user activity


Trr ------- TOP SECRET II SI ti REL TO USA, AUS, CAN, GBR, NZL

Data Flow - XKS Cluster

A cluster is comprised of one master server and one or more slaves.

• All slaves in a cluster have their own copy of configurations (/opt/xkeyscore/config) files via the xks rsync push_config cronjob.


There are two types of databases on ari XKEYSCORE system: insert (¡0) and query (qO)


NOTE: sotf_input_proc is now called, sotf_dist process_dataN's are now called, process_data_parent


registerjittiadatajables

) 1 0 O l 1

I O D I I 0 0 1 1 0 0 1 Trr ------- TOP SECRET II SI ti REL TO USA, AUS, CAN, GBR, NZL

Data Flow - Databases ) 1 0 1 O l O O O l 0 O l I O I Q I O

file_input_proc and sotf_dist take in sessions from the front-end and load balances them across multiple process_data_parent's. process_data_parent is responsible for processing sessions and extracting metadata xks_meta_ingester takes the metadata from the process_data_parent's and writes it to the insert database, iO register_metadata_tables takes completed insert tables, indexes them, and moves them to the query database, qO

TOP SECRET // SI fl REL TO USA, AUS, CAN, GBR, NZL 1 1

i> P " T ? n , 1 0 0 0 0 - , ; ooi ion. .on, not

! • ! I ' " • 1 u i » 0 3 J •


i

TOP SECRET // SI I! REL TO USA, AUS, CAN, GBR, NZL 1 2


Lesson Objectives

Operating System Services

s/ MYSQL •*/ NFS VAUTOFS

/ Mount Points //xks_data

/ Directory Structure



Operating System Services

• XKEYSCORE is typically installed on servers running Red Hat 5u8 operating system.

• This section discusses common operating system services used during XKEYSCORE operation.


The http daemon is needed for the web-based GUI, viewing content, and is required on all servers.

The master server is the web server and the slaves retrieve content through HTTPS.


TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL • .

MySQL Daemon

The mysql daemon is a SQL-based database server for processing, querying, and is needed for the XKEYSCORE GUI.

• It is required on all servers for administration, processing, and querying metadata in databases.


NFS TOP SECRET II SI II REL TO USA, AUS, CAN, GBR, NZL

• Mounting a directory uses the NFS service. • NFS allows file systems that physically reside

on one computer to be shared by other computers on the network.

• The machine with the hardware containing the directory must allow the hardware to be made available to other machines.

• Required on all computers for clustering.

TOP SECRET II SI II REL TO USA, AUS, CAN, GBR, NZL 1 7


/etc/exports /export/data/xkeyscore master(rw) slave(rw) /opt/xkeyscore/config/loadserver *(rw)


TOP SECRET // SI f! REL TO USA, AUS, CAN, GBR, NZL •

Computers requiring shared access to the /export/data/xkeyscore directory must be told where to find the directory. • This is accomplished via automounting.

The autofs daemon listens for computers trying to connect to the directories, or mounts, that it is responsible for. The mounts are dropped after a time out, but autofs remounts the drive when drives need to be accessed.


• For a clustered XKEYSCORE, automounts must be set up on all of the computers in the cluster.

• auto.master and auto.data files in the /etc directory must be edited or created.

• When finished, the mounted directories on the remote machines can be accessed.

• The oper account should have full read/write permissions on all shared drives.


• auto.master - designates mount points on the local computer and the directory to mount on the remote server. • Example:

• /xks_data /etc/auto, data ~timeout=60

• auto.data - enables all servers to see the /export/data/xkeyscore directory on other machines and locate databases, archived, data, and MAILORDER directory. • Example:

• xksl -rw,soft,intr,tcp xks1:/export/data/xkeyscore • xks2-rw,soft,intr,tcp xks2:/export/data/xkeyscore



Directory Structure • /opt/xkeyscore/ - contains all of the

XKEYSCORE software. Software includes the GUI, processing, scripts, and configurations. • bashrc - XKEYSCORE environment variables

file. • beacon/ - contains the beacon perl script

(shm_beacon.pl) and a link to the beacon configuration file (shm_beacon.config).

• bin.shells/and bin.shells/sysadmin - contains miscellaneous bash, python, and C shell scripts.

• build/ - contains libraries and plug-ins. • install/ - contains installation scripts.

TOP SECRET // SI tl REL TO USA, AUS, CAN, GBR, NZL


Directory Structure • /opt/xkeyseore/eonfig/ - consists of sub-directories

and each contain configuration files for building and running XKEYSCORE. • crontabI - contains the master and slave crontab file. • dictionaries/ - contains the dictionary files for the filtering,

selection, TRAFFICTHIEF, CADENCE, fist tables, and any other local dictionaries.

• mise I - contains miscellaneous per-plug-in configuration files, (i.e. sotf_input_proc.xml ).

• plugins/- contains event handler configuration files for each of the plugins (default.xml).

• www/ - contains web configuration files and xscore.cfg. • SERVICE/ - contains the config files for all the services

needed by XKEYSCORE (httpd, php, mysqld, etc.)



Directory Structure

• /opt/xkeyscore/www/- contains the contents of the web front end. • docs I - contains documents viewable through the

XKS GUI. • html I - contains web pages and scripts that are

not on the secure server. • secured/ - contains web pages and scripts that

are on the secure server including: • cronsI - location of cron job scripts

• srcI - contains source code for the XKS GUI.



Directory Structure • /export/data/xkeyscore/ - is used for both internal

databases and metadata archive databases, input, output, and archiving of data. • archives/ - (optional) destination for processed content • inputs/ - (optional) used for file based input • mysql/ - location of the MySQL database consisting of

admin, insert, and query databases. • outputs/ - (optional) contain the following sub-directories:

• mailorder/ - pickup point • mailorder_working/ - file creation point before being moved to

mailorder/



Directory Structure

/xks_data/ - logical mount point for all other XKEYSCORE (including itself) /export/data/xkeyscore.

<hostname>/ - mount point for the hostname's local directory /export/data/xkeyscore (referenced by host name). • All servers must export their /export/data/xkeyscore

directory and mount this on the /<hostname> directory for each hostname of each machine, including itself.


u A u j ê I i i * ï j



Lesson Objectives V Accessing the GUI

/Exiting a Session V Main Menu Bar V MyXKS V Admin

/Computer Resources Option VStart and Stop Processing VRun a Process Manually

V Users V Search V Workflow Central / Results / Fingerprints



Accessing the GUI

In the address field of a web browser, type https://<master hostname or IP address>. PKI's or a UserlD and password are required. After successfully launching a new session, the XKEYSCORE WELCOME window appears.

Note: Compatible web browsers for XKEYSCORE version 1.5 are: • Internet Explorer is not supported • Firefox/3.0.* and above



Accessing the GUI

Njrijdlu'iFll:* X t i ts

9-lon» VP«

A r d .ci

'V LrH iuffftä'y Map K€a:

^ f l c r :

.CC D\D ^JTPAavty g jMy-npim-: ? ] M}' Ol I <f I E l My ^cocrt

cL 'JCCIÌDJ» PsCSXIXSS

j?r^Mj ixTin i h •äCcbbros

SDBHccbiràn

=1 Ccö 53dtfcr Cboh: g KetodUrf cl-ta E ^

Sj-nwv fcn CUM I M M

É^QjstvFrrUr Q€3JÌ5-5

[U l« ì <*e:cirt-: JCk-M I rv

èJfvm^-

¿Ijwirh'A'^c

Orr 1 r Oi • a i ¿111-%% 1

lork>J3ers Orr I r i ^ f c f»IAI ki-

C5 tor User 5.tvcv fltr

af- lWnM

l&. iom» X ir^.'.dir & iscrs ff wykl*« LertfEd . kwjts £ hngetprrs 0 i ta« : *

XKEYSCORE

uaoested achievements

Crtaled WüiWIaw UrarriHw

Hislugrari LBamHw

Exportln.q Mela data m m Hew

HUMAN RIGHTS ACT, USSID 18 ANDUSSID9

I tMJ queries require a justification lo Misure I li iman Rights ArJ (I IRA). UÌÌRID 1fl ¿nil IJSSID 9 cwripfarirt'. Ply«* vnlyr iriurnialkin as prompted ¿v the query inldrfaco. .An audit trai has been established and w i be «arched as pari oJ Menwih Hill Station's response to any complaint brought under I IRA and as part of Ihe U f i » 1H .*iH 9 pmrj»ft*

PImm iwtR that fifMRITM" TARfìrTWfì APPROVAL (STA) is reqiiiwl fw HRA brfure submitling any quoiy which includes terms specific lo 0 person or company (eg rwrnc. address, identity details such as communications address, passport/bank araiund raimhw) vAn ("ITl IFR (a) i* ifcfinwl *•> d UK. Brilbfi Oy[wridyrtl Tynilury (BDT) ur S*:ond Party "person" orfol is located in tli9 UK. or a 80T or Second Peaty country. STA is also required tor wildcard pulls that are ineviably gniiw) tn ftffriv* a ^liisJanJial pmpmtinn nf <uir.h enti** (ft g *ilHr-arrinrj nn a UK rily r.fwlft) Till li^al guidance W avaWMv fnjtn Ihe HRA Compliance Officerai Merwih Kill Station.



• The main menu bar across the top of the window has menus that, when selected, each has additional options available in a drop down menu form.

A Horoe XMyXK5 ^ Admin ¿Users Q^ Search 9 Workflow Central Results fi Fingerprints \z legging i : Reports | ¡ » t i c s Tasking 0 KelpT



Main Menu Options | OPTION DESCRIPTION |

Home Returns to the main page. MyXKS Can edit user settings, disable/enable access to

databases, edit a search form search setting, and restore default settings.

Admin Computer resources, Input Directories, Category Throttle, Search DBs, and DB Registration settings.

Users Contains User Accounts, Clearances, Privileges, Send Email, Users Online, My Auditees, My Audit Logs, and All Audit Logs.

Search Provides different search query forms, such as email addresses, category, full log, and user activity.

Workflow Central Request, modify, and view standing queries that will execute at a specified time or interval.

Results Can search personal searches by date time, query type, query name, output table, and user.

Fingerprints Fingerprint builder and viewer.

Map Brings up Google Earth

Help Help Documentation, XK Forum, Account Maintenance, and About XKEYSCORE


SUB-MENU OPTION

DESCRIPTION

Computer Resources Allows for process configuration and management.

Input Directories Contains the configuration for file-based input directories.

Category Throttle Edit CADENCE quota limits by category and/or fist table.

Search DBs Configuration for query databases which are queried when a search is submitted.

DB Registration Contains the mapping from insert databases to query database.

News Add, modify, delete mandatory and home page News.



n

The Proeessing->Computer Resources option from the ADMIN menu allows control of the entire daemon-styled, or continuously running, processes for XKEYSCORE. Processes appears in a table following the convention:

<PROC HOSTxPROGRAM NAME><PROGRAM ARGUMENTS> xkey01 process_data_parent



Computer Resources Computer Resource Window Process Table

E H ®

Hone X MyXKS Adnin ¿ U s e r s 0 , Sea-ch Workflow Central .Resul ts ft Fingerprints E l Statistics © M a p 0 Heip -

Navigat ion Fl ter 1 1

B t 3 Processing

3 Computer Resources

§E] Inpu: Direc:ories

Category Throttle

B € 3 Databases

[=3 Sear:h DBs

DB Registration

Ulilities

£ ] Casenotation Blacklist

£ ] Reload Corfig Files

¡13 New;

(m Ip Summary Table

(rp Oashlogger

(to Startup

Query Profiler

Help

Color Key STOPPED STOPPING STARTING RUNNING WON'T START UP

S T O P P E D ? A P P L A U N C H E R S T O P P E D

STOPPING? 1 STARTING? APP LAUNCHER STOPPED | APP LAUNCHER STOPPED

RUNNING? APP LAUNCHER STOPPED

WON'T START UP? ¿PP LAUNCHER STOPPED

C o m p u t e r Resources

0 Help Add Acticns » A p p L a i m c h e i is Runn ing

Ac.ions Proc Host Progran Maire Program Arguments Program PO Commended Status Sletus Detebme Started Deteti-ne Stopped

tlxksvrOI OUId 7607 FUN RUN 2012-12-0316:44:49.0 2012-12-0316:44:49.0

tlxksvrOI query_proc 30955 FUN RUN 2012-11 -2717:3357.0 2012-11 -2717:32:47.0 query_proc

w tlxksvrOI check_mailorder_sr.e .php 31019 FUN RUN 2012-11 -2717:3357.0 2012-11 -2717:32:47.0

© tlxksvrOI x k s — ^ Q ^ 5 ® ^ 31051 FUN RUN 2012-11 -2717:3357.0 2012-11 -2717:32:47.0

t ixksvroi ciickstreamservicesh 310>3 FUN RUN 2012-11 -2717:3357.0 2012-11 -2717:32:47.0

© tlxksvrOI query_dispatch 1311 FUN 2012-12-03 21:0809.0 2012-12-0321:08:07.0

© tlxksvrOI r'ileJnput_proc 31104 FUN RUM 2012-11 -2717:3357.0 2012-11 -2717:32:47.0

© tlxksvrOI xks_systemjnonitcr 13956 FUN RUN 2012-11 -27 21:01:13.0 2012-11 -27 21:01:13.0

• tlxksvrOI sotftocl 24server 31108 FUN RUN 2012-11 -2717:3357.0 2012-11 -2717:32:47.0

& tlxksvrOI lorTicat iSh 31124 FUN RUN 2012-11 -2717:3357.0 2012-11 -2717:32:47.0

Ö tlxksvrOI cadence jask ing j xoc -my fd i XYD - p d d g IE --digraph X5 31136 FUN RUN 2012-11 -2717:3357.0 2012-11 -2717:32:47.0

© tlxksvrOI :<ks_server_stats 31138 FUN RUN 2012-11 -2717:3357.0 2012-11 -2717:32:47.0

m tlxksvrOI mailorcer_proc -copydir /export/data/xkeyscorefout... 31140 FUN RUN 2012-11 -2717:3357.0 2012-11 -2717:32:47.0

€1 tlxksvrOI register j ne tada to job les -loglevel error 31143 FUN RUN 2012-11 -2717:3357.0 2012-11 -2717:33:50.0



•Wfi O f * iEfct*

The xks_app_launcher process runs on all servers from the inittab. It tells the computer which program to run by looking at its tasking host. /opt/xkeyscore/config/www/xscore.cfg

The config file specifying the location of the tasking database.

Processes can be stopped, started, edited, or deleted from the Computer Resources window.



• W f i o f * isfct«

Add a new process - click Add Edit a process - click Stop in the ACTION column, then click Edit. Delete process - click Stop in the ACTION column, then click Delete. Stop the App Launcher - disables the xks_app_launcher on every host.


> 1 0 O l , W I

1 0 0 1 I 0 0 ) t O O l • • • t a t

Trr -------

w

TOP SECRET II SI ti REL TO USA, AUS, CAN, GBR, NZL

Color Conventions ) 1 0 1 O I O O O I 0 0 1 I O I O I O

Visual cues in the form of colors are used to help identify activities performed by XKEYSCORE and serve as status indicators for monitoring purposes.

Red - indicates processes have been stopped Green - indicates processes are running Yellow - indicates processes are starting Orange - indicates processes are being stopped White - indicates processes won't start

Visual cues are also available in the COMMANDED STATUS and STATUS columns of the table.


It may be necessary to stop or start processes for troubleshooting or for a graceful server restart. Individual processes and programs - Click Stop in the ACTION column. To start it, click Run.

• To stop all individual programs, select ACTIONS->Start/Stop Resources. Enter the program name in PROGRAMS field, then click OK.

• Can use 'xks proc' actions and commands to do the same function

TOP SECRET // SI ft REL TO USA, AUS, CAN, GBR, NZL 3 9

) 10 01 1

1 0 0 1 I 0 0 1 1 0 0 1 Trr ------- TOP SECRET II SI ti REL TO USA, AUS, CAN, GBR, NZL

Resources-Start/Stop Processing I ^ ^ r

W

All Processing - select START/STOP Resources from the ACTIONS drop-down menu, leave the PROGRAMS and ON HOSTS fields to their defaults, click OK. Specifying programs or hosts - select STOP or START, enter a wildcard expression such as * or ! in the PROGRAMS or HOSTS field, and click OK.

Example: process *

Alternatively, in a terminal window can run: xks proc stop process*


> 10 01 101 TOP SECRET II SI tl REL TO USA, AUS, CAN, GBR, NZL

Run a Process Manually 0 1 O l O O O l - J

It may be necessary to run a process manually for troubleshooting purposes. To run a process manually: 1. Launch the GUI and log on as oper or admin. 2. Click ADMIN > Processing > Computer Resources 3. Click Stop in the ACTION column for the process. 4. Open a terminal window and ssh to the host running the

process, as the user ^oper'. 5. Type ps -ef | grep <process name> to verify

that the process is stopped. 6. Type <program namexprogram argument>

Example: query_proc <program arguments, if any>

--loglevel debug TOP SECRET II SI II REL TO USA, AUS, CAN, GBR, NZL


Home X MyXK5 Adrr iin ^^^^ I

* ¡A « a— ^ 1

| Navigation Filter f*Tj ¡f£|

1= 1 User Accounts £=] Clearances ™ -1 D a • 1*1 f i r» - i privileges 5z] Send Email [Sj Users Online

• This menu is only accessible to users with system administration privileges.

• An SA can add/modify user accounts, add groups, clearance levels, privileges, and email users from this menu.

TOP SECRET // SI it REL TO USA, AUS, CAN, GBR, NZL


From the main menu bar, click MyXKS to view your profile, accesses, privileges, auditors, settings, fingerprints, workflows, and recent results. Right click on any search form name to add a shortcut for that search form.

Navigation Filter x m Full Log DNI

HTTP Activity

I S My Fingerprints

Jz) My Workflows

0 My Recent Results

S Profile

r _S

Full Log DNI \

(S V J

HTTP Activity My Fingerprints My Workf lows My Recent Results

Profile


10 Ol 101 TOP SECRET fi SI // REL TO USA, AUS, CAN, GBR, NZL

Search Menu

From the main menu bar, click SEARCH. Menu options display in the vertical pane on the left. N a v i g a t i o n F i l t e r * * j s l

1 3 S e a r c h W i z a r d

3 0 ) A n o r r i y m i z e r

'iff? Tea O n i o n A d d r e s s S u r v e y

Tot R o o t e r s

Tor 5 o f t w a r e D o w n l o a d s

<rn Tor U s e r 5 u r v e y

>(rn W e b A n c c i y m i j e i c

S ^ J B o t n e t

«5? Q U A M T I J M e O T Table-

a 3 R O < 3 U E 5 T I T C M

• v * R O G U E S T I T C H

f i R ? S t o l e n C r c d s

Q B c n e i m A W E M A L H O U S E A c c e s s L o g s

A c t i v e Pass ive . M a p

O N E friage V 2

•if? C o n n e c t i o n ¡ I n f o r m a t i o n

C o t t o n c h a i n s a w

E Y o u M i l l 5 e r v e r L o g e

^ v E m a i l A d d r e s s e s

i r n E n d p o i n l ; A t t a c h e d M e d i a

urn E n t f p g i n t F i l e h a f h e - S

E n d p c n n i R e l a t e d 5 e i f K ? n i * r n E n d p o < n t S t t f s E ^ d p o i r * R-e -e raWe

<pr> E n d i x x n t E r t d p o i r * R e l a t e d C o u n t s

E x t r a c t e d F i les

E y o u Ma i l L o g s

F o u r t h P a r t y P a s s i v e M a p 1 Fu l l L o g D N I

i ™ H T T P R e q u e s t C o o k i e s

I E C o o k i e s

K a s p e r sky P5P

in? K e y L o g g e r s

in? M P TRHi r A c c e s s l o g s

t ^ M a c h i n e I n f o r m a t i o n

IP? M a c h i n c : I n f o r r n a t c o n

fcv N e t w o r k I n f o r m a t i o n

i r r t N c i w o s r k I n f o r m a t i o n

Cm N o r t o n PSP

fa? O d n o W a s s n i k i

in? O d n o k J a s s n i k i

(•«? P a s s w o r d s i n R D P

i - V R e g i s t r y

¿J <¿3 Re-gis i t ty

in? u s e r A s s i s t

i m i n K ^ m - s ^ a r .

in? Ro-Uter S u r v e y

in? S C A R F A O E N e t w a r k Br»fa*i¥.6fcian

IP? U t m o I D s

(n? W e b S e r v e r s

t m W i n d o w s U s e r 5 c c u r * y I d e n t i f i e r ( 5 I D )

im i vBuVe t i r t S e s s i o n s

a Q c r

in? A r c h i v e K n o w n P a s s w o r d

(en A t a b m a i l l o g s

in? E b e l l o g s

0*>? Emai l

in? F f u M l f n t

t r * M S 2 U l *« r A c t i v i t y

in? Ms2 ' K e y s a n d U s e r A c t i v i t y

Op-?r-o M i r * H c - a d c r s

in? P a r c e l Track ing!

t r r . R A R K n o w n P o s s w o r d

i r n u s e r A c t i v i t y a n d p e r s i s t e n t c v c r c o o k i c r s

i m Y e m e n m o b i l e I P s

in? Z a m e n s i s

Or» Zari ie-rvsi i - and e x t f - s c t e d f i l e s

l&J Z a r n e n s i s a n d F u l l - t e x t

trr« v Bu l le t an S e s t s o n c

TOP SECRET // SI ft REL TO USA, AUS, CAN, GBR, NZL 4 4

TOP SECRET lì SI t! REL TO USA, AUS, CAN, GBR, NZL

Search Menu When choosing a plugin type from the menu options, the only data searched is the data that was identified as a hit when the plugin was processed SUB-MENU

OPTION DESCRIPTION

Category DNI Searches dictionary category hits.

Full Log DNI Searches all sessions received by XKEYSCORE.

User Activity Enables a user to search by a user's activity. Example: a user can find a hotmail user's msnMailToken

TOP SECRET If SI // REL TO USA, AUS, CAN, GBR, NZL 4 5


All searches are conducted on database tables where the results of the XKEYSCORE engine are stored.

• Each row of a database table contains values from an individual session that was identified as a hit by XKEYSCORE when that plugin or microplugin processed the session.

• Each search type query is related to a plugin or microplugin, which performs the metadata extraction.



Search Menu - Details

Search details can be accessed from the Search status window by clicking Details. CURRENT SEARCH DETAILS window displays and allows the user to watch a query run through the appropriate databases. RESULTS link in the main menu bar can be used to display a list of all previous search results. Queries operate in parallel on each host.


TOP SECRET lì SI t! REL TO USA, AUS, CAN, GBR, NZL

Search Menu Details Window Details Window

Q u i Q u i IS» =

l='.=ii_ Hid© f= I ri I h~i •=• d S h o w Q u e r y

o ^•IvkAVI'fl ' l " JlT"»ITi.- >1 -y Al : w =z=l . t lxk© vrOI>: ciO.«riew

A l y . k w r 0 2. *=-• l i t i a r i • r y /• i f w ^ T l - x r l i - ^ v K l 11 -2 • i-=|l l / n n w .«tlxko v r 0 3 : c i s u m m s ^ r Y / n e w

A l.v.k-=, vr 04.L10 •=• i iL .t lxl<GvrU4: d o u m m c i r y / n o w . t l xksv rOS: c i O / n e w / l l - A - ' k . ^ . vi-nf«- j i -n i t i .-^»i -y Ai «—: w .«tlxliovrub-: c i u / n c w

vr O©. iri i«r y ^ i f v v / H v . - k . - ^ \ - - i r i 7 r - q f l Vf—. : w

-TlxlCGvrU / : d 3 u r r i m o r v ^ 3 c r i t A l . v . k - a - v r O © . «_- |0 / i i f w / l l - A - k . ^ . v i - n n - j i x « i - n « i i - y / r i i - : w . t l xksv rOS: c i O / n e w A l . v . k - a - v r 0 9i. l i t i a n • t i r y /• i f w STI- ' -•=- I ' "Ti I I ' I / f - i «—: w . l l x k s v r l O: dsurnnns i rY/ lnew

I = 1 Al.v.kwr "1 1 . "-•jO Vi 1 fypv

i = 1 . l l x l c o v r l 1 : dGurnrnc i ry / lncw . l l x k s v r l d0.1- iew

i 1 Al:.:k = vi-1 2:: •c | = LiiniTiiM-y/ne-w . l lx lCGvrl ci: q U / n c w

. t l x k s v r l 3:c i©ummarY/cJon© ciuerYlnci

= | /ll-z-'k-^vi-^l •"-J n /n : w = 1 .«t lxkovr l d o u m m c i r y / n o w

1 Al.v.k-a vr "1 •-TO VI 1 f w y H v . h - . - - . V I - 1 • - n .-=. I J I T » I T l - - l | - y ^ f i 8—• i =. I —• : m-J .«tlxks v r l &: q O / n © w

Al.v.k-Si-vr 1 t - _ •=-• ir i ir i i r y A.1 «__• i i•=- «-. i "_b»=t y i i i l j STI- ' i • -Hi / ' r^l I /f-1 : w , t l xk©vr l "7": deu rn rns i rY / r i ew

n - n n I 0 . 3 7 - 3 x . t l x k s v r l ©: d s u m m a r y / n e w A1 .v. k -=•* v - 1 •=•. LTO/I

"1 H- jiTiiT-i-ni- :i-y inr i

Al.v.kvvr 20 . LT -Liri in lirir y A i f w .«tlxko vrLJI : oiLJ/"n o-w-

= 1 .«tlxk©vr2 l : cisumms^rY.'Tnew

TOP SECRET If SI // REL TO USA, AUS, CAN, GBR, NZL 4 8


• From the main menu bar, click RESULTS to retrieve the results of previous queries.

• By changing the start and stop dates, queries performed between those dates can be viewed.

• If the query name is known, it can be entered in the QUERY_NAME field.

• If the USERID is known, it can be entered. • When complete, a window displays with the

matching queries.


T r r - - - - - - -| inn i im , . >,0,

o «o o. 101 icohttÄltt,. 0 o O l I K o l 1

• • < »

»•«10.0.1 . • . 1 J O I E l




Lesson Objectives J

V XKEYSCORE Process Data Flow / Processing Programs V Query Processes V Other Processes / Cronjobs

^crontab


TOP SECRET II SI ti REL TO USA, AUS, CAN, GBR, NZL R'ïîl^

Back-End Process Data Flow l U U L

I sotfinpurhâïïdl"

WC2 SCTF

sotf dist

System and other forma^fNe-inPut-P12i Scans dirs

for new files

MAILORDER xscoreproc

{process, data_parent)

<ks_meta_ingester

Insert Host db_inputjile_handler1

xs web db

Master Host

xscoreproc (query_proc)

Query Host • I QueryDBs

J



Processing Programs Processing programs are the main processes that extract meta the traffic and then database the information in insert databases.

I PROGRAM DESCRIPTION fileJnput_proc Scans for new input files, (before

processing moves the file to the .tmp directory of the input directory specified)

sotf_dist Listens for incoming SOTF sessions

process_data_parent Processes all new files discovered by file_input_proc or sotf_dist; optionally archives content and databases metadata. Parent process loads all dictionaries and starts up, then forks child processes which do the actual processing.


TOP SECRET USA, AUS, CAN, GBR,

process_data_parent 310« 1»

r

This process replaces proeess_dataO through process_dataX The "parent" process starts up and loads all the dictionaries, and then "forks" child processes which actually do the processing Parent acts similar to the xks_app_launcher, managing restarts for the children when they die When dictionaries are modified, parent reloads them and restarts the children "xks proc" will show an "X/Y" number next to process_data_parent • This is the number of children currently running, over the number that

should be running (based on the xks.config num_data_processors setting) • pdp will show up yellow anytime X != Y and green when everything is

running normally • This means when you first (re)start pdp, it will show yellow while it is

loading the dictionaries, because none of the actual child process_data's are running yet

"xks proc" will report extra or missing process_dataX with a PID of 0 • Can't tell what PID missing process_data is suppose to have, because its

managed by the parent now



Query Processes

• Query processes are processes that search and submit all necessary tables for the analysts queries.

PROGRAM DESCRIPTION query_dispatch Submits search jobs to search databases

and propagates the status of the search arid results back to the web server

query_proc Searches through all the necessary tables for the analysts queries.



Other Process

Other process which is run from the Application Launcher.

mailorder_proc - polls the /export/data/xkeyscore/outputs/mailorder_working directory by default. Then renames and moves mailorder files to /export/data/xkeyscore/outputs/mailorder.



Other Process

xks_meta_ingester - streams metadata over socket. This process improves database performance. Instead of each xscore proc writing to the database independently, they stream their metadata over socket to the metajngester, which combines it by plugin and writes to the database. • Reduces the number of connections to MySQL and

gives better control over table size.



Other Process

register_metadata_tables - moves tables from processing database of XKEYSCORE system to query database.

Works against the uberjndex table uberjndex->table_name, base_table_name, join_table Base table - contains common information amongt tables (full_log_xxx_xxxxx table) Extension table - extends the base table Registration process takes place in two phases: • Register all base tables • Register all extension tables that have had its base table

registered



Other Processes

signal_acquisitionJoopback - process that feeds modified packets back into the system.

Front-erid for packet recursion or any other process that feeds modified packets back into the system • Reinjects back to front-end - xfip • Process is completel independent



Other Processes

mpmrserver - this is the map-reduce server for microplugins, which runs the "Reducer" portion of GENESIS v5 microplugins. Runs outside the normal processing flow, and will not affect the rest of the system. It has a telnet port (5850) just like an xscore_proc.



jr-• correlation_server_0 - in memory

map-reduce server for correlation engine. • Each machine has one correlationserver, and

every process_data_parent connects to every correlationserver • xscoreproc - 8GB by default • uses port 4321



r • xks_eomms_server - a more efficient way to

communicate with hosts within and outside an XKS cluster (not currently implement) • Automatically handles configuration for talking

between slaves, master and overlord at site • Configuration is needed to connect to the "peer"

on the path towards, other sites • Comms configuration lives in

$XSCORE_DIR/config/comms/comms.config • Supports a "quality of service" which "fairly"

distributes available bandwidth to the services that are using comms


> 10 01 ! jfji If*] TOP SECRET II SI ft REL TO USA, AUS, CAN, GBR, NZL

Other Processes

x ks_eom ms_se rve r Allow and Peer rules have a "network" parameter which the comms systems uses to determine an "inside" and an "outside" in proxies. Comms system will only accept connections from address ranges it has been specifically configured to allow. Every between 2 comms servers connection should have: • "bandwidth_rule" on each side, name doesn't matter but

both rules should usually have same bandwidth cap • "allow" rule on one side with a reciprocal "peer" rules on

the other side TOP SECRET // SI I! REL TO USA, AUS, CAN, GBR, NZL 6 3


Other Processes

x ks_eom ms_se rve r Example: If we have a site named "US-123" connecting to xks-central over a 1 Mbps link, US-123's config would be: band width [wo rid] = 1Mbps peer[00] = address=xks-central.corp.nsa.ic.gov, port=2412, bandwidth=world, network=extemal

And xks-central would have:

bandwidth[us123] = 1Mbps allow[00] = address=xkey~master.us123, bandwidth=us123, network=internal


Other process which is run from Application Launcher.


• GUId - rescans content against fingerprints when a user clicks to view the content of a session.

• tomcat.sh - web server used to host XKS GUI

• sotftod124server - downloads sessions • Gets called from the GUId process • Works with any downloaded traffic that is SOTF

TOP SECRET // SI it REL TO USA, AUS, CAN, GBR, NZL

TOP SECRET II SI tl REL TO USA, AUS, CAN, GBR, NZL

• Other process which is run from Application Launcher.

• xks_server_stats - sends to xks_system_monitor on Master and generates stats about the server itself. • CPU usage, memory usage, disk space, disk I/O,

network traffic, etc. • Stats are fed to xks_system_monitor and the

system monitor does magic with them.



Statistic Processes

xks_system_monitor - collects stats messages from all over the system (front-end and back-end and the server itself) and summarizes them for forwarding. Optionally it can database stats locally.


XKEYSCORE uses a number of cron jobs to perform tasks.

CRONJOB DESCRIPTION

age_off_new.php Ages off metadata and content when the disk is near capacity, or when thresholds have been met.

xks update_dictionaries Pulls updates from various sources.

xks rsync push config Copies the /opt/xkeyscore/config directory to the slaves.

rwc_post_to_p u b. py Once an hour kicks off an update request


> 10 01

CRONTAB ! jfji If*] TOP SECRET II SI ft REL TO USA, AUS, CAN, GBR, NZL

Crontab is the program used to install, uninstall or list the tables used to drive the cron daemon.

The crontab consists of age_off_new.php xks update_dictionaries xks rsync push_config rwc_post_to_pu b. py


AB TOP SECRET II SI t! REL TO USA, AUS, CAN, GBR, NZL

age_off_ne w. p h p Options: • -debug : extra debug statements in the output • -info : extra info statements in the output • - taskdb : explicitly state that the machine is a task host • -web_db : explicitly state that the machine is a web host • -nosleep : use if you want to run now

This process ages off tables and archived data based on the settings in the xks.config file and the percentage of disk space used.


tFFMBmkytf R B

I • xks update_dietionaries I • This process pulls the necessarily files from I various sources to update the dictionary. I • Configure /opt/xkeyscore/config/xks.config I • #[dictionaries] I dictionaryfO] = type=royale, \ I src=sftp://tssi_fvey:tssi_fvey@ I xks-control/home/tssLfvey/xks_dict_update.tar.gz, \ I dest=update/xks_dict_update.tar.gz, \ I action[0]="cd I $XSCORE_DIR/config/dictionaries/update;$XSCORE_DIR/config/ I dictionaries/update/dup_install.pl > /dev/null 2>&1" I dictionary[1] = type=cadence

TOP SECRET II SI IIREL TO USA, AUS, CAN, GBR, NZL


• xks rsync push_config • Transfers Master configurations to its slaves. • Excludes dot files, "httpd/logs",

loadserver/packages", "httpd/log" • force: option to xks to force push config when not

on the master


l i M p

I • rwc_post_to_pub.py I • The automatic starProc process is as follows: I • Hour 1: master asks whoever (say xks-control) for an I update, gets the rpm, installs it, there is much rejoicing. I The slaves asks the master for the rpm at the same time I the master asks xks-control, but obviously the master I doesn't have it, so nothing happens. I • Hour 2: everyone asks for an update again, this time the I master has the rpm, the slaves download it and install I and there is much rejoicing. I • The rpm is installed and process data parent's are I restarted as soon as the rpm is downloaded on a given I machine.



u A u j ê I i i *ï j


TOP SECRET // SI fl REL TO USA, AUS, CAN, GBR, NZL

• • ¡ • H V What is a DeepDive? V Why DeepDive? / What does a DeepDive look like? / Front-End Processes / xFIP / Promoter

ve

TOP SECRET // SI ft REL TO USA, AUS, CAN, GBR, NZL

• XKEYSCORE packet processing solution • XKEYSCORE's software handles all packet processing • No upfront filtering prior to XKEYSCORE • XKEYSCORE "promoter" tries to promote richest/most

interesting traffic • All Strong Selectors • Full take ASDF (User Activity metadata) • Subset of GENESIS signatures

• List managed by XKEYSCORE team in concert with collection managers and site engineers

• 20% - 30% of site traffic is fully processed and can be found via XKEYSCORE search • Typically does not include unknown or uninteresting

protocols TOP SECRET // SI I! REL TO USA, AUS, CAN, GBR, NZL 7 6


Why DeepDive?

Access to most relevant DNI data supporting SigDev and collection missions. Enables new mission capabilities (e.g., Correlation) Session promotion can be synchronized and managed based on Genesis signatures, traditional tasking selectors and available resources

• Provides better scaling • Drop unwanted data. Keep the rest and make decisions

later and more accurately Better control of the processing space

• Instantiate new mission capabilities and dataflows quickly • Troubleshooting and monitoring made easier

Need access to "raw" packets to support new mission (e.g., Cyber, Bulk Crypt) • Sessions can be displayed as Packet Bundles like Wireshark



ve r

What does a DEEPDIVE look like? XKEYSCORE full-take session processor (Back End) High speed packet ingest: an end-to-end solution Intelligent filtering to vary the proportion of traffic retained

DEEPDIVE V

Front End

Mett lesome Packet

Splatter Promoter I Defrag

Back End

Dictionary Scanner

m

Plugins

Dictionary Scanner I Microplugins

L 1 Fingerprints W^^ÊKÊÊk—WÊLJÊKÊÊÊÊ^^M

Metadata

Content

Packets |\ Partial Sessions I i>

Full Sessions

t>



XKEYSCORE Front-End Processes

W h a t i t ' s c a l l e d W h a t i t d o e s W h a t i t m e a n s

Packet Splatter Ingests packets (from files, from the network, from a capture card) in a variety of formats.

If it's a packet stream, It can probably be fed into a DEEPDIVE.

xFip Fast reassembly of TCP/IPv4, UDP/IPv4 streams*, and TCP/IPv6 and UDP/IPv6 streams*.

DEEPDIVE sessionizes everything Mb 1 1 LESOME Reassembly of streams from less

common protocol stacks. before making a keep/drop decision.

Promoter Rule-based filtering of reassembled sessions, based on keyword, country code or appid/fingerprint.

DEEPDIVE intelligently chooses the most useful traffic for retention.

Defrag Fully rebuilds sessions** Enough content available to do full decoding/document descent at the Back End


• Packet bundles • Preserves original packets and packet order • Preserves information that is lost during sessionization • Original pcap available in the XKS Viewer

- Packet API • Microplugins can iterate over raw packets • Microplugins can use information that is lost during

sessionization • E.g. timestamps, flags, checksums

• Packet fingerprints • Fired based on observations xFip has made

• E.g. large sequence gaps, TTL variation



Filters sessions prior to back end processing • keywords, regex, country code, appids* • SIGDEV: promotion rather than strong selection

Set the focus of the back end • traffic types of interest • regions of interest • legal/policy constraints

Set the width of the access aperture • promote 20% of 20 signals? • promote 100% of 4 signals?

Set the length of data retention • promote 20% and keep for 3 days? • promote 30% and keep for 2 days?

allow appid chat.*

allow country__code PK

block country_code US-IIS


/ Usage ^options

V General Commands V Services / Actions </ Options


• Usage: xks [options] <command> • Try 'xks help <name>' to get help on a specific

service or action


• General commands: • services - list available services • actions - list available actions • dependencies [invert] - show service dependencies • help [items] - print help on services or actions

• Services (specify one or more service names or 'all'): • start <services> - start the specified services • stop <services> - stop the specified services • restart <services> - restart the specified services • status <services> - print the status of the specified services • setup <services> - setup/configure/fix the current xks install


TOP SECRET II SI tl REL TO USA, AUS, CAN, GBR, NZL I I ^ B

Actions: • accounts_report

• add_admin • change_db_password

• cluster • compile_genesis • disk_check • ext4_format

• ext4_upgrade contents

• fetch • force_register • info • install_slave • local_tagging

- sends an email containing accounts usage to the specified users

- sets up a local Linux user to administer XKS - changes the XKS database user's password and

updates all references to it - cluster actions - compiles GENESIS signatures - get raid and disk status -format $XSCORE_DATA_DIR partition and convert

to ext4 filesystem - convert to ext4 filesystem while preserving

of $XSCORE_DATA_DIR (no formatting) - fetch a remote file - force metadata table registration - show cluster information

- install a slave machine in this cluster - checks and/or loads tagging file


TOP SECRET II SI tl REL TO USA, AUS, CAN, GBR, NZL I I ^ B

Actions: • monitor • mpmr_register • mysqls • onall • powertower • proc • query • query_dispatch

• rac • reload_dictionaries • rsync • search_fields • show_config • switch • sync_accounts

• tail • tasking_dump

- view XKS monitoring messages via activemq - force mpmr table registration - run a mysql script - run a command on all machines in this cluster - configure or run a powertower command - control XKS processes on this cluster - display query status or submit a query - command line interface to the XKS

query_dispatcher(s). - access remote admin ports -force running processes to reload dictionaries - push configs or files to slaves - populates user settings with search fields - show values from xksconfig for specified keys - query or rebalance data switch - synchronize user accounts (except for

classifications) - view realtime xks logs - print out the contents of the xkTasking and

xksTasking_voip databases. TOP SECRET // SI fl REL TO USA, AUS, CAN, GBR, NZL 86

) 10 01 , w

1 0 0 1 I 0 0 1 t o o •


Actions: • top • update • update • users

dictionaries gui_help

• version • watchdog • workflow

- display system performance - update all XKS dictionaries - update the 'help' pull downs in GUI - display the users currently logged into the GUI - show XKS version information - check and (re) start essential XKS processes. - manually submit a workflow


• Options: • -verbose : print extra information to the screen • -debug : used for debugging script problems



) io o; I JO ICC •


» t N O T \V General Commands w I V c U U U

0 1 > 0 1 0 1 0

Type: xks help services • This will list all available services:

first - initialization service that runs before all others virus_scanner- sets up virus scanner, assuming tarballs are present. ftpd - enables ftp on the master if mailorder is enabled distcc - sets up distributed compiler service s!ash_proc - setup optimal /proc parameters myricom - handles installation and configuration 10GigE network cards home - sets up the home directory for the xks user account gcc - check there is a working compiler on the system upgrade - updates configuration files when upgrading to a new version of xks bashrc - sets up bash environment variables beacon - sets up xks monitoring beacon based on xks.config t t - checks connectivity to TRAFFICTHIEF server

TOP SECRET II SI II REL TO USA, AUS, CAN, GBR, NZL



x>t \Y*f1 \V General Commands . u


sendmail - configures sendmail for use with xks role_files - this service installs role-specific files issue - sets up the DoD mandatory login warnings roya!e_with_cheese - setups automatic updates ntpd - configure ntp based on xks.config link_summary - sets up xks link summary GUI nfsd - sets up xks-specific nfs mounts server_certs - sets up server certificates for SSL applications openoffice - installs and configures OpenOffice for use in the xks GUI init_d - sets up the xks init.d services resolver - sets up resolver config php - sets up PHP related stuff. Except php.ini httpd - sets up xks-specific httpd configuration

TOP SECRET tl SI It REL TO USA, AUS, CAN, GBR, NZL 9 0



x>t \Y*f1 \V General Commands . u


www - sets up GUI configuration files volp - sets up voip processing crond - ensures xks can use cron and sets up xks cron jobs sshd - configures the secure shell service for use with xks license - checks for a valid license file and if one isn't found prints a message syslog - configures the syslog service for use with xks

• all xks processes log to /var/log/xks.log dictionaries - checks status of any configured dictionaries cluster_check - checks network connectivity across the cluster autofs - start, stop, restart automounts loadserver- start, stop, and setup loadserver directories - sets up directories used for xks auditd - no help available

TOP SECRET II SI II REL TO USA, AUS, CAN, GBR, NZL

• Type: xks help services • This will list all available services:

• Idap - no help available • mysqld - sets up the mysql server for use with xks • disks - checks status of disk partition used by xks • databases - maintains database scheme consistency • Ioeal_tasking - reapplies local tasking if necessary • workflows - sets up xks default workflows • category_throttle - overrides default category throttle settings based

on overrides specified in xks.config • enrichment_tomcat- sets up enrichment tomcat java application

server • plugin_setup - populate plugin database tables from xml files, appy

default plugin config specified in xks.config, apply overrides from xks.config, regenerate plugin config files from database

• crdb - no help available • tomcat - sets up tomcat java application server • clickstream - sets up clickstream service



• f i l e jnpu t - sets up directories and database entries needed for file-based input

• age_off_db - synchronizes the database (xs_task_db.age_off) with xks.config's settings for content and metadata. The values in the database will be unconditionally overwritten with those found in xks.config

• db_connectivity - verifies connectivity to critical databases • pd f - se ts up xpdf language packs • ul_age_off- sets the maximum data retention time to a little over an

hour in UL mode. • mDNSResponder - sets up mDNSResponder for use with SOTF

input

• appjauncher- controls the xks app launcher, which is responsible for monitoring xks processes and starting/stopping them as commanded from the GUI

• processes_setup- configures xks processes based on specifications in xks.config

• comms - sets up the XKS communications system configuration



• endace - handles all the installation and configuration for Endace Dag packet capture cards

• last - cleanup service that runs after all others



start • xks start mysql

stop • xks stop httpd

restart • xks restart nfs

status • xks status autofs

setup • xks setup plugins


> 1 0 0 1 • — ' • -1 0 0 1 I 0 0 1 i O O t i t / T I n

I A i 1

xks - Actions > 10 01 ! jfji If*] TOP SECRET II SI ft REL TO USA, AUS, CAN, GBR, NZL

I M .1)

• xks onall Éps -ef | grep xscore | grep - v grep' xks force_register [operQtlxksvrOl run]5 xks force_register

Forced update on xks_meta_ingester

xks rsync push_config -force • Usage: xks rsync <options>

[push_config|push_compìled|push <dest>

xks update_dictionaries • Usage: xks update dictionaries

[test|check|print|force|help] xks versioia

slaves|push] <src>

[oper@tlxksvr01 run]S xks versici! 1 . 5 . 9 - 6 5

xks info

[ o p e r O t l K k s v r O l r u n ] S x k s i n f o

s i L e

S I G A D

P D D G

X K S v e i r s i o n

M a s t e r

N U J I L B L A V E S

I n p u t

C o n f i c i

T i m b e t l i n e — S V

U S F - 7 9 0

I E

1 . 5 . 9 - 6 5

t l x k s v r O l

1 3

f i l e r s o t f

m a n a q e d s a r t ci


TOP SECRET II SI tl

xks query servers

REL TO USA, AUS, CAN, GBR, NZL

[ o p e r @ t l x k s v r O H t r u n ] 5 x k s c j u i e r y s e r v e r s

tlKksvr02 : c[0 3a 14s 98n 5 4w 2 0 12-12-0 5 16:07:22 (top)

tlxksvr03 : c£0 230a Os On Ow 2 O12-12-O 5 17 : 55: 02

tlxksvrO 4 : <qO 23 0a Os On Ow 20 12-12-0 5 17:55:02 (top)

tlxksvrOS : • 225.a Os 4n Ow 2 012-12-0 5 17:55:02 (top)

tlxksvr07 : «qO 230a Os O i-i Ow 2 O 12— 12— O 5 17:55:02

tlxksvrOS : c£ O 225a Os 5n Ow 2 0 12-12-05 17:55:02 (top)

tlxksvrO 9 : c£ O Oia: Os 17 3 n 2w 2 012-12-0 5 17:55:02 (top)

tlxksvrlO : c£0 23Ua Us O i-i Ow 2 O 12 — 12 — U 5 17 : 5 5: 02

tlxksvr 11 : c£ O 2 3 O -a 0 s= O ri Ow 2 O 12 - 12 - 0 5 17 : 5 5 : O 2 ft o p )

tlxksvr 12 : c£ O 2 2 5a Os 4 n Ow 20 12-12-0 5 17:55:02 (top)

a=awaiting dispatch, s = se nt, n=new, w=wo rking t i me s t amp s Snows earliest submitted but ui n tin! s 11 e cl query current time: 2012-12-•5 18:02:09


TOP SECRET //

xks proc

SI I! REL TO USA, AUS, CAN, GBR, NZL

[oper@tlxksvrQl run]$ xks proc GUI GUId qp query_proc eli clickstreamService.sh rrnt register_metadata_tables eras check mailorder site.php s2d sotftod124server csOO correlation server 0 sab signal acquisition base ctp cadence tasking proc sal signal_acquisition_loopback enr enrichment-tomcat.sh sd sotf dist file f i l e _ i n p u t _ p r o c sst strong selector targeting mp mailorder_proc torn tomcat.sh mpmr mpmr_server xcs xks_comm s _s e r v e r pd# process_dat.a# xmi xks_meta_ingester p d p p r o ce s s _dat a_p are nt Ksrn xks_system_mo nit o r qd query_dispatch X S S x k s _s e r v e r _s t at s Run T xks proc fullT to show full listing

tlKksvrOl GUI eli cms ctp enr file mp qd qp rmt s2d sab: 4/4 sst torn KCS xmi xsm xss tlxksvr02 csOO mpmr pdp 4/4 qp rmt sab: 4/4 sal 6 / 6 sd xcs xmi xss tlxksvr03 csOO mpmr pdp 4/4 qp rmt sab: 4/4 sal 6 / 6 sd xcs xmi xss tlxksvrQ 4 csOO mpmr pdp 4/4 qp rmt sab: 4/4 sal 6 / 6 sd xcs xmi xss tlxksvrQ5 csOO mpmr pdp 4/4 qp rmt sab: 4/4 sal 6 / 6 sd xcs xmi xss tlKksvrO6 csOO mpmr pdp 4/4 qp rmt sab: 4/4 sal 6 / 6 sd xcs xmi xss tlxksvrQ7 csOO mpmr pdp 4/4 qp rmt sab: 4/4 sal 6 / 6 sd xcs xmi xss tlKksvrO8 csOO mpmr pdp 4/4 qp rmt sab: 4/4 sal 6 / 6 sd xcs xmi xss tlKksvrO9 csOO mpmr pdp 4/4 qp rmt sab: 4/4 sal 6 / 6 sd xcs xmi xss tlKksvrlO cs 0 U mpmr pdp 4/4 qp rmt sab: 4/4 sal 6/6 sd xcs xmi xss tlKksvrll cs 0 0 mpmr pdp 4/4 qp rmt sab: 4/4 sal 6/6 sd xcs xmi xss tlxksvrl2 csOO mpmr pdp 4/4 qp rmt sab: 4/4 sal 6 / 6 sd xcs xmi xss tlxksvr'13 csOO mpmr pdp 4/4 qp rrnt sab: 4/4 sal 6 / 6 sd xcs xmi xss tlxksvr'14 csOO mpmr pdp 4/4 qp rrnt sab: 4/4 sal 6 / 6 sd xcs xmi xss Process consistency check OK on all hosts



xks Example xks proc full

[ o p e r @ t 1 x k s v r 0 1 r u n ] $ x k s p r o c f u l l

a p p l a u n c h e r s t a t u s : R U M ( p i d 3 0 7 0 5 )

i d h o s t n a m e p r o g r a m a r g u m e n t s c o m m a n d e d a c t u a l p i d

1 4 t l x k s v r O l c a d e n c e _ t a s k i n g _ p r o c — m y f d i X Y D - - p d d g I E - - d i g . . . R U M R U N 3 1 1 3 6

4 t l x k s v r O l c h e c k _ m a i l o r d e r _ s i t - e . p h p R U N R U M 3 1 0 1 9

7 2 3 t l x k s v r O l c l i c k s t r e a m S e r v i c e . s h R U M R U M 3 1 0 5 3

6 5 4 t l x k s v r O l e n r i c h m e n t - t o m c a t . s h R U M R U M 3 1 2 0 0

9 t l x k s v r O l f i l e i n p u t p r o c R U M R U M 3 1 1 0 4

1 t l x k s v r O l G U I d R U M R U M 9 3 3 5

5 4 8 t l x k s v r O l m a i l o r d e r p r o c - - e o p y d i r / e x p o r t / d a t a / x k e y . . . R U M R U M 3 1 1 4 0

8 t l x k s v r O l q u e r y d i s p a t c h R U M R U M 1 7 5 4 9

3 t l x k s v r O l q u e r y _ p r o c R U M R U M 3 0 9 6 5

1 9 3 t l x k s v r O l r e g i s t e r m e t a d a t a t a b l e s - - l o g l e v e l e r r o r R U M R U M 3 1 1 4 3

7 0 9 t l x k s v r O l s i g n a l _ a c q u i s i t i o n _ b a s e - f g e n e r i c _ p a c k e t _ t o _ b u n d l e . . . R U M R U M : 4 / 4 3 1 2 3 6

1 2 t l x k s v r O l s o t f t o d l 2 4 s e r v e r R U M R U M 3 1 1 0 8

4 6 1 t l x k s v r O l s t r o n g _ s e l e c t o r _ t a r g e t i n g R U M R U M 3 1 1 4 5

1 3 t l x k s v r O l t o m c a t . s h R U M R U M 3 1 1 2 4

6 5 3 t l x k s v r O l x k s c o m m s s e r v e r R U M R U M 3 1 1 4 8

5 t l x k s v r O l x k s _ m e t a _ i n g e s t e r R U M R U M 3 1 0 5 1

4 6 2 t l x k s v r O l x k s s e r v e r s t a t s R U M R U M 3 1 1 3 8

1 1 t l x k s v r O l x k s _ s y s t e m _ m o n i t o r R U M R U M 1 3 9 8 6

7 2 4 t l x k s v r 0 2 c o r r e l a t i o r i _ s e r v e r _ 0 - - l o g l e v e l d e b u g R U M R U M 1 3 9 4 6

3 1 t l x k s v r 0 2 m p m r s e r v e r R U M R U M 7 1 5 0

7 3 0 t l x k s v r 0 2 p r o c e s s _ d a t a _ p a r e n t - - m a x - m e m 2 0 R U M R U M : 4 / 4 2 2 6 6 1

3 0 t l x k s v r 0 2 q u e r y p r o c R U M R U M 1 4 7 5 4

1 8 7 t l x k s v r 0 2 r e g i s t e r m e t a d a t a t a b l e s - - l o g l e v e l e r r o r R U M R U M 1 4 9 9 4

7 1 0 t l x k s v r 0 2 s i g n a l _ a c q u i s i t i o n _ b a s e - f g e n e r i c _ p a c k e t _ t o _ b u n d l e . . . R U M R U M : 4 / 4 1 4 9 9 6

4 1 t l x k s v r 0 2 s i g n a l a c q u i s i t i o n l o o p b a c k - f p a c k e t a u x . c o n f i g - i l o o . . . R U M R U M : 6 / 6 1 4 9 9 0

2 5 9 t l x k s v r 0 2 s o t f _ d i s t . R U M R U M 1 4 8 4 5

6 7 9 t l x k s v r 0 2 x k s c o m m s s e r v e r R U M R U M 1 4 9 9 2

3 8 t l x k s v r 0 2 x k s _ m e t a _ i n g e s t e r R U M R U M 1 4 9 7 0

4 7 3 t l x k s v r 0 2 x k s _ s e r v e r _ s t a t s R U M R U M 1 4 9 8 8


• xks query


[operStlxksvrOl run]$ xks query

id user type search start search stop duration status 66250201 http parser 00:00 12/2/12 00:00 12/6/12 00: 00:: 10 ongoing 66250183 full log 00: 00 12/3/12 00:00 12/6/12 00:00:17 ongoing 66250155 fullJLog 00: 00 12/4/12 00:00 12/6/12 00: 00:: 40 ongoing 66250127 geo_info 00: 00 11/30/12 00:00 12/6/12 00:01:16 ongoing 66250052 email addresses 00: 00 11/21/12 00:00 12/6/12 00:03:36 ongoing 66249873 full log 22: 00 12/3/12 21:59 12/4/12 00:11:31 ongoing 66249660 fullJLog 00: 00 12/2/12 00:00 12/6/12 00:18:: 57 ongoing 66244233 category 00: 00 11/5/12 00:00 12/6/12 00:42:17 ongoing 66244135 fullJLog 00: 00 11/28/12 00:00 12/6/12 00:44:30 ongoing 66244009 http_parser 00: 00 11/5/12 00:00 12/6/12 00:48:49 ongoing 66243967 http parser 00: 00 11/5/12 00:00 12/6/12 00:49:34 ongoing 66243855 document metadata 00: 00 11/21/12 00:00 12/6/12 00:50:40 ongoing 66243785 correlation 00: 00 11/5/12 00:00 12/6/12 00:52:46 ongoing 66243463 correlation 00: 00 11/21/12 00:00 12/6/12 00:56:13 ongoing 66243071 email addresses 00: 00 11/1/12 00:00 12/6/12 01:08:24 ongoing 66242973 user_activity_exif 00: 00 11/21/12 00:00 12/6/12 01:12:03 ongoing 66242413 http parser 00: 00 11/28/12 00:00 12/6/12 01:26:32 ongoing 66242315 fullJLog 00: 00 12/4/12 00:00 12/6/12 01:30:52 ongoing

There are 18 queries in progress

TOP SECRET // SI I! REL TO USA, AUS, CAN, GBR, NZL 1 0 0


xks Example xks query detail

[ o p s c @ t l x k = v r 0 1 r u n ] * x k s q u e r y d e i a i l i d = 6 6 2 5 1 C f 5

Q u e r y S u m m a r y

U s e r i d :

T y p e : e m a i l _ a d d r e s = e s

S e a r c h i n g f r o m C O : C O 1 1 / 3 C / 1 2 t o 0 0 : 0 0 1 2 / 6 / 1 2

D u r a t i o n : 0 0 : 0 1 : 2 9

P r i o r i t y : 5

C s n c e i : X ( o )

H a x R e s u l t s : 1 0 C 0 0

M a x T i m e : 6 0 D O

Q u e r y S Q L

N a m e : s m = t e v 3 _ 4

C l a s s i f i c a t i o n : S , I S / S I , N £ A N C F O R N , I S , H C S , S / S I , M U S C U L A R , R E L J J S A f N S A J » Q F O R N ^ S I , C , R

W h e r e : WHERE d a t e t i m e > = T 2 0 1 2 - 1 1 - 3 C 0 0 : 0 0 : 0 0 T A N D d s t e t i n e < = ' 2 0 1 2 - 1 2 - 0 6 0 0 : 0 0 : 0 0 r A N D e r a a i l - T A N D d o m a i n = T h o t m a i l , c o n T

Q u e r y S t a t u s

h o s t

t l x k s v t O l

t l x k s v r 0 2

t l x k s v r O S

t l x k s v r 0 4

t l x k s v r O S

t l x k s v r 0 6

t l x k s v r 0 7

t l x k s v r 0 8

t l x k s v r O S

t l x k s v e i l

t l x k s v r l Z

t l x k s v r l 3

t l x k s v r l 4

t l x k s v r l O

d a t a b a s e

q 0

q 0

q 0

q 0

q 0

q 0

q 0

q 0

q 0

q 0

q 0

q 0

q 0

q 0

s t a t u s

f i n i s h e d

o n g o i n g

f i n i s h e d

f i n i s h e d

f i n i s h e d

f i n i s h e d

f i n i s h e d

f i n i s h e d

f i n i s h e d

f i n i s h e d

f i n i s h e d

f i n i s h e d

f i n i s h e d

f i n i s h e d

TOP SECRET // SI fl REL TO USA, AUS, CAN, GBR, NZL 1 0 1


Lesson Objectives

/ Exécutables Vmysqls Vonall Vxks onall Vxks monitor /sotf_stat Vxks top

V1 Web Status V Additional Monitoring



Exécutables

System monitoring can be performed from the command line using the following executable commands:

mysqls onall xks onall sotf_stat xks top



The mysqls bash shell script can be used to execute MySQL statements from in the /opt/xkeyscore/bin. shells/sysadmiri/mysqls directory. The most commonly used options in mysqls are: • status - displays file-based input statistics. • speed - displays the total file based input

processing rate (Mbps) • speedl - displays file-based input processing rate

(Mbps) per input source. • speed2 - displays file-based input processing rate

(Mbps) per xkeyscore processing server. • count - displays the count of input files in the

new, working, error, and done states. TOP SECRET // SI t! REL TO USA, AUS, CAN, GBR, NZL 1 0 5

STOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL

— • — - — — • —

• mysqls status

[operfltlxksvrOl run]S mysqls status

status count (*) sim(filesize) priority

bit rate Jibps

NULL



• xks onall 'xks mysql status' [ o pe rr @t 1 xks v if U 1 run]5 xks onall Txks mysql status1"

I d o you want to e^ec ut e- "xks my s q 1 s t at us " o ra all'? [ y | ra ] y

, tlxksvrOl

status mysqld my s ql-cì i s running

, tlxksvr02 - — - —

st atus mysqld mys q1 <d i s cun ning

~ ~ tlxks vr03 - — - —

status mysqld my s ql dl i s trmrarairacf

—, tlsíksvr G 4

status mysqld my s ql -di i s trmrarairacr

^ tlxksvr G S

s t at u s my s q 1 ci mysqld i s trmrarairaçr


• This script will monitor your front-end processes.

• Type: xks montor or xks monitor h to receive the help menu


-XKEYSCORE-Henu Command c

d b m, h

Name config dataflowall dat aflow_be menu packetsplatte r process _d a t a quit servers sotfinput xf i p

Description Co n f i g u re t hi s u t il i ty F r o nt E n d D a t a f 1 o w M e n u BackEnd Dataflow Menu View this menu Packet Acquisition [Front End] Process Data [Back End] Quit/Exit Server Stats (CPU,IO,etc.) SOTF Input [Back End] Txks top5

Sessionization [Front End] replacement


TOP SECRET // SI f! REL TO USA, AUS, CAN, GBR, NZL K *JlL>

Type: xks montor f to receive xfip stats • K K E Y S C O R B - S e s s i o n i z a t i o i i ( S e r v e t v i e w P r e s s ' 4 ' ] C a s e n o t a r i o n R ò c c e ( M b p s ) L c » c ) i d i i i < ( ^ T C P Q u a l i r y t 7 D H 1 1 5 1 Ö l ^ ô û û û û Û . Û Û D . Û Û 0 . 0 0 7 D H 1 1 5 2 D 9 0 4 O O O O O.OO O.OO D.OO

R a r e ( P k . r ) O . 0 0 o . DO

C o u n t ( P k c ) O O

P u n t t¡. O. oo o . oo

0 . OO o . oo

- T ^ . t A l : D . O O [ K E Y : S f M l , S Ï M 4 / S T M 1 G ,



• The sotf_stat command is used to display the SOTF (streaming object transfer format) input statistics for an entire cluster.

• The statistics include total number of process_data's running on the cluster, session input rate (sessions/sec), total bytes input (Mbps), and total bytes output to process_data(s) (Mbps).



To execute the sotf_stat script: Log on to the server and open a terminal window. Type sotf_stat because the command is in the path Type s to toggle the summary statistics view from total statistics to individual host statistics. Type q to quit the program


• The sotf_stat script lists the hostname, number of process_data's currently running, Mbps, number of sessions, and number of bvtes.

—XKEYSCORE SOTF Stat ;istics— Hostname #In #OA/#OC Mbps In Sess In Bytes In Sess Q MaxElk mhxkssvr02 7 4/4 18. 66 41092014 7 4818112974976 0 0 mhxkssvr03 3 4/4 16. 15 410121822 4783549865004 0 0 mhxkssvi:04 3 4/4 16. 65 410444622 4781320276992 0 0 mhxkssvrOS 3 4/4 15. 79 409831857 4759939303920 0 1

I •—PRC: 15/ 15 Rate: 64.52 Mbps Sessions: 1641358767 Bytes: 19143289212772


• The xks top script lists the hostname, Mbps sotf rate, number of process_data's running, the % of CPU, and % of 10 wait.

hostname sotf fprocs cpul lowaitl mhKkssvrOl -0,00 0 0,53 0.02 mhxlcssvi:02 21,08 12,88 7,94 mhxkssvr03 13,55 4 13.35 7.28 bihxlcssvi:04 14,97 4 14.50 8.63 rahxkssvrOS 14,13 4 17.14 8.01 TOTAL 63,74 16 11.68 6.38



Additional Monitoring

[oper@mhxkssvr02 xks tail Dec 5 18:27:29 mhxkssvrQ2 registerjrietadata_tables [13877] : <register st (automatic?) repair failed Dec 5 18:27:29 mhxkssvrQ2 register jnetadata_tables[13877) : <register st (automatic?) repair failed Dec 5 18:27:29 mhxkssvrQ2 registerjmetadata_tables I[13877 j : <register st (automatic?) repair failed Dec 5 18:27:59 mhxkssvr02 registerjnetadata_tables[13877j : <register st (automatic?) repair failed Dec 5 18:27:59 mhxkssvrQ2 register jne tadata_tables I[13877 j : < register st (automatic?) repair failed Dec 5 18:27:59 mhxkssvr02 register jnetadataJ:ables[13877 ] ;: <register st (automatic?) repair failed Dec 5 18:28:08 mhxkssvr02 sotfjlist[13986] : <sot£_dist_t> NOTICE: cu Dec 5 18:28:29 mhxkssvr02 regis ter jne tada,ta_tables (13877 J : cregister st (automatic?) repair failed Dec 5 18:28:29 rnhxkssvr02 register_metadata_tables(13877j : cregister st (automatic?) repair failed Dec 5 18:28:29 rnhxkssvr02 registerjmetadatactables(13877j : cregister st (automatic?) repair failed



Lesson Objectives

V Common Troubleshooting techniques VFull Disk ^/Sotf Problems

^Processing Problems ^Outputs VQuery Problems ^Directory Permissions


/var/log/xks.log (xks tail) - Relevant error messages can be viewed in this file. This directory may fill the disk, some known reasons are: • process_data has lost its connection with the

sotf_dist and is continuously trying to reconnect to sotf_dist.

• nfs error may have occurred and a detailed message can be found in the file /var/log/messages.

• Corrupt tables in the insert database. • Check to make sure the age_off_new.php cronjob

aged off old metadata and content. TOP SECRET // SI tl REL TO USA, AUS, CAN, GBR, NZL 1 1 7

• /export/data/xkeyscore/inputs • If there are too many files in the directory:

• file_iriput_proc may be running improperly or not at all. Verify that file_input_proc is running from the command line type:

• ps -e f | grep file | grep - v grep • xks proc

• The file_input_proc may need to be restarted.

• No new files in the directory: • The directory may not be cross-mounted properly,

if automounting is used.



Disk Full continued... /export/data/xkeyscore/mysql/iO or i1 • If /export/data/xkeyscore/mysql/iO or ¡1 are filling and

qO and/or q1 maintains its size, register_metadata_tables may not be working properly. • Restart process and watch the databases to see if it is

transferring files or run the process by hand to troubleshoot further.

• If /export/data/xkeyscore/mysql/qO or q1 is filling, the age_off_new.php script may be running improperly or not at all. • First run the command: ps -ef | grep age_

• If script isn't running, try running it by hand. • If script is running, then stop script and try running it by hand to

see if there are any errors. TOP SECRET // SI I! REL TO USA, AUS, CAN, GBR, NZL 1 1 9


SOTF Problems Can an sotf_input_proc run with a file_based file_input_proc? • Yes. Both input types can run on XKEYSCORE given that

each are independently configured correctly. Can file-based input be disabled so that only sotfjnput is processed? • If moving from file-based input to sotfjnput, and no

additional file-based input is expected, the plug-in for file-based input, dbjnput_file_handler, should be disabled.

• From the TERMINAL WINDOW: • Stop all the processes : xks stop all • Change /opt/xkeyscore/config/xks.config to set f i lejnput to 'no'. • Setup the config : xks setup plug ins, xks setup processes • Rsync change to slaves : xks rsync push_config • Restart process_data's : xks proc restart pdp



SOTF Problems continued... Is XKEYSCORE receiving input?

To verify whether XKEYSCORE is receiving input, run the sotf_stat command to get the current input statistics. If no connection is visible, from the command line: 1. Type telnet localhost 5042 2. Output statistics for the specified sotf dist 3. If running, type ps -ef | grep sotf_dist 4. Determine if sotf_dist's are listening on the

specified port: Type telnet localhost 5040 If command is refused, the sotf_dist is not listening on the port. Continue with step 5.

5. Type netstat -a | grep 5040 If a connection is established for this port then most likely the sotf_dist is listening on this port.



SOTF Problems continued...

netstat will tell if... sotf_dist is listening for connections If connections have been made to the sotf_dist If we are "backing up"- i.e., if sotf dist is running but has no process_dataJs connected to it, it won't be able to send data anywhere, so eventually its network receive queue will get large. • Ideally, the receive queue should always be 0.



SOTF Problems continued...

Is the proeess_datajparent running? At least one process_data must be running and synchronized with the sotf_dist for it to receive input. • If problems continue, run the sotf_dist in a terminal to

further troubleshoot and identify error messages.



D124 file troubleshooting Symptom: A lot of errors or too many errors display when performing the command 'mysqls status':

1. First try, mysqls cleanup, in a terminal window.

2. Type mysqls status 3. Typemysqi xs task db; to log into MySQL

database and use the xs_task_db database 4. Execute the following command: delete

from tar_files where status="error,/; 5. Exit out of the MySQL database 6. Type mysqls status

There error files. 1 2 4

• The heart of the XKEYSCORE processing engine is the xscore_proc with related plugins.

• Input to the xscore proc is either file-based and from an file_input_proc, or streaming from an sotf_input_proc.

• After processing, the written metadata to the insert databases can be sent to a follow on system for additional processing.


• How many process_data's should be running on a host? . From the XKEYSCORE GUI:

• Click ADMIN > Processing > Computer Resources • Determine how many process_data's are configured to

be running on the specified host.


y running on a host?

Log onto the XKEYSCORE server and open a terminal window. Type ps -ef I grep xscore grep -v managed

[3fp t J á r - T ü > ] S ps -ef 1 çrep s s r r - 1 ra -v rad

m • /Hy u . ^ v . H E : 'E, V i 1 f î ' r ' ^

-J - v * v J • V 1 J v 1 1 Ksc:rej)roc —parent —base_pDr 1555(1 -external appid rampile --db ii] -reriice level -5 -coiarid db forre next -mmi 20 -• •löflevel errer -restart childre

r -- : : : l c f ü 1 .390 - • r l t ì S ® - ± J ± U C • • s d ì --chiidoié a n i • jpSC u . . v u ¿¿L'C 1 2 ó is¡:e CO : OS: K s c z e j M elìild-l —port 5551 —external appid compilé --db ill -reriice level -5 -coiand db forre next -mmi 20 -• • M e v e ì pfccr

m u . j x j "VCl i 6 « 1 ÙU l;.v 'j . IU.UL'.JI : œ r e j c o c ehild-2 -port. 5552 -external appid compilé --db iO -Ferace level -5 -coiarid db ta next -mm 20 -• •löflevel errer

opsr ¿ " I D i ¿L'I; 1 U I S l S ; CO:OS:24 TO6JCQC elìild-3 --port. 5553 -external appid compile --db iO -reriice level -5 -coiand db forre next - » r a 20 -• • I 0 5 M errer

opsr .• •• ¡y à U V J

1 U lS : i ! : (0:05:11 OTejrac chili-i -port 5550 -external appid compile --db iO -reriice level -5 -coiarid db forre next -mnm 20 -•lopevel error


Processing Problems continued..Nm*™ TOP SECRET II SI tl REL TO USA, AUS, CAN, GBR, NZL

xks_app_launcher is running, but not starting processes specified in the Computer Resources window?

This may indicate that the xks_app_launcher is defunct. Use the kill command to kill the appjauncher and its related sub-processes: • Typepkill -f app_

• If a PID is not being specified, use the pkill command. The - f option kills all of the sub-processes.

• Type ps to look for the new xks_app_lauricher process.



If, after performing the procedures, the xks_app_launcher is still not starting applications: • In a terminal window, manually run the problem process

to see if there are any error messages. • The xks_app_launcher on any host is dependent on the

access of the xs_task_db.proc_resources database table on the master. Verify that the specified host can access the master's database and /opt directory.

• On the slave system type mysqi xs task db -h <masterhostname>

• performs a remote MySQL server login



To test the xscoreproc, type: telnet <process host> <port number> Optional commands to assist trouble shooting are sbr - prints the processing rate for the single

xscore-proc. - displays dictionary hit statistics. - displays statistics on the internal plug-in

processing rates. help - there are many commands and can be

described in the help menu.

sh


• If the process_data_parent continues to deny access through the command port, and input still has not started processing, check the input source.

• Run the process in a terminal window with the argument -loglevel debug, to view debug messages.

• The command port also provides processing rates and statistics for troubleshooting performance issues, outages, and general administration issues.



Outputs - Mailorder

/ex po rt/data/x keysco re/o u tp u ts/ma i I o rde r If there are no new files in the MAILORDER directory, MAILORDER may not be working properly. Possible causes are that:

Files are being written to the wrong directory or it is not configured properly Permissions on the MAILORDER directory will not allow MAILORDER to move files



Query dispatch is the process that submits search jobs to search databases and propagates the status of the search and the results of the search back to the web server. After submitting a new query, Search Status window displays a summary listing query name, date and time submitted, number of databases complete, and number of results.



Query Problems

The query never moves to the finished state. If a database outage or a comms outage occurs, results will not be reported from the single system However, results from all other databases will return properly with the query results, but they will not appear in this state.



Query Problems

• Query job status is stuck in a wa i t i n g_d i s ba tc h. • If a status appears stuck in this state, the

query_dispatch may not be running on the web server. To determine whether it is running: • Type ps -ef | grep query_

• If the process is not running, restart it from the XKEYSCORE GUI or troubleshoot the xks_app_launcher.



Query Problems

• Another cause of this scenario is that a query database may have hung up the query dispatch process. Check the progress of queries on the query database hosts by viewing the table sdb_queryJobs in the query database, which tracks the status of queries: • Type mysql qO • Type select status,count (*) from sdb_query_jobs where group by status;

• The select statement displays the current state of the queries on the query host. If many more queries appear in the new state when compared to other query databases, begin troubleshooting the problem query_proc on the specified query database.



Query Processing

• The query is in the sent state, but never appears in new. • After the query_dispatch process disbatches the

query, the status is moved to sent. A query moves to the new state when the query has been placed in the query processing queue on the queryJiost.

• If a query does not move to the new state in a reasonable amount of time, the connectivity of the database should be tested.



Query Processing • To check the progress of queries on the query

database hosts, view the table sdb_queryJobs in the query database, which tracks the status of queries: • Type mysql qO • Type select status, count (*) from sdb_query_jobs where group by status;

• The select statement displays the current state of the queries on the query host. If many more queries appear in the new state when compared to other query databases, begin troubleshooting the problem query_proc on the specified query database.



Query Processing

• The query appears in the new state, but never finishes. • query is in the new state, has been received by

the query host and placed in a queue waiting to be processed.

• Queries can become backlogged with a large number of queries waiting in the new state, though the query_proc is processing the queries properly. It is hard to predict the time to work off a query backlog, but using the following select statement the status of queries for the current day can be checked for processing trends.



Query Processing

• To display the number queries in each state for the current day: • Type

s e l e c t s t a t u s , c o u n t ( * ) , d a t e t i m e _ s u b m i t t e d , {UNIX_TIMESTAMP(now() ) -U N I X _ T I M E S T A M P ( d a t e t i m e _ s u b m i t t e d ) ) / 3 6 0 0 f r o m s d b _ q u e r y _ j o b s where ( d a t e t i m e _ s u b m i t t e d > ( n o w ( ) - INTERVAL ' 1 ' DAY}) g r o u p b y s t a t u s ;

• To display the number of queries processed per hour for the current day: • Type

s e l e c t s t a t u s , c o u n t ( * ) / 2 4 AS q u e r i e s _ p e r _ h o u r f r o m s d b _ q u e r y _ j o b s where c a n c e l ! = " C " and ( d a t e t i m e ~ s u b m i t t e d > ( n o w ( ) - INTERVAL ' 1 ' DAY)) AS B a c k l o g g r o u p by c a n c e l ;

• If processing properly, queries can take hours, if not days, to complete based on the backlog and the processing trends.


• Queries complete but there are no results. • If queries complete, but no results are visible,

verify that the date range of the query coincides with the collection date of the data. If using test data, test the query system by putting the start date range at a year or two older to assure it is not old test data.

• Verify that query metadata is in the query database by checking the contents of the /export/data/xkeyscore/mysql/{query_db}/ directory.


• Queries complete and metadata returns, but there is no content. • The metadata in the XKEYSCORE viewer

displays the host and directory path of the content file. Verify the content file exists using the Is -I command. Trace a dataflow issue if the file does not exists. If the content file exists, confirm the httpd daemon is started on all slave systems. To confirm the httpd daemon:

• 1. Type su - oper • 2. Type xks status httpd • 3. If the daemon is not on, type xks start httpd



Query Results

To troubleshoot problems with metadata or content from a query, it will be necessary to retrieve the actual content, since recreating the problem is very difficult. This can be accomplished from the XKEYSCORE GUI. Click RESULTS and begin a search of the questionable queries.


top secret ii si tl rel to usa, aus, can, gbr, nzl

Documents