ail framework for analysis of information leaks · to help users of ail framework, a document is...
TRANSCRIPT
![Page 1: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/1.jpg)
AIL Framework for Analysis of Information LeaksPractical and Efficient Data-Mining of Suspicious Websites, Forums and Tor
Hidden-Services
Alexandre [email protected]
Aurelien [email protected]
Jean-Louis [email protected]
April 1, 2021
![Page 2: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/2.jpg)
Links
• AIL project https://github.com/ail-project
• AIL frameworkhttps://github.com/ail-project/ail-framework
• Training materialshttps://github.com/ail-project/ail-training
• Online chat https://gitter.im/ail-project/community
2 of 88
![Page 3: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/3.jpg)
Privacy, AIL and GDPR (PII)
• Many modules in AIL can process personal data and even specialcategories of data as defined in GDPR (Art. 9).
• The data controller is often the operator of the AIL framework(limited to the organisation) and has to define legal grounds forprocessing personal data.
• To help users of AIL framework, a document is available whichdescribe points of AIL in regards to the regulation1.
1https:
//www.circl.lu/assets/files/information-leaks-analysis-and-gdpr.pdf3 of 88
![Page 4: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/4.jpg)
Potential legal grounds
• Consent of the data subject is in many cases not feasible inpractice and often impossible or illogical to obtain (Art. 6(1)(a)).
• Legal obligation (Art. 6(1)(c)) - This legal ground applies mostlyto CSIRTs, in accordance with the powers and responsibilities setout in CSIRTs mandate and with their constituency, as they mayhave the legal obligation to collect, analyse and share informationleaks without having a prior consent of the data subject.
• Art. 6(1)(f) - Legitimate interest - Recital 49 explicitly refers toCSIRTs’ right to process personal data provided that they have alegitimate interest but not colliding with fundamental rights andfreedoms of data subject.
4 of 88
![Page 5: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/5.jpg)
Objectives
5 of 88
![Page 6: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/6.jpg)
Our objectives
• Show how to use and extend an open source tool to monitor webpages, pastes, forums and hidden services
• Explain challenges and the design of the AIL open sourceframework
• Learn how to create new modules
• Learn how to use, install and start AIL
• Supporting investigation using the AIL framework
6 of 88
![Page 7: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/7.jpg)
AIL Framework
7 of 88
![Page 8: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/8.jpg)
From a requirement to a solution: AIL Framework
History:
• AIL initially started as an internship project (2014) toevaluate the feasibility to automate the analysis of(un)structured information to find leaks.
• In 2019, AIL framework is an open source software inPython. The software is actively used (and maintained) byCIRCL and many organisations.
• In 2020, AIL framework is now a complete project called ailproject2.
2https://github.com/ail-project/8 of 88
![Page 9: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/9.jpg)
AIL Framework: A framework for Analysis ofInformation Leaks
”AIL is a modular framework to analyse potential informationleaks from unstructured data sources.”
Other leaks
9 of 88
![Page 10: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/10.jpg)
Capabilities Overview
10 of 88
![Page 11: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/11.jpg)
Common usage
• Check if mail/password/other sensitive information (termstracked) leaked
• Detect reconnaissance of your infrastructure
• Search for leaks inside an archive
• Monitor and crawl websites
11 of 88
![Page 12: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/12.jpg)
Support CERT and Law Enforcement activities
• Proactive investigation: leaks detection◦ List of emails and passwords◦ Leaked database◦ AWS Keys◦ Credit-cards◦ PGP private keys◦ Certificate private keys
• Feed Passive DNS or any passive collection system
• CVE and PoC of vulnerabilities most used by attackers
12 of 88
![Page 13: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/13.jpg)
Support CERT and Law Enforcement activities
• Website monitoring◦ monitor booters◦ Detect encoded exploits (WebShell, malware encoded in Base64, ...)◦ SQL injections
• Automatic and manual submission to threat sharing and incidentresponse platforms◦ MISP◦ TheHive
• Term/Regex/YARA monitoring for local companies/government
13 of 88
![Page 14: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/14.jpg)
Sources of leaks
14 of 88
![Page 15: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/15.jpg)
Mistakes from users:
15 of 88
![Page 16: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/16.jpg)
Sources of leaks: Paste monitoring
• Example: https://gist.github.com/
◦ Easily storing and sharing text online◦ Used by programmers and legitimate users→ Source code & information about configurations
• Abused by attackers to store:◦ List of vulnerable/compromised sites◦ Software vulnerabilities (e.g. exploits)◦ Database dumps→ User data→ Credentials→ Credit card details
◦ More and more ...
16 of 88
![Page 17: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/17.jpg)
Sources of leaks: Paste monitoring
• Example: https://gist.github.com/
◦ Easily storing and sharing text online◦ Used by programmers and legitimate users→ Source code & information about configurations
• Abused by attackers to store:◦ List of vulnerable/compromised sites◦ Software vulnerabilities (e.g. exploits)◦ Database dumps→ User data→ Credentials→ Credit card details
◦ More and more ...
16 of 88
![Page 18: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/18.jpg)
Examples of pastes (items)
![Page 19: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/19.jpg)
Why so many leaks?
• Economical interests (e.g. Adversaries promoting services)
• Ransom model (e.g. To publicly pressure the victims)
• Political motives (e.g. Adversaries showing off)
• Collaboration (e.g. Criminals need to collaborate)
• Operational infrastructure (e.g. malware exfiltrating information ona pastie website)
• Mistakes and errors
18 of 88
![Page 20: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/20.jpg)
Are leaks frequent?
Yes!and we have to deal with this as a CSIRT.
• Contacting companies or organisations who did specificaccidental leaks
• Discussing with media about specific case of leaks and how tomake it more practical/factual for everyone
• Evaluating the economical market for cyber criminals (e.g. DDoSbooters3 or reselling personal information - reality versus mediacoverage)
• Analysing collateral effects of malware, software vulnerabilities orexfiltration
→ And it’s important to detect them automatically.3https://github.com/D4-project/19 of 88
![Page 21: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/21.jpg)
Paste monitoring at CIRCL: Statistics
• Monitored paste sites: 27◦ gist.github.com◦ ideone.com◦ ...
2016 2017 08.2018Collected pastes 18,565,124 19,145,300 11,591,987Incidents 244 266 208
Table: Pastes collected and incident4 raised by CIRCL
4http://www.circl.lu/pub/tr-4620 of 88
![Page 22: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/22.jpg)
Current capabilities
21 of 88
![Page 23: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/23.jpg)
AIL Framework: Current capabilities
• Extending AIL to add a new analysis module can be done in 50lines of Python
• The framework supports multi-processors/cores by default.Any analysis module can be started multiple times to supportfaster processing during peak times or bulk import
• Multiple concurrent data input
• Tor Crawler (handle cookies authentication)
22 of 88
![Page 24: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/24.jpg)
AIL Framework: Current features
• Extracting credit cards numbers, credentials, phone numbers,...
• Extracting and validating potential hostnames
• Keeps track of duplicates
• Submission to threat sharing and incident response platform(MISP and TheHive)
• Full-text indexer to index unstructured information
• Tagging for classification and searches
• Terms, sets, regex and YARA tracking and occurences
• Archives, files and raw submission from the UI
• PGP, Cryptocurrency, Decoded (Base64, ...) and usernameCorrelation
• And many more23 of 88
![Page 25: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/25.jpg)
Terms Tracker
• Search and monitor specific keywords/patterns◦ Automatic Tagging◦ Email Notifications
• Track Term◦ ddos
• Track Set◦ booter,ddos,stresser;2
• Track Regex◦ circl\.lu
• YARA rules◦ https://github.com/ail-project/ail-yara-rules
24 of 88
![Page 26: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/26.jpg)
Terms Tracker:
25 of 88
![Page 27: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/27.jpg)
YARA Tracker:
26 of 88
![Page 28: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/28.jpg)
Terms Tracker - Practical part
• Create and test your own tracker
27 of 88
![Page 29: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/29.jpg)
Recon and intelligence gathering tools
• Attacker also share informations
• Recon tools detected: 94◦ sqlmap◦ dnscan◦ whois◦ msfconsole (metasploit)◦ dnmap◦ nmap◦ ...
28 of 88
![Page 30: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/30.jpg)
Recon and intelligence gathering tools
29 of 88
![Page 31: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/31.jpg)
Decoder
• Search for encoded strings◦ Base64◦ Hexadecimal◦ Binary
• Guess Mime-type
• Correlate paste with decoded items
30 of 88
![Page 32: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/32.jpg)
Decoder:
31 of 88
![Page 33: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/33.jpg)
Crawler
• Crawlers are used to navigate on regular website as well as .onionaddresses (via automatic extraction of urls or manual submission)
• Splash (”scriptable” browser) is rending the pages (includingjavascript) and produce screenshots (HAR archive too)
Docker container
Splash
...Docker container
Splash
AIL-framework
32 of 88
![Page 34: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/34.jpg)
Crawler
How a domain is crawled by default
1. Fetch the first url2. Render javascript (webkit browser)3. Extract all urls4. Filter url: keep all url of this domain5. crawl next url (max depth = 1)
33 of 88
![Page 35: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/35.jpg)
Crawler: Cookiejar
Use your cookies to login and bypass captcha
34 of 88
![Page 36: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/36.jpg)
Crawler: Cookiejar
35 of 88
![Page 37: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/37.jpg)
Crawler: DDoS Booter
36 of 88
![Page 38: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/38.jpg)
Correlations and relationship
37 of 88
![Page 39: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/39.jpg)
Live demo!
38 of 88
![Page 40: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/40.jpg)
Example: Dashboard
39 of 88
![Page 41: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/41.jpg)
Example: Text search
40 of 88
![Page 42: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/42.jpg)
Example: Items Metadata (1)
41 of 88
![Page 43: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/43.jpg)
Example: Items Metadata (2)
42 of 88
![Page 44: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/44.jpg)
Example: Items Metadata (3)
43 of 88
![Page 45: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/45.jpg)
Example: Browsing content
44 of 88
![Page 46: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/46.jpg)
Example: Browsing content
45 of 88
![Page 47: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/47.jpg)
Example: Search by tags
46 of 88
![Page 48: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/48.jpg)
MISP
47 of 88
![Page 49: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/49.jpg)
MISP Taxonomies
• Tagging is a simple way to attach a classification to an event oranattribute.
• Classification must be globally used to be efficient.
• Provide a set of already defined classifications modeling estimativelanguage
• Taxonomies are implemented in a simple JSON format 5.
• Can be easily cherry-picked or extended
5https://github.com/MISP/misp-taxonomies48 of 88
![Page 50: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/50.jpg)
Taxonomies useful in AIL
• infoleak: Information classified as being potential leak.
• estimative-language: Describe quality and credibility ofunderlying sources, data, and methodologies.
• admiralty-scale: Rank the reliability of a source and the credibilityof an information
• fpf6: Evaluate the degree of identifiability of personal data and thetypes of pseudonymous data, de-identified data and anonymousdata.
6Future of Privacy Forum49 of 88
![Page 51: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/51.jpg)
Taxonomies useful in AIL
• tor: Describe Tor network infrastructure.
• dark-web: Criminal motivation on the dark web.
• copine-scale7: Categorise the severity of images of child sex abuse.
7Combating Paedophile Information Networks in Europe50 of 88
![Page 52: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/52.jpg)
threat sharing and incident response platforms
−→
Goal: submission to threat sharing and incident response platforms.
51 of 88
![Page 53: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/53.jpg)
threat sharing and incident response platforms
−→
1. Use infoleak taxonomy8
2. Add your own tags
3. Export AIL objects to MISP core format
4. Download it or Create a MISP Event9
8https://www.misp-project.org/taxonomies.html9https://www.misp-standard.org/rfc/misp-standard-core.txt
52 of 88
![Page 54: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/54.jpg)
MISP Export
53 of 88
![Page 55: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/55.jpg)
MISP Export
54 of 88
![Page 56: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/56.jpg)
MISP Export
55 of 88
![Page 57: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/57.jpg)
Automatic submission on tags
56 of 88
![Page 58: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/58.jpg)
API
57 of 88
![Page 59: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/59.jpg)
AIL exposes a ReST API which can be used to interact with theback-end10.
1 curl https ://127.0.0.1:7000/ api/v1/get/item/default
2 --header "Authorization:
iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj "
3 -H "Content -Type: application/json"
4 --data @input.json -X POST
5
• AIL API is currently covering 60% of the functionality of back-end.
10https:
//github.com/ail-project/ail-framework/blob/master/doc/README.md58 of 88
![Page 60: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/60.jpg)
Setting up the framework
59 of 88
![Page 61: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/61.jpg)
Setting up AIL-Framework from source or virtualmachine
Setting up AIL-Framework from source
1 git clone
https://github.com/ail-project/ail-framework.git
2 cd AIL-framework
3 ./installing_deps.sh
60 of 88
![Page 62: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/62.jpg)
Feeding the framework
61 of 88
![Page 63: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/63.jpg)
Feeding AIL
There are different way to feed AIL with data:
1. Setup pystemon and use the custom feeder◦ pystemon will collect items for you
2. Use the new JSON Feeder (twitter)
3. Feed your own data using the API or the import dir.py script
4. Feed your own file/text using the UI (Submit section)
62 of 88
![Page 64: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/64.jpg)
Via the UI (1)
63 of 88
![Page 65: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/65.jpg)
Via the UI (2)
64 of 88
![Page 66: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/66.jpg)
Feeding AIL with your own data - API
api/v1/import/item
1 {
2 "type": "text",
3 "tags": [
4 "infoleak:analyst-detection=\"private-key\""
5 ],
6 "text": "text to import"
7 }
65 of 88
![Page 67: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/67.jpg)
Feeding AIL with Twitter posts and associated urls
• AIL - feeder from Twitter11
• The AIL-feeder-twitter search in Twitter using Twint (withoutAPI), crawls the urls and pushes the results in AIL
• The JSON format format can be extended via meta fields
11https://github.com/ail-project/ail-feeder-twitter66 of 88
![Page 68: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/68.jpg)
Feeding AIL with your own data - import dir.py (1)
/!\ requirements:
• Each file to be fed must be of a reasonable size:◦ ∼ 3 Mb / file is already large◦ This is because some modules are doing regex matching◦ If you want to feed a large file, better split it in multiple ones
67 of 88
![Page 69: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/69.jpg)
Feeding AIL with your own data - import dir.py (2)
1. Check your local configuration configs/core.cfg◦ In the file configs/core.cfg,◦ Add 127.0.0.1:5556 in ZMQ Global◦ (should already be set by default)
2. Launch import dir.py with de directory you want to import◦ import dir.py -d dir path
68 of 88
![Page 70: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/70.jpg)
Starting the framework
69 of 88
![Page 71: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/71.jpg)
Running your own instance from source
Accessing the environment and starting AIL
1
2 # Launch the system and the web interface
3 cd bin/
4 ./LAUNCH -l
70 of 88
![Page 72: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/72.jpg)
Running your own instance using the virtual machine
Login and passwords:
1 # Web interface (default network settings)
2 https://127.0.0.1:7000/
3 # Web interface:
5 Password1234
6 # SSH:
7 ail
8 Password1234
71 of 88
![Page 73: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/73.jpg)
Updating AIL
Launch the updater:
1 cd bin/
2 # git pull and launch all updates:
3 ./LAUNCH -u
4
5
6 # PS:
7 # The Updater is launched by default each time
8 # you start the framework with
9 # ./LAUNCH -l
72 of 88
![Page 74: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/74.jpg)
AIL ecosystem - Challenges and design
73 of 88
![Page 75: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/75.jpg)
AIL ecosystem: Technologies used
Programming language: Full python3
Databases: Redis and ARDB
Server: Flask
Data message passing: ZMQ, Redis list and RedisPublisher/Subscriber
74 of 88
![Page 76: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/76.jpg)
AIL global architecture: Data streaming betweenmodule
75 of 88
![Page 77: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/77.jpg)
AIL global architecture: Data streaming betweenmodule (Credential example)
76 of 88
![Page 78: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/78.jpg)
Message consuming
Modulex
Redis set
Moduley Moduley
SPOP SPOP
SADD
→ No message lost nor double processing
→ Multiprocessing!
77 of 88
![Page 79: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/79.jpg)
Creating new features
78 of 88
![Page 80: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/80.jpg)
Developing new features: Plug-in a module in thesystem
Choose where to put your module in the data flow:
Then, modify bin/package/modules.cfg accordingly79 of 88
![Page 81: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/81.jpg)
Writing your own modules - /bin/template.py
1 import time
2 from pubsublogger import publisher
3 from Helper import Process
4 if __name__ == ’__main__ ’:
5 # logger setup
6 publisher.port = 6380
7 publisher.channel = ’Script ’
8 # Section name in configs/core.cfg
9 config_section = ’<section name >’
10 # Setup the I/O queues
11 p = Process(config_section)
12 # Endless loop getting messages from the input queue
13 while True:
14 # Get one message from the input queue
15 message = p.get_from_set ()
16 if message is None:
17 publisher.debug("{} queue is empty , waiting".format(config_section))
18 time.sleep (1)
19 continue
20 # Do something with the message from the queue
21 something_has_been_done = do_something(message)
22
80 of 88
![Page 82: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/82.jpg)
Contribution rules
81 of 88
![Page 83: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/83.jpg)
How to contribute
82 of 88
![Page 84: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/84.jpg)
Glimpse of contributed features
• Docker
• Ansible
• Email alerting
• SQL injection detection
• Phone number detection
83 of 88
![Page 85: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/85.jpg)
How to contribute
• Feel free to fork the code, play with it, make some patches or addadditional analysis modules.
• Feel free to make a pull request for your contribution
• That’s it!
84 of 88
![Page 86: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/86.jpg)
How to contribute
• Feel free to fork the code, play with it, make some patches or addadditional analysis modules.
• Feel free to make a pull request for your contribution
• That’s it!
84 of 88
![Page 87: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/87.jpg)
How to contribute
• Feel free to fork the code, play with it, make some patches or addadditional analysis modules.
• Feel free to make a pull request for your contribution
• That’s it!
84 of 88
![Page 88: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/88.jpg)
Final words
• Building AIL helped us to find additional leaks which cannot befound using manual analysis and improve the time to detectduplicate/recycled leaks.
→ Therefore quicker response time to assist and/or informproactively affected constituents.
85 of 88
![Page 89: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/89.jpg)
Ongoing developments
• New JSON feeders
• Python API wrapper
• Data retention (export/import)
• MISP modules expansion
• auto Classify content by set of terms◦ CE contents◦ DDOS booters◦ ...
• Crawled items◦ duplicate crawled domains◦ tor indexer
86 of 88
![Page 90: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/90.jpg)
Annexes
87 of 88
![Page 91: AIL Framework for Analysis of Information Leaks · To help users of AIL framework, a document is available which describe points of AIL in regards to the regulation1. ... Automatic](https://reader036.vdocuments.mx/reader036/viewer/2022071605/6141d9862035ff3bc7624a81/html5/thumbnails/91.jpg)
Managing AIL: Old fashion way
Access the script screen
1 screen -r Script
Table: GNU screen shortcuts
Shortcut Action
C-a d detach screen
C-a c Create new window
C-a n next window screen
C-a p previous window screen
88 of 88