attacking iot developers - foss backstage
TRANSCRIPT
![Page 2: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/2.jpg)
About me
PhD in computational physics
Doing Opensource for over 25 years:Math libs for: Minix/gcc 68k, Linux libm.so.5,
ported perl and python to psion/epoc, contributed to flightgear port to Windows/
Mac, Maintainer of msktutil
PMC of Apache Bigtop, ASF Member
Backend Software Architect Bosch eBike
![Page 3: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/3.jpg)
Attacking a Big Data IoT Developer
Dr. Olaf Flebbeof ät oflebbe.de
ApacheCon Bigdata Europe
![Page 4: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/4.jpg)
Aftermath: Attacking Big Data Developer
• codehaus.org now hosted by Apache
![Page 5: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/5.jpg)
Security
• The Internet is not a safe place any more• Attackers are using increasingly complex attacks in order to penetrate
enterprises• There is no well established awareness for:
![Page 6: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/6.jpg)
Developer Attack Vector
• Any user of an insecure build process may download software artifacts which may penetrate himself or his customer• Investigate• Upstream fixes• Watch community to iron things out
![Page 7: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/7.jpg)
Method 1: Network analysis
• Catching complete network traffic while compiling codebase• Create in depth package analysis of the traffic with an sophisticated network
security monitor• ...• Profit
![Page 8: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/8.jpg)
Toolset
![Page 9: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/9.jpg)
Docker
• Clean Room, network separation• Apache Maven images from dockerhub library• apt-get update && apt-get install tcpdump• tcpdump -i eth0 -s 0 -w /FILE &• mvn (-DskipTests) package• stop tcpdump
• docker cp container:/FILE .
![Page 10: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/10.jpg)
eth0
tcpdump -i eth0
![Page 11: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/11.jpg)
Bro
• Bro: The Network Security Monitor• www.bro.org• Flexible, High performance, Stateful in depth Analysis• Analyse HTTP, HTTPS Certificate Chains, Fingerprinting of Downloads,
Analyse DNS Requests and Answers• blacktop/bro docker image• docker run --rm -v $(pwd):/pcap blacktop/bro -C -r FILE
![Page 12: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/12.jpg)
Watch out for:
• plain http traffic• SSL Servers• DNS queries• Unidentified TCP packages
![Page 13: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/13.jpg)
Repo issues
![Page 14: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/14.jpg)
#1 eclipse
![Page 15: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/15.jpg)
repo.eclipse.org
• Reported it on Feb 18th to [email protected]: No reaction.• Aggravated that maven central will only accept TLS 1.2 in 3 days
(15th of June, see sonatype blog)• Asked Ralph Müller (German eclipse Representative) for comment:• Eclipse is already working on it #515595, new reverse proxy soon in place for
repo.eclipse.org
![Page 16: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/16.jpg)
#2 maven.java.net
• According to ssllabsmaven.java.net usesone of the oldsymantec certificate.Mozilla and Googledistrusting them in Sep.because of repeated issues of symantec certificate practice.
![Page 17: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/17.jpg)
Will java follow google?
• Will java/openjdk/oracle follow Google and mozilla?Yes: java.maven.net will be broken.
• Still time to fix...• I bet it will break openjdk-7/8 on Debian, if they
dont renew.• Asking for comment on Jun 5th: No answer so far.
![Page 18: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/18.jpg)
Go(bot) repos
• Let's look at gobot.org• go framework for robots• go get (depency resolution) is designed with security in mind• What possibly can go wrong?
![Page 19: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/19.jpg)
Triggered bro! Investigated with mitmproxy
![Page 20: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/20.jpg)
gobot fails:
• No project contact • No security@ address• Wrote email to one of the top contributors, reaction within a day: assigned to
dev• dev dropped the ball, since it may be an cloud provider issue • Lesson to pickup: Do not try to host your own "go" infra unless you have
control over everything!• Still open since April 1st.
![Page 21: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/21.jpg)
Project issues
![Page 22: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/22.jpg)
Apache Incubator: Skywalking (1)
• 17th Mar : There is a repoToken in pom.xml !• 1,5 h later please file a pull request• Merged within next hour.
![Page 23: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/23.jpg)
Skywalking (2)
• Maven is trying to downloading this insecurely (from bro http.log)
• 1521301611.462089 CketVrXC3jqw3xoSi 172.17.0.3 41818 35.186.232.213 80 1 GET repo.spring.io /ext-release-local/org/jboss/shrinkwrap/shrinkwrap-bom/1.2.3/shrinkwrap-bom-1.2.3.pom - 1.1 Apache-Maven/3.5.2 (Java 1.8.0_151; Linux 4.9.60-linuxkit-aufs) 0 80 404 Not Found (empty) - - - - - - F01i8kqElKM97ajU6 - text/json
• Reason <repository> with repo.spring.io in spring-boot.pom
![Page 24: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/24.jpg)
Spring Boot : transitive dependency
Maven dependencies may have its own repositories.Spring Boot had an insecure repository configuration in its pom
Reported Mar. 24th with pull requestAck 2 days laterOn Mar 27th, a much more in-depth patch was committed in git trunk: Yeah!Included in 1.5.11 Release on April 5thPull request to change spring boot dependency on Jun 11th ack same day.
![Page 25: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/25.jpg)
Apache Camel• Cleanup of repository was done before• Bro traces still shows unnecessary, insecure <repositories>• Pull request on May 2nd, accepted May 7th• Second on 25th of May, accepted same day.
• Open are calls to google analytics ?!?
1523739005.773764 CeLBdP2CjSjsqcchNk 172.17.0.2 46116 216.58.210.14 80 1 GET www.google-analytics.com /__utm.gif?utmwv=1&utmn=1755593970&utmcs=UTF-8&utmsr=1440x900&utmsc=32-bit&utmul=en-us&utmje=1&utmfl=9.0 r28&utmcr=1&utmdt=runtime-2.4.23-jetty/9.4.6.v20170531&utmhn=7df51ed6aecb&utmr=http://async-io.org&utmp=/runtime/2.4.23/jetty/9.4.6.v20170531&utmac=UA-31990725-1&utmcc=__utma='-775698071.1076780858.1523739005744.1523739005744.1523739005744.2;+__utmb=-775698071;+__utmc=-775698071;+__utmz=-775698071.1523739005744.2.2.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none);+__utmv=-775698071 - 1.1 Java/1.8.0_151 (amd64; Linux 4.9.87-linuxkit-aufs) 0 35 200 OK - - (empty) - - - - - - FwmNR41yBTzsO16Kvi - image/gif •
![Page 26: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/26.jpg)
Apache Servicemix repo• Misused subversion repo : svn.apache.org/repos/asf/servicemix/m2-repo . Oops.
• Tested again maven central. JAR are ok, pom seem to be handcrafted, MD5 and SHA1 are sometimes only partially valid.
• Three JAR are not legit:
• jpam-1.1 is compiled with a different compiler
• jsr-157 is the release bundle from the JSR Process (STAX) and obsoloete
• jsch-0.1.44 in maven central is corrupt
• Looks like a repository when back in time it was not clear how to upload things when author doesn't do it. (POM only
![Page 27: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/27.jpg)
Apache plc4x
![Page 28: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/28.jpg)
eclipse hawkbit
• Within a Dockerfile:> gpg --keyserver pgp.mit.edu --recv-keys 385CBC1C7F667FAE> wget ....> gpg --batch --verify ...
GPG Keyserver is not secure (intended), since there are collision attacks possible on key id. You need to get keys from a trusted party.Pull request on Feb 18th, accepted Feb 28th.
![Page 29: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/29.jpg)
Oracle j2ee: jaxbNot part of eclipse jakarta, btw• Pull request still open since Apr 15th
<pluginRepositories> <pluginRepository> <id>releases.java.net</id> <url>http://maven.java.net/content/repositories/releases/</url> </pluginRepository>....
Demo of an attack
![Page 30: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/30.jpg)
http://
• Data may be modified in between• Data are not authenticated• Data may be from a different server• Data may be forged by attacker
![Page 31: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/31.jpg)
Attacking
• Men in the middle (MITM) Attack• Intercepting http traffic• Demo with ettercap: • ARP Poisoning• DNS Attack• Redirects maven.java.org to own, tainted repository with fake maven-
compiler-plugin
![Page 32: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/32.jpg)
Live Hack
• How to attack jaxb implementation ...
• Will start a windows calc.exe when compiling demo
![Page 33: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/33.jpg)
Method 2: Dependency checker
• Insert the OWASP Dependency Checker into pom.xml• https://jeremylong.github.io/DependencyCheck/
• Run: mvn verify• Look at the results (Many false positives)• Try to patch• Watch community to iron things out• Profit
![Page 34: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/34.jpg)
Apache JMeter
Found by OWASP Dependency Checker:Tripped over outdated Bouncy Castle library.Submitted a fix in Feb 17th, refined fix committed within 12h !
![Page 35: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/35.jpg)
paho.mqtt.java (eclipse)
![Page 36: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/36.jpg)
paho.mqtt.java (eclipse)
![Page 37: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/37.jpg)
paho.mqtt.java (eclipse)
• Turns out that it uses the release still uses the "kepler" release (2013) unmaintained.
• Trunk uses "mars", same problem.• Filed pull request to change to "oxygen" via github. Rejected 10days later:
Will break thinks but devs want to remove that component anyway, since there are better alternatives.
• RESOLUTION: superseded by mqtt-spy
![Page 38: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/38.jpg)
random project
![Page 39: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/39.jpg)
random project
A lot of false positives on OWASP dependency checker.fasterxml-jackson seems relevant.A transitive dependency seems to be from jboss resteasy 3.0.7.FinalWas not able to fix it with updating to 3.0.24 since maven dependency resolution kicked in. I did not find the dependency triggering it.Still open...
![Page 40: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/40.jpg)
Receiving a security report
![Page 41: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/41.jpg)
Fixing Apache Bigtop
• Apache Bigtop received a security report a few weeks ago• "Download your stuff securely ... and assign CVE ..."• OK, I know about that we do insecure things, but I missed to fixed it in
Apache Bigtop, I cared about others. • Interesting: • It took me 3 days to discuss how to react, since PMC's are located in asia,
usa and europe (me)
![Page 42: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/42.jpg)
Download and Verify
• Download securely from Apache Infra:• INFRA doesn't like downloads from www.apache.org/dist . Should use
mirroring• But most mirrors are http:// only or not so trusted domains like "klaus-
uwe.me"• See http://maven.apache.org/download.cgi for detailed information how to do
it.
![Page 43: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/43.jpg)
Wrap up
![Page 44: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/44.jpg)
Takeaway: Insecure/Dubious Dependencies
• Maven: Look out for <repository> tags in pom.xml• Look at transitive dependencies in maven output• Even well-known frameworks may have serious issues• In real life you have to do an network analysis
![Page 45: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/45.jpg)
![Page 46: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/46.jpg)
Takeaway: Vulnerable Dependencies
• Fix maven dependencies issue is really tough• OWASP plugin good, handling complicated• Maybe we need support from Apache Maven for logging the dependency
resolution for one single artifact • You have to be very educated to eliminate the false positives (Example
almost every apache .. triggers CVE for Apache http, bzip java implementations triggers CVE reports for C implementation)
![Page 47: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/47.jpg)
![Page 48: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/48.jpg)
Takeaway: Infrastructure
• Infrastructure problems:• If you do not get a response within 5 Days, you likely won't get any response.
![Page 49: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/49.jpg)
![Page 50: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/50.jpg)
Personal wishlist for maven.next
• Bundle essential plugins: Do not download plugins for "clean", or simple java compile
• Make repositories explicit, not implicit in a dependency.• Write out what triggered the versioning decision.• Enforce https:// for repositories (like go get)
![Page 51: Attacking IoT Developers - FOSS Backstage](https://reader030.vdocuments.mx/reader030/viewer/2022012711/61aaf18d80c9fb318b0f0497/html5/thumbnails/51.jpg)
Questions/Comments?
• Contact me at • of ät oflebbe.de
• Slides will be available at www.oflebbe.de