rdm#2- the distributed web

RDM#2

LISBON

Research & Development Meetup

LISBON

David Dias, P2P SE&R @ Protocol Labs

João Ascensão, Data Scientist @ Uniplaces

tw: @daviddias gh: @diasdavid

tw: @jtascensao gh: @jtascensao

LISBON

What is RDM?

LISBON

Space to invest time in learning things that are hard to grasp

LISBON

world of possibilities

LISBON

what actually gets researched

LISBON

what gets developed

LISBON

what gets deployed

LISBON

what most people are doing

PROTOCOL LABS

LISBON

FORMAT

- New topics / Research - Hacking + Digesting + Publishing

LISBON

TODAY

- The Distributed Web - Machine Learning / AI

The Distributed Web

Serve data to everywhere

• Motivations

• IPFS, the Protocol

• MerkleDAG (Directed Acyclic Graph)

• DHT (Distributed Hash Table)

• Motivations

• IPFS, the Protocol, MerkleDAG, IPNS, Network

• IPFS in action

• IPFS, the project

• filecoin

• libp2p

• starship

• gx

• full IPFS JavaScript implementation

• mutable file system and the files API

• Other DHT

• Structure P2P Networks

• Unstructured P2P Networks

• NAT Traversal

• Relay/Turn

• registry-mirror

• smart chunking

• Transports

• Multiplexers

• Self Describing Data Types

• Search

• Station

• and more

Location Addressing

http://10.20.30.40/foo/bar/baz.png

location path

http://example.com/foo/bar/baz.png

http://10.20.30.40/foo/bar/baz.png

you

http://10.20.30.40/foo/bar/baz.png

10.20.30.40

you

http://10.20.30.40/foo/bar/baz.png

10.20.30.40

you

200 MB

200 MB x 30 x 8 = 48 GB

2,544,837,694

2,387,990,609

508.968 PB

Bandwidth

https://www.backblaze.com/blog/price-gap-storage-vs-bandwidth/

Latency

Amazon Google

DataCenters

Disconnected / Offline

low bw interference congestion travelisp outagedatacenter

Permanence Security

IoT

Control

Offline

Bandwidth

Disconnected

InterPlanetary FileSystem

a protocol to upgrade the web

InterPlanetary FileSystem

InterPlanetary FileSystem

DISTRIBUTEDPERMANENT

MERKLETHE WEB

PermanentDistributed Safer

SmarterOffline

Faster

SFS

web

DHT+

web

DHT

SFS

routing

network

exchange

merkledag

naming

applications

The Stack

routing

network

exchange

merkledag

naming

applicationsThe Stack

Defining the Data

Moving the Data

Using the Data

IPLD

libp2p

IPNS

8AyaTQMhJklMoGa3WaDgwcAEPcKQeipgDu64qajkr3fNcVt1LVMD

KQsXSobGtbNhFsv3qA44E7iXAygcuSfVhW8RV3zR9F3MFvbVaqf9

8AyaTQMhJklMoGa3WaDgwcAEPcKQeipgDu64qajkr3fNcVt1LVMD

hash( ) -> KQsXSobGtbNhF

KQsXSobGtbNhFsv3qA44E7iXAygcuSfVhW8RV3zR9F3MFvbVaqf9

8AyaTQMhJklMoGa3WaDgwcAEPcKQeipgDu64qajkr3fNcVt1LVMD

hash( ) -> KQsXSobGtbNhF

KQsXSobGtbNhFsv3qA44E7iXAygcuSfVhW8RV3zR9F3MFvbVaqf9

MERKLE LINK

cvs/svn

cvs/svn

cvs/svn

merkledag

in IPFS data forms a dag

it’s called the merkle dagbecause the links are hashes

any data structures are represented as dags

unix files and dirs

any data structures are represented as dags

blob, tree, commit

unix files and dirs

any data structures are represented as dags

blob, tree, commit

unix files and dirs

any data structures are represented as dags

blocks, txns, wallets

blob, tree, commit

unix files and dirs

any data structures are represented as dags

blocks, txns, wallets

kv-stores

blob, tree, commit

unix files and dirs

any data structures are represented as dags

blocks, txns, wallets

kv-stores

unixfs as a dag

files are dag nodes

unixfs as a dag

files are dag nodes

BIG files may be split into many

unixfs as a dag

directories are also dag nodes

files are dag nodes

BIG files may be split into many

unixfs as a dag

which link to others

directories are also dag nodes

files are dag nodes

BIG files may be split into many

unixfs as a dag

http://10.20.30.40/foo/bar/baz.pnglocation path

http://example.com/foo/bar/baz.png

http://10.20.30.40/foo/bar/baz.pnglocation path

http://example.com/foo/bar/baz.png

/ipns/example.com/foo/bar/baz.png

/ipfs/QmW98pJrc6FZ6/foo/bar/baz.pngcontent path

10.20.30.40

you

HTTP

http://10.20.30.40/foo/bar/baz.png

/ipfs/QmW98pJrc6FZ6/foo/bar/baz.png

http://10.20.30.40/foo/bar/baz.png

10.20.30.40

you

/ipfs/QmW98pJrc6FZ6/foo/bar/baz.png

IPFS

/ipfs/QmW98pJrc6FZ6/foo/bar/baz.pngcontent path

/ipfs/QmW98pJrc6FZ6/foo/bar/baz.pngcontent path

/ipfs/QmW98pJrc6FZ6/foo/bar/baz.pngcontent path

foo

bar

baz.png

QmW98pJrc6FZ6

/ipfs/QmW98pJrc6FZ6/foo/bar/baz.pngcontent path

foo:HB

HBHC

HD

bar:HC

baz.png:HDQmW98pJrc6FZ6

foo

bar

baz.png

DHT

DHT are the Holy Grail* of P2P Networks

*from a survey paper

15

610

50

55

1 10 12 20 30 50 54 58 70 88 81 83 88 90 92 95 97 99 101105 300411 500

1 10 12 20 30 50 54 58 70 88 81 83 88 90 92 95 97 99 101105 300411 500

1 10 12 20 30 50 54 58 70 88 81 83 88 90 92 95 97 99 101105 300411 500

hash( )

1 10 12 20 30 50 54 58 70 88 81 83 88 90 92 95 97 99 101105 300411 500

hash( ) = 91

1 10 12 20 30 50 54 58 70 88 81 83 88 90 92 95 97 99 101105 300411 500

hash( ) = 91

1 10 12 20 30 50 54 58 70 88 81 83 88 90 92 95 97 99 101105 300411 500

Who has fim , which hash( ) = 91?

The Distributed Web

• Applications with no-origin

• Leverage network cache

• Resilient to network splits

Serve data to everywhere

Serve data from everywhere

David Dias - http://daviddias.me IPFS http://ipfs.io Protocol Labs http://ipn.io

Thank you!David Dias - http://daviddias.me IPFS http://ipfs.io Protocol Labs http://ipn.io