“data over time” delay tolerant networks joy ghosh landercse@buffalo

““Data over Time”Data over Time”Delay Tolerant NetworksDelay Tolerant Networks

Joy GhoshJoy GhoshLANDERLANDER

cse@buffalocse@buffalo

OverviewOverview

Challenged Internets (CI)Challenged Internets (CI) Shortfalls of TCP/IP in CIShortfalls of TCP/IP in CI Concept of DTNConcept of DTN

Message-oriented Message-oriented Overlay architectureOverlay architecture Classes of Service (CoS)Classes of Service (CoS)

Routing IssuesRouting Issues ConclusionConclusion

Challenged Internets (CI)Challenged Internets (CI)

Some causes:Some causes: Host / Router mobility extremesHost / Router mobility extremes Power managementPower management InterferenceInterference

ExamplesExamples Terrestrial mobile nets: partitionsTerrestrial mobile nets: partitions Sensor / Actuator: power savingSensor / Actuator: power saving Military ad-hoc: jammingMilitary ad-hoc: jamming

Paths & LinksPaths & Links

High end-to-end latencyHigh end-to-end latency Transmission timeTransmission time Propagation delayPropagation delay Processing timeProcessing time Queuing timeQueuing time

Low asymmetric data ratesLow asymmetric data rates Return channel may be even absentReturn channel may be even absent Avoid chatty conversationAvoid chatty conversation Data carousels / de-fragmentationData carousels / de-fragmentation

Non-faulty DisconnectionNon-faulty Disconnection Due to motionDue to motion

Predictable (satellites, busses) Predictable (satellites, busses) Opportunistic (node coming in range)Opportunistic (node coming in range) End user or router motionEnd user or router motion Third entity motion causing blockThird entity motion causing block

Due to low-duty-cycleDue to low-duty-cycle PredictablePredictable Low capability devices save powerLow capability devices save power Usually triggered by events - sensorsUsually triggered by events - sensors

Routing Routing lack of reach is normal – not a faultlack of reach is normal – not a fault Pre-schedule data based on outage infoPre-schedule data based on outage info


Long Queuing TimesLong Queuing Times Next hop unavailableNext hop unavailable Alternative next hops Alternative next hops Persistent storagePersistent storage


Interoperability considerationsInteroperability considerations Most protocols only use LL & MACMost protocols only use LL & MAC Simple & local in scopeSimple & local in scope May use application specific framing May use application specific framing

formatsformats May fail to implement reliability, May fail to implement reliability,

congestion control, and securitycongestion control, and security Not well matched for Internet ProtocolsNot well matched for Internet Protocols Minimal assumptions of underlying Minimal assumptions of underlying

protocol stack capabilities for protocol stack capabilities for interoperationinteroperation

Network ArchitectureNetwork Architecture

SecuritySecurity Exotic media Exotic media forwarding costly forwarding costly Protection of forwarding serviceProtection of forwarding service Access control matrix for CoSAccess control matrix for CoS Authentication at critical pointsAuthentication at critical points End-to-end security not suitableEnd-to-end security not suitable

Too much delay for key exchangeToo much delay for key exchange Drop unwanted traffic ASAPDrop unwanted traffic ASAP

Network ArchitectureNetwork Architecture

Limited longevityLimited longevity Hostile environmentsHostile environments Power exhasutionPower exhasution Delegation of reliable deliveryDelegation of reliable delivery

Low duty cycleLow duty cycle Save power / event triggeredSave power / event triggered

Limited resourcesLimited resources Limited memory & processingLimited memory & processing

End System CharacteristicsEnd System Characteristics

OverviewOverview




TCP/IP basicsTCP/IP basics

Expectations of TCP/IPExpectations of TCP/IP Map: IP packet format, domain names Map: IP packet format, domain names

physical network physical network Fundamental serviceFundamental service

Fully-connected, best-effort, unicast Fully-connected, best-effort, unicast datagram transportdatagram transport

Suffers when:Suffers when: Links change radically (wireless, ATM)Links change radically (wireless, ATM) Service enhanced significantly (IP Service enhanced significantly (IP

multicast, QoS)multicast, QoS)

TCP/IP applied directly to CITCP/IP applied directly to CI

Transport Layer (TCP, SCTP, Transport Layer (TCP, SCTP, UDP)UDP) RTT too high for slow startRTT too high for slow start Waste of bandwidthWaste of bandwidth False congestion indicationFalse congestion indication MSL in RFC793 is 2 mins MSL in RFC793 is 2 mins Low Low


Network Layer (IP)Network Layer (IP) Lossy network Lossy network smaller frames smaller frames Frequent fragmentationFrequent fragmentation No fragment retransmissionNo fragment retransmission Maximum TTL is 225 Maximum TTL is 225 Low Low


Application Layer (Routing)Application Layer (Routing) Delayed soft-state refreshesDelayed soft-state refreshes Label links as non-operating (e.g., Label links as non-operating (e.g.,

RIP, BGP)RIP, BGP) Lack of response to HELLO Lack of response to HELLO

messages (e.g., OSPF)messages (e.g., OSPF) Application Layer (Others)Application Layer (Others)

Delay cause application timeoutDelay cause application timeout Delay may outlive application lifeDelay may outlive application life

Application Development PracticeApplication Development Practice

Network based applicationsNetwork based applications Application level timeoutsApplication level timeouts Lack of failover – no auto routeLack of failover – no auto route Synchronous programming Synchronous programming Chatty applications protocols Chatty applications protocols

(FTP requires 6 RTT)(FTP requires 6 RTT) New network API requiredNew network API required

Proxies & Protocol BoostersProxies & Protocol Boosters

Performance Enhancing Proxies (PEPs), Performance Enhancing Proxies (PEPs), Protocol Boosters - Link repair approachesProtocol Boosters - Link repair approaches

““fool” TCP/IP based end stations to operate fool” TCP/IP based end stations to operate more efficiently over paths with poor linksmore efficiently over paths with poor links

Different levels of transparencyDifferent levels of transparency Local connection terminationLocal connection termination Modification of TCP ACK streamModification of TCP ACK stream Retransmit lost TCP packets w/o end user inputRetransmit lost TCP packets w/o end user input

Use of PEPs discouraged due to fragilityUse of PEPs discouraged due to fragility Confound end-to-end diagnosticsConfound end-to-end diagnostics Hurt security below transport (IPSEC)Hurt security below transport (IPSEC) Alternative: application layer proxiesAlternative: application layer proxies

Internet to special network name mapping and Internet to special network name mapping and protocol translationprotocol translation

Use of Electronic MailUse of Electronic Mail

ProsPros Asynchronous message deliveryAsynchronous message delivery Delay toleranceDelay tolerance Flexible name/address semanticsFlexible name/address semantics

ConsCons Lack of dynamic routingLack of dynamic routing SMTP is a chatty protocolSMTP is a chatty protocol Mostly reliable with likely failure Mostly reliable with likely failure

notificationnotification

OverviewOverview




Delay Tolerant NetworkDelay Tolerant Networkmessage-oriented overlay architecturemessage-oriented overlay architecture

Interoperability among CI & InternetInteroperability among CI & Internet Overlaid above different protocol stacks in Overlaid above different protocol stacks in

various networksvarious networks Store & forward message delivery for non-Store & forward message delivery for non-

interactive trafficinteractive traffic a-priori knowledge of communication a-priori knowledge of communication

capacity needs (intermediate storage, capacity needs (intermediate storage, retransmission bandwidth)retransmission bandwidth)

Virtual circuits unsuitable due to pre-Virtual circuits unsuitable due to pre-establishment of networkestablishment of network

Favors asynchronous I/O programmingFavors asynchronous I/O programming

Network of Regional NetworksNetwork of Regional Networks

Similar protocol stacks in one regionSimilar protocol stacks in one region DTN Gateways – Metanet WaypointDTN Gateways – Metanet Waypoint

Provides entry to a region (protocol translation)Provides entry to a region (protocol translation) Name mapping from global tuple to locally resolved nameName mapping from global tuple to locally resolved name Enforces policy and controlEnforces policy and control Persistent storage for reliable deliveryPersistent storage for reliable delivery

Delay Isolation via Delay Isolation via Transport Layer TerminationTransport Layer Termination

Persistent storage

Bundle EncapsulationBundle Encapsulation

Bundles consist of 3 things:Bundles consist of 3 things: Source application’s user dataSource application’s user data Control information from source application to Control information from source application to

destination application (how to handle data)destination application (how to handle data) Bundle headerBundle header

DTN Region NamingDTN Region Naming

Name TuplesName Tuples

(Late Binding)

InterPlaNetary (IPN) Internet InterPlaNetary (IPN) Internet

Class of Service (CoS)Class of Service (CoS)

Challenged Internet Challenged Internet limited limited resourcesresources

Priority based resource Priority based resource allocationallocation

Subset of types of service by Subset of types of service by USPSUSPS

Class of Service (CoS)Class of Service (CoS)

Challenged Internet Challenged Internet limited resources limited resources Priority based resource allocationPriority based resource allocation USPS has simple but elaborate servicesUSPS has simple but elaborate services Subset of types of service by USPSSubset of types of service by USPS DTN CoSDTN CoS

Custody TransferCustody Transfer Return ReceiptReturn Receipt Custody-Transfer NotificationCustody-Transfer Notification Bundle-Forwarding NotificationBundle-Forwarding Notification Priority of Delivery: Bulk, Normal, ExpeditedPriority of Delivery: Bulk, Normal, Expedited AuthenticationAuthentication

DTN Class of Service (CoS)DTN Class of Service (CoS)

Custody TransferCustody Transfer

Path Selection & SchedulingPath Selection & Scheduling No end-to-end pathNo end-to-end path Routes are a cascade of time-dependent Routes are a cascade of time-dependent

contactscontacts Contact parametersContact parameters

Start & end timesStart & end times CapacityCapacity LatencyLatency End pointsEnd points DirectionDirection

Predictable vs. Opportunistic contactsPredictable vs. Opportunistic contacts Multi-commodity flow optimization problemMulti-commodity flow optimization problem ““flows over time” optimization problem – flows over time” optimization problem –

NP-hardNP-hard More under “Routing Issues”More under “Routing Issues”

Protocol Translation &Protocol Translation &Convergence LayersConvergence Layers

Bundle forwarder may augment Bundle forwarder may augment underlying protocol stackunderlying protocol stack Reliable deliveryReliable delivery Connection setup with notificationConnection setup with notification Flow controlFlow control Congestion controlCongestion control Message boundariesMessage boundaries

May fall back to retransmission May fall back to retransmission timers in case of failed connectionstimers in case of failed connections

Structure of Bundle ForwarderStructure of Bundle Forwarder

Time SynchronizationTime Synchronization

Required for scheduling, path Required for scheduling, path selection, and congestion selection, and congestion managementmanagement

Devices in challenged environments Devices in challenged environments often collect data / position w.r.t. timeoften collect data / position w.r.t. time

Pre-programmed control instructions to Pre-programmed control instructions to be executed in future timebe executed in future time

Reliability of Reliability of post-facto post-facto data analysisdata analysis Removal of expired pending messagesRemoval of expired pending messages Protocols like NTP are used Protocols like NTP are used

SecuritySecurity

Per-hop security to avoid delayPer-hop security to avoid delay Drop unwanted packets ASAPDrop unwanted packets ASAP Only edge routers need per user certificateOnly edge routers need per user certificate Not secure against router compromiseNot secure against router compromise

Usercertificate

Congestion & Flow ControlCongestion & Flow Control Flow Flow

DTN forwarder uses available underlying region-DTN forwarder uses available underlying region-local transport protocollocal transport protocol

If not available, its constructed at the DTN If not available, its constructed at the DTN convergence layerconvergence layer

CongestionCongestion Long absence of contacts leads to massive Long absence of contacts leads to massive

storage of datastorage of data Data for which custody has been accepted cannot Data for which custody has been accepted cannot

be discarded be discarded Priority queue for custody storagePriority queue for custody storage

Large messages are denied custodyLarge messages are denied custody Messages are spooled FCFS based on priorityMessages are spooled FCFS based on priority

Potential problemsPotential problems Priority invasion (lower custodially recvd blocks high)Priority invasion (lower custodially recvd blocks high) Head-of-line blocking (custodially recvd w/o contact)Head-of-line blocking (custodially recvd w/o contact)

Proactive : admission controlProactive : admission control Reactive: reservation, rejection, etc.Reactive: reservation, rejection, etc.

OverviewOverview




Network model for RoutingNetwork model for Routing

Routing IssuesRouting Issues

No end-to-end pathNo end-to-end path Both proactive and reactive failBoth proactive and reactive fail

Capacity is time dependentCapacity is time dependent Long buffering of messagesLong buffering of messages

Multiple contactsMultiple contacts Might be more optimal to wait for Might be more optimal to wait for

future contact than send on future contact than send on available oneavailable one

Routing ObjectiveRouting Objective Maximize probability of message deliveryMaximize probability of message delivery Prevent buffer overflowPrevent buffer overflow Speed up the release of resourcesSpeed up the release of resources Minimize the end-to-end delayMinimize the end-to-end delay

Use knowledge of future topology dynamics in Use knowledge of future topology dynamics in path selectionpath selection

Per-hop routing preferred over Source Per-hop routing preferred over Source routingrouting Flexibility to change next hopFlexibility to change next hop

Message splitting is allowedMessage splitting is allowed Fragmentation of large messagesFragmentation of large messages Each fragment routed independently Each fragment routed independently

Knowledge OraclesKnowledge Oracles

Contacts Summary OracleContacts Summary Oracle Time-invariant / aggregate Time-invariant / aggregate

characteristics of contactscharacteristics of contacts Contacts OracleContacts Oracle

Time-varying DTN multi-graphTime-varying DTN multi-graph Queuing OracleQueuing Oracle

Buffer occupancy at any node at Buffer occupancy at any node at any timeany time

Traffic Demand OracleTraffic Demand Oracle Future traffic demandFuture traffic demand

Routing Algorithm ClassesRouting Algorithm Classes

Zero KnowledgeZero Knowledge No oracles / minimal extremeNo oracles / minimal extreme

Complete KnowledgeComplete Knowledge All oracles / LP formulation All oracles / LP formulation

Partial KnowledgePartial Knowledge One or more oraclesOne or more oracles Practical algorithmsPractical algorithms

Knowledge vs. Performance Knowledge vs. Performance

Zero Knowledge RoutingZero Knowledge Routing

First Contact (FC)First Contact (FC) Forward to first available contactForward to first available contact No sense of direction / may oscillateNo sense of direction / may oscillate

Requires only local knowledgeRequires only local knowledge Trivial implementationTrivial implementation ImprovementsImprovements

Incorporate sense of trajectoryIncorporate sense of trajectory Use path vector to prevent loopsUse path vector to prevent loops

Partial Knowledge RoutingPartial Knowledge Routing

Assigns cost to edgesAssigns cost to edges Costs reflect estimated delay on edgeCosts reflect estimated delay on edge

Queuing time: time till contact availableQueuing time: time till contact available Transmission delay: time to inject into edgeTransmission delay: time to inject into edge Propagation delay: time to travel on edgePropagation delay: time to travel on edge

Computes minimum cost pathComputes minimum cost path Conveniently uses different oraclesConveniently uses different oracles Computationally efficient distributed Computationally efficient distributed

algorithms already exist for shortest-algorithms already exist for shortest-path based routing problemspath based routing problems

Finds however single path from source Finds however single path from source to destination – no optimal splittingto destination – no optimal splitting

Partial Knowledge RoutingPartial Knowledge Routing Cost function: Cost function: ωω (e, t) (e, t)

If message arrives at node ‘u’ at time ‘t’, and if edge ‘e’ If message arrives at node ‘u’ at time ‘t’, and if edge ‘e’ (between ‘u’ & ‘v’) is chosen, message will reach node ‘v’ (between ‘u’ & ‘v’) is chosen, message will reach node ‘v’ at time ‘t + at time ‘t + ωω (e, t)’ (e, t)’

FIFO: if tFIFO: if t11 < t < t22, t, t11 + + ωω (e, t (e, t11) <= t) <= t22 + + ωω (e, t (e, t22)) Algorithm with Time-invariant CostsAlgorithm with Time-invariant Costs

Use modified Dijkstra’s algorithmUse modified Dijkstra’s algorithm If L[v] > (L[u] + If L[v] > (L[u] + ωω (e, L[u] + T)) then (e, L[u] + T)) then

L[v] L[v] (L[u] + (L[u] + ωω (e, L[u] + T)) (T (e, L[u] + T)) (T start time) start time)

Minimum Expected Delay (MED)Minimum Expected Delay (MED) Oracles: Contacts SummaryOracles: Contacts Summary Edge cost = avg. wait time + prop delay + trx delayEdge cost = avg. wait time + prop delay + trx delay Proactive routing is used for time-invariant costProactive routing is used for time-invariant cost Fixed routes for all messagesFixed routes for all messages Minimizes avg. waiting time but doesn’t optimize pathMinimizes avg. waiting time but doesn’t optimize path ImprovementImprovement

Dynamically make use of superior contacts per-hopDynamically make use of superior contacts per-hop Multiple disjoint paths to balance loadMultiple disjoint paths to balance load

Partial Knowledge RoutingPartial Knowledge Routing Cost function: Cost function: ωω’ (e, t, m, s)’ (e, t, m, s)

Edge ‘e’, time ‘t’, message size ‘m’, node assigning cost ‘s’Edge ‘e’, time ‘t’, message size ‘m’, node assigning cost ‘s’ ωω’ (e, t, m, s) = t’ (e, t, m, s) – t + d (e, t’)’ (e, t, m, s) = t’ (e, t, m, s) – t + d (e, t’)where,where,

c (e, t) c (e, t) capacity of edge ‘e’ at time ‘t’ capacity of edge ‘e’ at time ‘t’ Q (e, t, s) Q (e, t, s) queue size at source of edge ‘e’, at time ‘t’ as queue size at source of edge ‘e’, at time ‘t’ as

predicted by node ‘s’predicted by node ‘s’ t’ t’ earliest time queues data at ‘e’ and message can be earliest time queues data at ‘e’ and message can be

injected into the edgeinjected into the edge Integral Integral volume of data through ‘e’ in interval [t, t’’] volume of data through ‘e’ in interval [t, t’’] d (e, t’) d (e, t’) propagation delay seen by message propagation delay seen by message

Partial Knowledge RoutingPartial Knowledge Routing

Algorithms with Time-varying CostsAlgorithms with Time-varying Costs Earliest Delivery (ED)Earliest Delivery (ED)

Contacts Oracle Contacts Oracle Q (e, t, s) = 0 Q (e, t, s) = 0 Source routedSource routed Buffer overflow Buffer overflow cascaded delay cascaded delay

Earliest Delivery with Local Queuing (EDLQ)Earliest Delivery with Local Queuing (EDLQ) Contacts OracleContacts Oracle Q (e, t, s) = data queued for ‘e’ at ‘t’ if e = (s , *)Q (e, t, s) = data queued for ‘e’ at ‘t’ if e = (s , *)

= 0 otherwise= 0 otherwise Per-hop routed Per-hop routed path vector to avoid loops path vector to avoid loops

Earliest Delivery with All Queues (EDAQ)Earliest Delivery with All Queues (EDAQ) Contacts + Queuing OraclesContacts + Queuing Oracles Q (e, t, s) = data queued for ‘e’ at ‘t’ at node sQ (e, t, s) = data queued for ‘e’ at ‘t’ at node s Source routedSource routed Reservation of edge capacity along computed pathReservation of edge capacity along computed path

Complete Knowledge RoutingComplete Knowledge Routing

Simulation Scene 1: Simulation Scene 1: Routing to Remote VillageRouting to Remote Village Village: Kwazulu-NatalVillage: Kwazulu-Natal City: CapetownCity: Capetown Country: South AfricaCountry: South Africa Dialup Dialup 4 Kb/s, (11pm to 6am contact) 4 Kb/s, (11pm to 6am contact) 3 PACSAT satellites3 PACSAT satellites

OSCAR-11, PACSAT, PCSATOSCAR-11, PACSAT, PCSAT Each making 4 trips over city & village Each making 4 trips over city & village

(connecting them)(connecting them) 3 motorbikes 3 motorbikes 1Mbps (5 min contacts) 1Mbps (5 min contacts)

Each making several trips in dayEach making several trips in day MessagesMessages

From Village From Village 1 KB, From City 1 KB, From City 10 KB 10 KB TimeTime

48 Hrs, starting 01/11/2004 11:59pm48 Hrs, starting 01/11/2004 11:59pm

Simulation Scene 1: Simulation Scene 1: Routing to Remote VillageRouting to Remote Village

Simulation Scene 2: Simulation Scene 2: Network of City BusesNetwork of City Buses 20 city buses in San Francisco20 city buses in San Francisco 3 scheduled bus trips (A, B, C)3 scheduled bus trips (A, B, C) ““stop” generated at each turnstop” generated at each turn Constant speed between stopsConstant speed between stops Pause of 0 to 5 min at each stopPause of 0 to 5 min at each stop Speed between 10 and 20 m/sSpeed between 10 and 20 m/s Radio range 100 m by defaultRadio range 100 m by default Traffic: 12 hrs, 1 hr intervals, 20 random src / dst Traffic: 12 hrs, 1 hr intervals, 20 random src / dst

bus pairs, 200 messages per intervalbus pairs, 200 messages per interval

Simulation Scene 2: Simulation Scene 2: Network of City BusesNetwork of City Buses

OverviewOverview




Delay Tolerant Network Delay Tolerant Network Overlay ArchitectureOverlay Architecture Need to explore challenged InternetsNeed to explore challenged Internets Interoperability amongst different Interoperability amongst different

regions of networksregions of networks Current Internet Protocol stack not Current Internet Protocol stack not

suitablesuitable Delay Tolerant Network is an optionDelay Tolerant Network is an option Overlaid architecture above Overlaid architecture above

Transport protocolsTransport protocols Routing protocols for DTN need to Routing protocols for DTN need to

cope with “Data over Time”cope with “Data over Time”

ReferencesReferences

Kevin Fall, Kevin Fall, A Delay-Tolerant Network A Delay-Tolerant Network Architecture for Challenged Internets, Architecture for Challenged Internets, Intel Intel Research Berkeley, Technical Report, IRB-Research Berkeley, Technical Report, IRB-TR-03-003TR-03-003

S. Jain, K. Fall, R. Patra, S. Jain, K. Fall, R. Patra, Routing in a Routing in a Delay Tolerant Network, Delay Tolerant Network, ACM ACM SIGCOMM’04, Sep, 2004SIGCOMM’04, Sep, 2004

Forrest Warthman, (Warthman Forrest Warthman, (Warthman Associates), Associates), Delay-Tolerant Networks Delay-Tolerant Networks (DTNs), (DTNs), A Tutorial, Version 1.1, 3/5/03A Tutorial, Version 1.1, 3/5/03

“data over time” delay tolerant networks joy ghosh landercse@buffalo

Documents

ciconcept of dtnmessage

citransport layer tcp

congestion control

faultpreschedule data

mins lowtcpip

ip multicast

end security

end stations