linux io introduction-fudcon-2015-with-demo-slides

Agenda

• SCSI Targets

• STGT and Need of LIO

• LIO Introduction

• LIO Architecture

• LIO Directory structure

• Linux Storage Stack and LIO

• LIO Configfs Interface

• LIO Configfs tree

• LIO Command Line interface

• Use cases

• LIO users and partners

• Demo

2

SCSI targets

• The subsystem enables a computer node to behave as a SCSI storage device, responding to storage requests by other SCSI initiator nodes.

• Opens up the possibility of creating custom SCSI devices and putting intelligence behind the storage.

• Extra functionality can be added on top of it – RAID, snapshots, replication, HA, backup, data deduplication.

• Some existing SCSI target implementations – SCST, IET, STGT

3

STGT and Need of LIO

• Many major distributions included STGT support

for userspace iSCSI target mode

• But is still unsupported in many commercial cases

• Major distributors wanted to see proper in-kernel

iSCSI target mode support and the community

offered out of tree modules for those interested

• Many developers, hardware vendors and users

wanted upstream kernel level iSCSI target mode

4

LIO Introduction

• TCM/LIO = Target core module / Linux- IO

• Open source Linux SCSI target

• Developed by Datera Inc. (Rising tide systems)

• Merged in Linux Kernel on March, 2011

• Requirements

o Linux Kernel version >= 2.6.38

• Supported by almost all Linux distributions by

default

5

LIO Introduction…Contd

• Supports SCSI Advanced featureso Persistent Reservations (SPC3/SPC4)

o ALUA (Asymmetric Logical Unit Assignment)�Both explicit and implicit

o T10 Data Integrity Format

o VAAI (vStorage APIs for Array Integration)

o Multiple connection per session (MC/S)

o iSCSI ERLs (Error Recovery Levels)�ERL 0 – Session Recovery

�ERL 1 – Digest failure Recovery

�ERL 2 – Connection Recovery

6

LIO Architecture

7

LIO Architecture…Contd

LIO consists of three group of modules :

• Fabric moduleso Frontend of LIO, encapsulates supported storage protocols

o Supports iSCSI, FC, FCOE, SRP, infiniband, vHost

• Storage modules / backstoreso Backend of LIO – provide access to imported devices via device drivers

o Supports –� Block ( HDDs, SSDs, LVM, Raids )

� File

� RAMdisk

• Core Functionality (target_core_mod.ko)o Manages resources and memory

o Implements Persistent resvervation, ALUA and other core features.

8

Linux Storage Stack and LIO

9

VFS (Virtual File System)Memory Management

ext3 ext4 others

Block Layer and Device Mapper

SCSI Layer

target

/dev/sdb

LIO core modules

FC iSCSI SRP

Block Fileio Raw

/dev/mapper/xxx /dev/vg/lv ext4file

Logical Volume Manager

Low Level

drivers

user

Kernel

Storage

LIO directory Structure

10

Storage and

core modules

Core data

structures

declarations

LIO dir

structure

LIO Configfs Interface

• Configfs is a RAM-based virtual file system similar to sysfs (but not same)

• Precisely, it is inverse of sysfs.

• Provide better control from userland

• Unlike sysfs, configfs allows to create, manage and destroy kernel objects from user-space

• Typically mounted at /config or /sys/kernel/config

• LIO uses it for management and configuration of storage and fabric modules.

11

LIO Configfs tree

12

LIO Command Line Interface

• lio_node and tcm_nodeo Low level utility to manage storage and fabric configfs nodes

o Python based cli for developers, intergrators and advanced users

o Different tcm utilities available for fabric modules

• lio-dump, tcm-dumpo Save configuration

• target-clio Provide unified single node SAN configuration shell

o Fabric agnostic

o Exports functionality via RTSlib library and API�python-rtslib (python API library over configfs)

13

Use Cases

• HA clustering

• Openstack Cinder apis written to consume LIO iSCSI storage

• IO virtualization(IOV) with KVM on IOV capable hardware

• iSCSI LAN and WAN replication – DRBD and pacemaker support

• iSCSI boot using PXE

• Implementing advanced features on top of LIO –dedupe, thin provisioning, etc

14

LIO users and partners

15

Its DEMO time

16

iSCSI discovery and LIO

Data structures

17

STEP 2:

# Create 2 sparse files and we will export them as file based LUNs

dd if=/dev/zero of=/data/file1 bs=1K count=1 seek=10000

dd if=/dev/zero of=/data/file2 bs=1K count=1 seek=10000

file1

file2 STEP 1:

# Start the target rc script

# This will insert target core module (core Engine activated)

/etc/init.d/target start


Data structures

18

fileio_0disk1

disk2

default_tg_

pt_gp

default_tg_

pt_gp

STEP 3:

# Allocate fd_dev and create file based luns

# This will insert LIO’s fileio storage module

tcm_node --fileio fileio_0/disk1 /data/file1 10240000

tcm_node --fileio fileio_0/disk2 /data/file2 10240000

file1(fd_dev)

file2 (fd_dev)

se_hba

se_device

t10_alua_tg_

pt_gp


Data structures

19

fileio_0disk1

disk2

default_tg_

pt_gp

default_tg_

pt_gp

STEP 4:

# Add and bind LUN0 to 1st target port of portal group 1

lio_node --addlun <target_iqn1> 1 0 iscsi00 fileio_0/disk1

# Add and bind LUN0 to 2nd target port of portal group 1


# Add and bind LUN1 to 1st target port of portal group 1


# Add and bind LUN1 to 2nd target port of portal group 1


file1(fd_dev)

file2 (fd_dev)

se_hba

se_device

t10_alua_tg_

pt_gp

tpgt_1

lun_0

lun_1

se_lun

iscsi_portal

_group

iSCSI discovery – Target port binding

20

TCM/LIO

file1 file2

lun 0

(iscisi00)

lun 1

(iscsi01)

target_port0 target_port1

==TARGET PORT BINDING==

lun0 binds to target_port0 and target_port1

lun1 binds to target_port0 and target_port1


Data structures

21

fileio_0disk1

disk2

default_tg_

pt_gp

default_tg_

pt_gp

file1(fd_dev)

file2 (fd_dev)

se_hba

se_device

t10_alua_tg_

pt_gp

tpgt_1

lun_0

lun_1

se_lun

iscsi_portal

_group

STEP 5:

# Create network portals for target ports

lio_node --addnp <target_iqn1> 1 <IP address:port>

lio_node --addnp <target_iqn2> 1 <IP address:port>

IP:port

IP:port

iscsi_np

iscsi_np

STEP 6:

# Disable iSCSI authentication of target portal group . Enabled by default

lio_node --disableauth <target_iqn1> 1

lio_node --disableauth <target_iqn2> 1


Data structures

22

fileio_0disk1

disk2

default_tg_

pt_gp

default_tg_

pt_gp

file1(fd_dev)

file2 (fd_dev)

se_hba

se_device

t10_alua_tg_

pt_gp

tpgt_1

lun_0

lun_1

se_lun

iscsi_portal

_group

STEP 7:

# Give access of bound luns to initiator and assign mapped lun numbers

lio_node –addlunacl <target_iqn1> 1 <initiator_iqn> 0 0

lio_node --addlunacl <target_iqn2> 1 <initiator_iqn> 0 1



IP:port

IP:port

iscsi_np

iscsi_np

lun_0 lun_2

lun_1 lun_3

Mapped

luns

STEP 8:

# Enable target portal group

lio_node --enabletpg <target_iqn1> 1

lio_node --enabletpg <target_iqn2> 1

23

TCM/LIO

file1 file2

lun 0

(iscisi00)

lun 1

(iscsi01)


IP:Port IP:Port

Network portals Target portal

group

iSCSI discovery – Network Portals and

Target Portal Group

lun 0

lun 1 lun 2

lun 3

lnitiator


Data structures

24

fileio_0disk1

disk2

default_tg_

pt_gp

default_tg_

pt_gp

file1(fd_dev)

file2 (fd_dev)

se_hba

se_device

t10_alua_tg_

pt_gp

tpgt_1

lun_0

lun_1

se_lun

iscsi_portal

_group

STEP 9:

# Initiator discovers and login to target ports to establish iscsi session with target

iscsiadm --mode discovery -t sendtargets --portal <IPaddr:Port>

iscsiadm --mode node --targetname <target_iqn1> --portal <IPaddr:Port> --login

iscsiadm --mode node --targetname <target_iqn2> --portal <IPaddr:Port> --login

IP:port

IP:port

iscsi_np

iscsi_np

lun_0 lun_2

lun_1 lun_3

Mapped

luns

iSCSI login

and session

creation

25

TCM/LIO

file1 file2

lun 0

(iscisi00)

lun 1

(iscsi01)


IP:Port IP:Port

Network portals Target portal

group

/dev/sda

/dev/sdb

/dev/sdc

/dev/sdd

Initiator

lun 0

lun 1 lun 2

lun 3

References

• RFC 3720

• LIO wiki page

• SCST vs LIO

• iscsi targets wiki page

• LIO-utils – github

• Configfs

• Targetcli and rtslib

• Cinder plugin for LIO

• Open iSCSI utility

• STGT project

• Distributed replicated block device

26

Questions ?

27

Author Biography

28

Kashish Bhatia – Member of Technical Staff, VMware

• Technology enthusiast, with most of the experience in storage domain

• Has exposure to other Linux subsystems like filesystems, block layer, Selinux

• Experience of working on distributed file system (Lustre) and storage virtualization

• Interests : open source, linux kernel hacking

• Published technical paper in Openstack Summit Nov 2013

• Bachelor of Engineering, Computer Science, Pune University

• To know more about Kashish connect on LinkedIn

linux io introduction-fudcon-2015-with-demo-slides

Software