xmi library

xmi libraryRelease 0.5

Philip Young

Apr 13, 2021

CONTENTS:

1 Overview 31.1 xmi Python Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171.3 Cookbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171.4 Understanding XMI Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191.5 Understanding Virtual Tape Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251.6 IEBCOPY File Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281.7 Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

2 Indices and tables 33

Python Module Index 35

Index 37

i

xmi library, Release 0.5

xmi is a python library developed to handle NETDATA (XMI) and virtual tape files (AWS and HET). It was developedto open and extract (unload) XMI/AWS/HET mainframe files.

CONTENTS: 1


2 CONTENTS:

CHAPTER

ONE

OVERVIEW

Below are links to the xmi python library documentation. XMI/AWS/HET files were poorly documented with infor-mation scattered throughout IBM, CBT Tape, and elsewhere. The sections Understanding XMI Files, UnderstandingVirtual Tapes and IEBCOPY File Format describe the file structure of each.

1.1 xmi Python Library

The NETDATA file format (CMS SENDFILE or TSO TRANSMIT/XMIT) is used primarily to transferfiles between mainframes. The file consists of a dataset “unloaded” with either INMCOPY or IEBCOPYand metadata. Multiple control records exist, see https://en.wikipedia.org/wiki/NETDATA for more de-tails. NETDATA and XMIT/XMI are used interchangably however NETDATA files are more commonlyrefered to as XMI.

The AWSTAPE file format is used to transfer virtual tape files. Originally created for P/390 it is usedprimarily today with virtual tape offerings. AWS is the short name for these tape file types. Later theopensource project Hercules created the Hercules Emulated Tape, or HET, which builds on the AWSformat by adding compression. Virtual tape files consist of one or more datasets and optional metadatastored in labels.

This module consists of methods to extract or open XMI/AWS/HET mainframe files. It also contains theXMIT class which implements XMI, AWS and HET file management to read and extract datasets andmembers.

xmi.convert_ebcdic(ebcdic_file, lrecl=80)Converts an ebcdic mainframe file to text. Returns string.

Parameters

• ebcdic_file (byte) – the file to be converted

• lrecl (int) – Record length. A newline is inserted every lrecl bytes.

xmi.extract_all(mainframe_file, output='./')Extracts all datasets and members to output directory.

Parameters

• mainframe_file (str) – path to XMI/AWS/HET file

• output (str) – output folder. Defaults to current working directory

xmi.list_all(mainframe_file)Returns a list of all datasets and members in XMI/AWS/HET file.

3

https://en.wikipedia.org/wiki/NETDATA


xmi.open_file(filename=None, LRECL=80, loglevel=30, infile=None, outputfolder='./', encod-ing='cp1140', unnum=True, quiet=False, force_convert=False, binary=False, modify-date=False)

Opens a XMI/AWS/HET file and returns an XMIT object

Parameters

• filename (str) – The path and filename of an XMI/AWS/HET file. Defaults to None.

• LRECL (int) – If record length cannot be determined this value is used

• converting from EBCDIC to UTF-8. Defaults to 80. (when) –

• loglevel (int) – Level of logging, based on https://docs.python.org/3/library/logging.html#levels. Defaults to logging.WARNING.

• outputfolder (str) – Output file path for extracted files. Detaults to current workingdirectory.

• encoding (str) – EBCDIC codepage used when translating from EBCDIC to UTF-8.Defaults to cp1140

• infile (str) – folder/file name to use for XMI output instead of the detaset name in-cluded in the metadata.

• unnum (bool) – Some mainframe files have numbers in columns 72-80 denoting line num-bers. If True files converted from EBCDIC will have these columns removed. Default toTrue.

• quiet (bool) – Do not print any output messages whle extracting files. Default to False.

• force_convert (bool) – Converts all files utf-8 ignoring mimetype. Defaults to False.

• binary (bool) – Extract files as binaries, ignoring mimetype. Defaults to False.

• modifydate (bool) – If created date/last modified date information is available changethe last modified date of the extracted file to match. Defaults to False.

class xmi.XMIT(filename=None, LRECL=80, loglevel=30, infile=None, outputfolder='./', encod-ing='cp1140', unnum=True, quiet=False, force_convert=False, binary=False, modify-date=False)

This class contains modules to parse the control records for NETDATA, AWSTAPE and HET file format aswell as methods to parse IEBCOPY records for partitioned datasets (PDS) and ISPF statistics. In addition itcan identify filetypes stored within providing mimetype information. By default files will be converted fromEBCDIC based on their mimetype as determined by libmagic.

After parsing various functions exist to extract one, many or all files and folders (i.e. datasets) contained within.It also provides interfaces to gather file information and provides all metadata as json.

Examples

Load an XMI file and extract all contents:

>>> from xmilib import XMIT>>> obj = XMIT(filename="/path/to/FILE100.XMI")>>> obj.open()>>> obj.set_output_folder("/path/to")>>> obj.unload_files()

Load an AWS file and view metatdata JSON:

4 Chapter 1. Overview

https://docs.python.org/3/library/logging.html#levels



>>> from xmilib import XMIT>>> obj = XMIT(filename="/path/to/FILE420.AWS")>>> obj.open()>>> print(obj.get_json())

Load an XMI file, extract member (aka file) from paritioned dataset (i.e PDS aka folder):

>>> from xmilib import XMIT>>> obj = XMIT(filename="/path/to/FILE720.XMI")>>> obj.parse()>>> obj.unload_file("PDS.IN.XMI", "FILE001")

Load a HET file, extract datasets (PDS or sequential):

>>> from xmilib import XMIT>>> obj = XMIT(filename="/path/to/tapefile01.het")>>> obj.parse()>>> obj.unload_files("PDS.IN.HET")

Get a list of all members from a PDS and get their metadata:

>>> members = obj.get_members("SOME.PDS")>>> for m in members:>>> info = obj.get_member_info("SOME.PDS", m)>>> for k in info:>>> print(k, info[k])

Parameters

• filename (str) – The path and filename of an XMI/AWS/HET file. Defaults to None.

• LRECL (int) – If record length cannot be determined this value is used when convertingfrom EBCDIC to UTF-8. Defaults to 80.

• loglevel (int) – Level of logging, based on https://docs.python.org/3/library/logging.html#levels. Defaults to loggin.WARNING.

• outputfolder (str) – Output file path for extracted files. Detaults to current workingdirectory.

• encoding (str) – EBCDIC codepage used when translating from EBCDIC to UTF-8.Defaults to cp1140

• infile (str) – folder/file name to use for XMI output instead of the detaset name in-cluded in the metadata.

• unnum (bool) – Some mainframe files have numbers in columns 72-80 denoting line num-bers. If True files converted from EBCDIC will have these columns removed. Default toTrue.

• quiet (bool) – Do not print any output messages whle extracting files. Default to False.

• force_convert (bool) – Converts all files utf-8 ignoring mimetype. Defaults to False.

• binary (bool) – Extract files as binaries, ignoring mimetype. Defaults to False.

• modifydate (bool) – If created date/last modified date information is available changethe last modified date of the extracted file to match. Defaults to False.

1.1. xmi Python Library 5




open(infile=None)Loads an XMI/AWS/HET file.

Parameters

• infile (str) – folder/file name to use for output instead of what

• included in the metadata. Only used for XMI files. (is) –

Use either set_filename(filename=) or set_file_object(data=) before using this func-tion if you haven’t passed filename when you initialized this object.

set_overwrite(setting=True)Sets file overwrite. If true extracted files will overwrite existing.

set_modify(setting=True)Sets file modify date for extracting a file. If true extracted files will have their date changed based on theirmetadata, either created date or last modified date if available.

set_quiet(setting=True)Enables/Disables printing when extracting files.

set_filename(filename)Sets the filename used to load the XMI/AWS/HET file

set_xmit_file(filename)Alias of set_filename() with debug statements.

set_tape_file(filename)Alias of set_filename() with debug statements.

set_file_object(data)Used to open already loaded file bytes instead of file name.

set_xmit_object(xmit_data)Used to open already loaded XMI file bytes instead of file name.

set_tape_object(virtual_tape_data)Used to open already loaded AWS/HET file bytes instead of file name.

set_output_folder(outputfolder)Default folder to extract to. Defaults to current working directory.

set_codepage(codepage='cp1140')Sets EBCDIC codepage used for EBCDIC to utf-8 conversion. Defaults to cp1140.

Warning: If this setting changed after a file has been opened/parsed you must reload the file for thechanges to take effect.

set_force(setting=True)When setting=True force all files to be converted to utf-8.


set_binary(setting=True)When setting=True ignore mimetype and do not convert any files to UTF-8.




set_unnum(setting=True)When set to True plaintext files translted to utf-8 will have their number columns removed.

Some files on mainframes have number in columns 72-80, this library removes those columns by defaultwhen converting files. To disable this feature use set_unnum(False).


read_file()Reads the XMI/AWS/HET file but does not parse it.

read_xmit_file()Reads the XMI file but does not parse it.

read_tape_file()Reads the AWS/HET file but does not parse it.

is_xmi(pds, member_name)Returns true if a member is an XMI file.

Parameters

• pds (str) – partioned dataset name

• member_name (str) – pds member

has_xmi()Returns True if this object has opened an XMI file.

has_tape()Returns True if this object has opened an AWS/HET file.

get_file()Returns the name of the first dataset in the XMI/AWS/HET file.

This function is useful when used with XMI files as they can only contain one file/directory.

get_files()Returns a list of all sequential datasets and partitioned dataset (PDS) (i.e. files/folders) contained withinopened XMI/AWS/HET files.

get_last_modified(filename)Returns the last modified date of a dataset. Tape files do not have a last modified so create date (if available)is used instead. The format of the date is string in ISO format.

get_owner()Returns the username of the dataset owner, if available.

get_dataset_size(dsn)Returns the size of a dataset.

get_total_size()Returns the total size of all datasets in the file ignoring metadata.

get_codecs()Returns supported codecs



get_codec()Returns current codec

has_message()Returns true if the XMI file has a message.

get_message()Returns a string containing the XMI file message.

get_num_files()Returns the total number of files (datasets and members).

get_members(pds)Returns an array of all members in the provided partitioned dataset.

get_member_info(pds, member)Returns a dict containing information about the partitioned dataset member.

The returned dict contains:

• mimetype (str): the member mimetype

• extenstion (str): the member extention based on mimetype

• RECFM (str): the member record format

• LRECL (int): the member line/record length

• size (int): size of the member

If ISPF stats are available it also contains:

• modified (str): The last modify date of the file in ISO format

• owner (str): The username of the file owner

• version (str): The current file version

If the member is an alias (i.e. symbolic link) it also contains:

• alias (str): name of the member this alias points to

If the member is an alias all other information is pulled from the member the alias points to.

get_member_info_simple(pds, member)Alias of get_member_info().

get_file_info_simple(filename)Returns a dict containing a small subset of metadata for the dataset.

The returned dict contains:



• modified (str): output from get_last_modified() in ISO format

• size (int): output from get_dataset_size()

• owner (str): output from get_owner()

get_pds_info_simple(pds)Alias of get_file_info_simple().

get_file_info_detailed(filename)Returns a dict with metadata. Currently only supports tape files.



The returned dict contains the following:



• size (int): output from get_dataset_size()

• owner (str): output from get_owner()

It may also contain the following:

• dsnser (str): The serial number of the dataset

• created (str): The date the dataset was created in ISO format

• expires (str): The date the dataset can be removed in ISO format

• syscode (str): The system code of the system that generated this this tape file

• jobid (str): The job id used to move this dataset to this tape

• RECFM (str): the member record format

• LRECL (int): the member line/record length

get_volser()Returns the tape volume serial if available.

get_user_label()Returns all user labels on the tape concatenated together in a string.

get_member_size(pds, member)Returns the size of a partitioned dataset member (int).

get_member_decoded(pds, member)Returns either UTF-8 string or EBCDIC bytes depending on member mimetype.

get_member_binary(pds, member)Returns partitioned dataset member as bytes.

get_member_text(pds, member)Returns paritioned dataset member converted to utf-8 based on current codepage (default is cp1141).

Use set_codepage() to change current code page.

get_file_decoded(filename)Returns either UTF-8 string or EBCDIC bytes depending on dataset mimetype.

get_seq_decoded(pds)Alias of get_file_decoded().

get_file_binary(filename)Returns EBCDIC bytes of dataset.

get_seq_raw(pds)Alias of get_file_binary().

get_file_text(filename)Returns dataset converted to utf-8 based on current codepage (default is cp1141).

Use set_codepage() to change current code page.

is_alias(pds, member)Returns True if the partitioned dataset member is an alias.



is_member(pds, member)Returns true if the member exists in the provided partioned dataset.

is_sequential(pds)Returns true if the dataset is a sequential dataset (i.e. file).

is_file(pds)Alias of is_sequential().

is_pds(pds)Returns true if the dataset is a partitioned dataset (i.e. folder).

get_alias(pds, member)Returns the member name that the alias points to if available.

get_xmi_node_user()Returns a list containing XMI file information around nodes/owners.

The list contains the following strings:

• Originating node name

• Originating user name

• Destination node name

• Destination user name

print_message()If an XMI file has a message prints the message and returns.

get_json(text=False, indent=2)Returns a string containing all available metadata for the file.

Parameters

• text (bool) – If True the metadata also includes the file converted

• utf-8. Default to False. (to) –

• indent (int) – json file indentation, default to 2

get_xmit_json()Alias of get_json().

get_tape_json()Alias of get_json().

dump_xmit_json(json_file_target=None)Extracts all file metadata to filename.json. Where filename is the name of the XMI/AWS/HET file.

Parameters json_file_target (str) – folder where to place json file

_pprint()Prints the XMI/TAPE object using pprint.

_get_clean_json()Returns a dict with binary data from tape/xmi dicts removed and appends class information. For use withget_json()

_get_clean_json_no_text()Returns a dict with utf-8 data from tape/xmi dicts removed. For use with get_json()

filetype_is_xmi(current_file)Determines if a file is an XMI file.



NETDATA files must have the first record header INRM01 which is located after the record size halfwordat the beggening of the file.

filetype_is_tape(current_file)Determines if a file is a virtual tape (AWS/HET) file.

Virtual tape files begin with a six byte header which contains three halfwords: 1 the number of bytes inthis block 2 the number of bytes in the previous block 3 end of file flag

To check if the file is a virtual tape file it confirms that the first record previous bytes header is zero as therecannot be any previous bytes at the beggining of the file.

print_details(human=True)Prints a subset of available metadata for all datasets/members included in the file.

Parameters human (bool) – Is True converts file sizes to human readable. Default is True.

print_xmit(human=True)Alias of print_details().

print_tape(human=True)Alias of print_details().

unload_files()Extracts all datasets and members to output folder and appends file extensions based on mimetype.

If there are partitioned datasets folders will be created based on the dataset name and all members will beplaced in that folder.

Output folder can be changed with set_outputfolder(), default is current working directory.

unload_xmit()Alias of unload_files().

unload_tape()Alias of unload_files().

extract_all()Alias of unload_files().

unload_pds(pds)Extracts all the dataset or members to output folder and appends file extensions based on mimetype.

If the dataset is a partitioned dataset a folder will be created and all members will be placed in that sub-folder.


unload_folder(pds)Alias of unload_pds().

extract_pds(pds)Alias of unload_pds()

unload_file(filename, member=None)Extracts one file to output folder.

Parameters

• filename (str) – dataset to extract, required.

• member (str) – optional member name




extract_dataset(dataset)Alias of unload_file().

sizeof_fmt(num)Returns human friendly size of int.

convert_text_file(ebcdic_text, recl)Converts EBCDIC files to utf-8.

Parameters

• ebcdic_text (*) – EBCDIC data to convert

• lrecl (*) – How long each line is

Why lrecl? Mainframe fixed length (RECFM = F) files have no concept of line terminators, each line is theexact same length padded to LRECL with spaces. Therefore, if ebcdic_text is 160 bytes and an LRECLof 80 there’s two lines in the provided data. Some files can have variable length (RECFM = V). The onlychange is that each line has its own record length, but it will still be padded with spaces.

Some mainframe files may have a numbers column (columns 72-80 have numbers in them). Bydefault those columns are removed if they only contain numbers. To disable this feature useset_unnum(False).

convert_message()Converts XMI file message to utf-8

get_dsorg(dsorg)Returns a string of the dataset organization (DSORG).

DSORG contains file layout information. Typically datasets will be either “PS” or “PO”. PS datasets aresequential datasets (i.e. a single file), PO dataset are partitioned datasets (i.e. folders).

get_recfm(recfm)Returns a string of the dataset record format (RECFM).

RECFM contains file layout information and is cummulative.

The first letter is one of F, V, U where:

• F = fixed length records

• V = Variable length records

• U = Unknown

The additional letters may be:

• B = blocked

• A = ANSI control characters (for printers)

• M = machine control characters (for printers)

• S = standard blocks

For more information see DS1RECFM in https://www.ibm.com/support/knowledgecenter/SSLTBW_2.3.0/com.ibm.zos.v2r3.idas300/s3013.htm

check_parsed()Raises an exception if no XMI/AWS/HET has been opened.

make_int(num)Converts string to integer, mostly used in tape labels.


https://www.ibm.com/support/knowledgecenter/SSLTBW_2.3.0/com.ibm.zos.v2r3.idas300/s3013.htm

https://www.ibm.com/support/knowledgecenter/SSLTBW_2.3.0/com.ibm.zos.v2r3.idas300/s3013.htm


ispf_date(ispfdate, seconds=0)Converts ISPF date to ISO format string with microseconds.

change_outfile_date(outfile, date)Modifies extracted files created/last modified date to match metadata

Parameters

• outfile (str) – path to file

• date (str) – ISO format date

parse_xmi()Parses an XMI file collecting metadata and files and stores them in the object.

NETDATA files (otherwise known as XMI files) are composed of control records which contain metadataand dataset(s).

Control Records:

• INMR01 - Header records

• INMR02 - File control record(s)

• INMR03 - Data control record(s)

• INMR04 - User control record

• INMR06 - Final record

• INMR07 - Notification record

This library only processes INMR01, INRM02, INMR03, INMR04, and INMR06 records. INMR07records are notification records and do not contain any files.

INMR records are composed of the name (INMR01, etc) followed by IBM text units which containsmetadata about the record.

get_xmi_files()Extracts files from a parsed XMI file and stores them in the XMIT object

parse_INMR01(inmr01_record)Parses INMR01 records

INRM01 records always contain the following text units:

• INMFTIME - date/time the XMI was created

• INMLRECL - Record length for this XMI

• INMFNODE - name of the originating system

• INMTNODE - name of the target system

• INMFUID - userid of the person who created the XMI

• INMTUID - userid of the user this XMI is being sent to

The following text units are optional:

• INMFACK - notification receipt

• INMFVERS - version number

• INMNUMF - number of files

• INMUSERP - user options



XMI files on the mainframe are generated with the TSO TRANSMIT command, when an output file is notsupplied the XMI file will be sent using network job entry (NJE) to the target userid and node. If an outputfile is provided the TRANSMIT command still puts a target user and target node in the XMI file.

parse_INMR02(inmr02_record)Parses INRM02 control records.

An XMI file may contain multiple INMR02 control records. These records always contains the followingtext units:

• INMDSORG - dataset organization

• INMLRECL - Record length

• INMSIZE - size in bytes

• INMUTILN - Utility program

Optional text units are:

• INMDSNAM - dataset name (messages do not have this text unit)

• INMCREAT - the date the file was created

There are multiple other optional text units which can be read here: https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.ikjb400/inmr02.htm

The utility program defines how the file was generated and it can be INMCOPY, IEBCOPY, and AMSCI-PHR:

• INMCOPY - converts a sequential dataset (file) for XMI

• IEBCOPY - converts a partitioned dataset (folder) for XMI

• AMSCIPHR - encrypts the files in XMI, this library does not support extracting encrypted files.

Depending on the dataset type the XMI may contain multiple records. The process is:

• If the dataset is sequential - INMCOPY

• If it is a partitioned dataset - IEBCOPY -> INMCOPY

Therefore, partitioned datasets will have two INMR02 records.

parse_INMR03(inmr03_record)Parses INMR03 records

Defines the file format and contains the following text units:


• INMLRECL - dataset record length

• INMRECFM - dataset record format

• INMSIZE - size of the dataset in bytes

parse_INMR04(inmr04_record)Print debug message for INMR04 records.

INMR04 records are used to pass data to instalation specific exits (i.e. APIs), this function is provided ifneeded to be overloaded.

parse_tape()Parses a virtual tape file (AWS or HET) collecting metadata and files and stores them in the object.

Virtual tapes are broken down as follows:


https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.ikjb400/inmr02.htm



• Header (3 bytes)

– Current block size (little endian)

– Previous block size (little endian)

– Flag:

* 0x2000 ENDREC - End of record

* 0x4000 EOF - tape mark

* 0x8000 NEWREC - Start of new record

* HET file flags can also contain compression flags:

· 0x02 BZIP2 compression

· 0x01 ZLIB compression

• Labels (optional):

– VOL1 (80 bytes)

* Volume serial number

* Tape owner

* More information: https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.idam300/formds1.htm

– HDR1 (80 bytes):

* Dataset name

* Dataset serial number

* Volume sequence number

* Dataset sequence number

* Generation number

* Version number

* Created date

* Expiration date

* System code (i.e. what OS version)

* More information: https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.idam300/formds2.htm

– HDR2 (80 bytes)

* Record format

* Block length

* Tape density

* Position

* Job name and step used to copy files to this tape

* Tape recording technique

* Control character, used for printing

* Block attribute


https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.idam300/formds1.htm





* Device serial number

* Security flag

* Large block length

* More information: https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.idam300/dshead.htm

– UHL1 - UHL8: (80 bytes):

* Contains user headers 76 bytes long

* More info here: https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.idam300/m3208.htm

get_tape_files()Extracts files from a parsed AWS/HET file and stores them in the XMIT object

get_tape_date(tape_date)Converts tape label date to ISO format string with microseconds.

iebcopy_record_1(first_record)Returns a dict containing IEBCOPY COPYR1 metatdata

More information available here: https://www.ibm.com/support/knowledgecenter/SSLTBW_2.2.0/com.ibm.zos.v2r2.idau100/u1322.htm

iebcopy_record_2(second_record)Returns a dict containing IEBCOPY COPYR2 metatdata

More information available here: https://www.ibm.com/support/knowledgecenter/SSLTBW_2.2.0/com.ibm.zos.v2r2.idau100/u1322.htm

__all_members(members)If all members in a pds have been processed returns True, otherwise False.

__get_members_info(directory)Returns a dict containing metadata and filenames for all members contained in a PDS

__process_blocks(filename, member_blocks=b'')Processes partitioned dataset directory blocks, returns dict of members and file data/metadata.

Parameters

• filename (*) – partitioned dataset name

• member_blocks (*) – binary PDS file data

__get_file_mimetype_and_convert(file_name, file_data, vb_file_data=None, recfm='F',lrecl=80)

Guesses file extension based on mimetype and converts to utf-8 if file is plain/ebcdic

Parameters

• member_name (str) – member name

• file_data (bytes, str or list) – the file to get mimetype for

• vb_file_data (list) – variable block data split in to records

• recfm (str) – the record formart (used to convert variable length)

• lrecl (int) – record length


https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.idam300/dshead.htm


https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.idam300/m3208.htm


https://www.ibm.com/support/knowledgecenter/SSLTBW_2.2.0/com.ibm.zos.v2r2.idau100/u1322.htm





Returns a dict with: mimetype (str): file guessed mimetype from libmagic datatype (str): either binary,text, etc extension (str): file extention with period (i.e. “.txt”) data (byte): binary file data text (str): Isincluded when force_text is enabled or when the file mimetype is determined to be a plain/text file.

__is_jcl(text_lines='')Returns true if the first line starts with // and contains JOB

__is_rexx(text_lines='')Returns true if the first line starts with /* and contains REXX

__fix_circular_alias(pds)Some XMI files have circular aliases, this function updates the XMIT object to fix them.

__member_ttrs(pds)Returns a dict of TTRs to members.

__get_ttr(pds, member)Returns the TTR for a given pds and member.

__text_units(text_records)Parses IBM text units from XMI control records, returns a dict with text unit name and value.

Text units in INMR## records are broken down like this:

• First two bytes are the ‘key’/type

• Second two bytes are how many text unit records there are

• Then records are broken down by size (two bytes) and the data

• Data can be string, int or hex

1.2 Installation

To install this script use:

python3 -m pip install xmi

xmi requires the following external libraries:

• ebcdic: used for EBCDIC to utf-8 translation

• python-magic: Used to guess dataset/member file mimetype

• prettytable: used to create printable output tables

1.3 Cookbook

Open and extract (unload) XMI/AWS/HET mainframe files.

The most simple way to use this library is to import this module and use xmi.open_file() to open an XMI, AWS,or HET file:

import xmixmi_obj = xmi.open_file("/path/to/file.xmi")het_obj = xmi.open_file("/path/to/file.het")aws_obj = xmi.open_file("/path/to/file.aws")

1.2. Installation 17


To list all datasets and dataset members:

for f in het_obj.get_files():if het_obj.is_pds(f):

for m in het_obj.get_members(f):print("{}({})".format(f, m))

else:print(f)

Print JSON metatdata:

print(xmi_obj.get_json())print(het_obj.get_json(text=True)) # Adds plaintext files to json outputprint(aws_obj.get_json(indent=6)) # Increases the json indent

Silently extract all files/folders to /tmp/xmi_files/:

aws_obj.set_output_folder("/tmp/xmi_files/")aws_obj.set_quiet(True)aws_obj.extract_all()

Print detailed file information:

xmi_obj.print_details()xmi_obj.print_xmit() # Same output as previous, print_xmit() is an alias to print_→˓details()het_obj.print_tape() # print_tape() is an alias to print_details()aws_obj.print_tape(human=True) # Converts size to human readable

Print message:

if xmi_obj.has_message():print(xmi_obj.get_message())

or just:

print(xmi_obj.get_message()) # Prints 'None' if no message

If you you’re having problems with the library or want to see whats happening behind the scenes you can enabledebugging:

import loggingimport xmi

xmi_obj = xmi.XMIT(filename="/path/to/file.xmi",loglevel=logging.DEBUG)xmi_obj.open()

As you can see, using this library is fairly easy.



1.4 Understanding XMI Files

1.4.1 NETDATA/XMIT

NETDATA files are primarily used to transfer sequential and partitioned datasets between mainframe environments,sometimes via non mainframe environments. NETDATA is the official name for the file format of the output from thez/OS TRANSMIT, z/VM NETDATA or the opensource tool XMIT370. However, it is more often reffered to as an XMIfile. This documentation uses XMI and NETDATA interchangably. Typically third parties and even IBM will refer tothem as XMI files.

Note: Some quick terminology:

• Dataset: a mainframe file, usually refered to as a sequential dataset or seq

• Partitioned dataset: a mainframe folder usually refered to as a PDS

• Member: files in a partitioned dataset

• Unload: extracting data to be used elsewhere

• LRECL: The record length. This is how long each line in a file is, padded with spaces.

• RECFM: Record format, where:

– The first letter is one of F, V, U where:

* F = fixed length records

* V = Variable length records

* U = Unknown

– And additional letters may be:

* B = blocked

* A = ANSI control characters (for printers)

* M = machine control characters (for printers)

* S = standard blocks

XMI files contain either a sequential dataset or a partitioned dataset, and optionally a message. They cannot containmore than one dataset, paritioned or sequential at a time. They can, however, also include an optional message whichis technically sequential dataset, however the dataset name is lost.

Sequential datasets are ‘unloaded’ by XMIT using the program INMCOPY whereas partitioned datasets are unloadedusing a program called IEBCOPY.

Think about XMI files as tar files on Linux but only if you could add one file or one folder to the tar file. OftentimesXMI files will contain nested XMI files due to this limitation.

XMI files are commonly used by IBM, Broadcom, and many other mainframe vendors to send files to customers.There’s also a large collection of software and programs made available for free using XMI by the amazing CBTTAPEproject available at http://cbttape.org/cbtdowns.htm.

1.4. Understanding XMI Files 19

http://cbttape.org/cbtdowns.htm


1.4.2 Creating XMI Files

To create a XMI file on z/OS you use the TSO program XMIT/TRANSMIT:

XMIT NODE.USER DATASET('DATASET.TO.SEND') OUTDATASET('OUTPUT.FROM.XMIT.XMI')

TRANSMIT NODE.USER DATASET('DATASET.TO.SEND') OUTDATASET('OUTPUT.FROM.XMIT.XMI')

You can also add a message to XMI files:

XMIT NODE.USER DATASET('DATASET.TO.SEND') OUTDATASET('OUTPUT.FROM.XMIT.XMI')→˓MSGDATASET('SEQ.MSG.FILE')

If you are using TK4- you can use XMIT370 and some JCL to generate XMI files:

//XMIMAKE JOB (01),'COPY TO TAPE',CLASS=H,MSGCLASS=H,NOTIFY=HERC01//* ------------------------------------------------------------------//* CREATES XMILIB TEST XMIT FILES//* ------------------------------------------------------------------//* EXAMPLE 1: STEP XMITSEQ//* CREATES THE XMI FILE PYTHON.XMI.SEQ.XMIT FROM THE//* SEQUENTIAL DATASET PYTHON.XMI.SEQ//XMITSEQ EXEC PGM=XMIT370//XMITLOG DD SYSOUT=*//SYSPRINT DD SYSOUT=*//SYSUDUMP DD SYSOUT=*//COPYR1 DD DUMMY//SYSIN DD DUMMY//SYSUT1 DD DSN=PYTHON.XMI.SEQ,DISP=SHR//SYSUT2 DD DSN=&&SYSUT2,UNIT=3390,// SPACE=(TRK,(255,255)),// DISP=(NEW,DELETE,DELETE)//XMITOUT DD DSN=PYTHON.XMI.SEQ.XMIT,DISP=(,CATLG,DELETE),// UNIT=3350,VOL=SER=KICKS,SPACE=(TRK,(50,50))//* EXAMPLE 2: STEP XMIPDS//* CREATES THE XMI FILE PYTHON.XMI.PDS.XMIT FROM THE//* PARTITIONED DATASET PYTHON.XMI.PDS//XMIPDS EXEC PGM=XMIT370//XMITLOG DD SYSOUT=*//SYSPRINT DD SYSOUT=*//SYSUDUMP DD SYSOUT=*//COPYR1 DD DUMMY//SYSIN DD DUMMY//SYSUT1 DD DSN=PYTHON.XMI.PDS,DISP=SHR//SYSUT2 DD DSN=&&SYSUT2,UNIT=3390,// SPACE=(TRK,(255,255)),// DISP=(NEW,DELETE,DELETE)//XMITOUT DD DSN=PYTHON.XMI.PDS.XMIT,DISP=(,CATLG,DELETE),// UNIT=3350,VOL=SER=KICKS,SPACE=(TRK,(50,50))

I’ll leave generating XMI files on z/VM up to the reader.



1.4.3 Transferring XMI files

XMI files (as with most mainframe files) are in EBCDIC, therefore to download the XMI file from the mainframe youwill need to use FTP in binary file transfer mode. Fortunately enabling binary on FTP is simple, just issue the FTPcommand binary once connected and transfer the XMI file to your machine.

1.4.4 File Structure

XMI files are composed of control records which contain metadata and dataset information.

Control Records:

• INMR01 - Header records

• INMR02 - File control record(s)

• INMR03 - Data control record(s)

• INMR04 - User control record

• INMR06 - Final record

• INMR07 - Notification record

This library only processes INMR01, INRM02, INMR03, INMR04, and INMR06 records. INMR07 records arenotification records and do not contain any files.

INMR records are composed of the name, two digit number (INMR01, etc) followed by IBM text units which containsmetadata about the record.

Text units in INMR## records are broken down like this:

• First two bytes are the ‘key’/type

• Second two bytes are how many text unit records there are

• Then records are broken down by size (two bytes) and the data

• Data can be string, int or hex

More information about text units is available here: https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.ikjb400/txunit.htm

1.4.5 INRM01 Records

INRM01 records always contain the following text units:

• INMFTIME - date/time the XMI was created

• INMLRECL - Record length for this XMI

• INMFNODE - name of the originating system

• INMTNODE - name of the target system

• INMFUID - userid of the person who created the XMI

• INMTUID - userid of the user this XMI is being sent to

The following text units are optional:

• INMFACK - notification receipt

• INMFVERS - version number


https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.ikjb400/txunit.htm

https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.ikjb400/txunit.htm


• INMNUMF - number of files

• INMUSERP - user options


An XMI file may contain multiple INMR02 control records. These records always contains the following text units:


• INMLRECL - Record length

• INMSIZE - size in bytes

• INMUTILN - Utility program

Optional text units are:

• INMDSNAM - dataset name (messages do not have this text unit)

• INMCREAT - the date the file was created

There are multiple other optional text units which can be read here: https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.ikjb400/inmr02.htm

The utility program (INMUTILN) defines how the file was generated and it can be INMCOPY, or IEBCOPY, andAMSCIPHR:

• INMCOPY - converts a sequential dataset (file) for XMI

• IEBCOPY - converts a partitioned dataset (folder) for XMI

• AMSCIPHR - encrypts the files in XMI, this library does not support extracting encrypted files.

Depending on the dataset type the XMI may contain multiple INMR02 records. The process used when generating anXMI file is:

• If the dataset is sequential - INMCOPY -> Stop

• If it is a partitioned dataset - IEBCOPY -> INMCOPY -> Stop

• If there’s also a message the first INMR02 record is INMCOPY and doesn’t have a dataset name (INMDSNAM).

Therefore, partitioned datasets will have two or more INMR02 records.


Defines the file format and contains the following text units:


• INMLRECL - dataset record length

• INMRECFM - dataset record format

• INMSIZE - size of the dataset in bytes






INMR04 records are used to pass data to instalation specific exits (i.e. APIs).

1.4.9 Metadata

Let’s take a look at the file test_pds_msg.xmi (generated with XMIT on TSO) in the tests folder. Using thislibrary we can extract the XMI metadata as json:

{"INMR01": {

"INMLRECL": 80,"INMFNODE": "SMOG","INMFUID": "PHIL","INMTNODE": "XMIT","INMTUID": "PHIL","INMFTIME": "2021-03-09T05:14:41.000000","INMNUMF": 2

},"INMR02": {

"1": {"INMUTILN": "INMCOPY","INMSIZE": 58786,"INMDSORG": "PS","INMLRECL": 251,"INMBLKSZ": 3120,"INMRECFM": "VB","numfile": 1

},"2": {

"INMUTILN": "IEBCOPY","INMSIZE": 176358,"INMDSORG": "PO","INMTYPE": "None","INMLRECL": 80,"INMBLKSZ": 27920,"INMRECFM": "FB","INMDIR": 6,"INMDSNAM": "PYTHON.XMI.PDS","numfile": 2

},"3": {

"INMUTILN": "INMCOPY","INMSIZE": 176358,"INMDSORG": "PS","INMLRECL": 32756,"INMBLKSZ": 3120,"INMRECFM": "VS","numfile": 2

}},"INMR03": {

"1": {"INMSIZE": 176358,"INMDSORG": "PS","INMLRECL": 80,

(continues on next page)



(continued from previous page)

"INMRECFM": "?"},"2": {

"INMSIZE": 176358,"INMDSORG": "PS","INMLRECL": 80,"INMRECFM": "?"

}}}

Notice that test_pds_msg.xmi had a message, hence there being three INMR02 records. And since it was a PDSit contains the records, IEBCOPY and another for INMCOPY.

Now lets look at the sequential dataset test_seq.xmi in the tests folder. This XMI file was generated withXMIT370.

{"INMR01": {

"INMLRECL": 80,"INMFNODE": "ORIGNODE","INMFUID": "ORIGUID","INMTNODE": "DESTNODE","INMTUID": "DESTUID","INMFTIME": "2021-03-09T04:53:18.000000","INMNUMF": 1

},"INMR02": {

"1": {"INMUTILN": "INMCOPY","INMSIZE": 0,"INMDSORG": "PS","INMLRECL": 80,"INMBLKSZ": 3200,"INMRECFM": "FB","numfile": 1,

}},"INMR03": {

"1": {"INMSIZE": 0,"INMDSORG": "PS","INMLRECL": 80,"INMRECFM": "?"

}}}

Notice how there is only one INMR02 record. Also notice that XMIT370 omits the INMDSNAM text unit for sequen-tial files.



1.4.10 The File Contents XMI

After parsing the control records the actual file contents follow. If the file is a sequential dataset its easy enough todetect the mime type using file and extract its content. If the file is a PDS then that means it was “unloaded” usingIEBCOPY which is a little more complicated.

1.5 Understanding Virtual Tape Files

1.5.1 AWSTAPE

The AWSTAPE file format is used to transfer virtual tape files. Originally created for P/390 it is used primarily todaywith virtual tape offerings. AWS is the short name for these tape file types.

Virtual tape files are fairly simple in design, they contain a 6 bytes header which contains info on how long the currentrecord is, how long the previous record was and a flag, followed by EBCDIC data.

1.5.2 Hercules Emulated Tape (HET)

Later the opensource project Hercules created the Hercules Emulated Tape, or HET, which builds on the AWSTAPEformat by adding compression using either Bzip or ZLIB.

1.5.3 AWS and HET Format

Each label/dataset stored on a virtual tape is preceded by a header record 6 bytes long made up of the following:

• Current block size (short, little endian)

• Previous block size (short, little endian)

• Flag (2 bytes):

– 0x2000 ENDREC - End of record

– 0x4000 EOF - tape mark

– 0x8000 NEWREC - Start of new record

– HET file flags can also contain compression flags:

* 0x02 BZIP2 compression

* 0x01 ZLIB compression

Following the header record is data. On some tapes (not all) optional label records can exist. These records identifymetatdata about the dataset(s) on the tape. Each label starts with 3 characters and a number and are 80 bytes long. InHET files labels are compressed based on the flag.

• VOL1 label (80 bytes)

– Volume serial number

– Tape owner

– More information: https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.idam300/formds1.htm

• HDR1 label (80 bytes):

– Dataset name

1.5. Understanding Virtual Tape Files 25




– Dataset serial number

– Volume sequence number

– Dataset sequence number

– Generation number

– Version number

– Created date

– Expiration date

– System code (i.e. what OS version)

– More information: https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.idam300/formds2.htm

• HDR2 label (80 bytes)

– Record format

– Block length

– Tape density

– Position

– Job name and step used to copy files to this tape

– Tape recording technique

– Control character, used for printing

– Block attribute

– Device serial number

– Security flag

– Large block length

– More information: https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.idam300/dshead.htm

• UHL1 - UHL8 label (80 bytes):

– Contains user headers 76 bytes long

– More info here: https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.idam300/m3208.htm

1.5.4 Metadata Virtual Tape

So what does this look like on actual files? Using this library we can export the metadata from the virtual tape filetest_tape.aws in the tests folder:

{"file": {

"PYTHON.XMI.SEQ": {"HDR1": {

"dsn": "PYTHON.XMI.SEQ","dsnser": "XMILIB","volseq": 1,











"dsnseq": 1,"gennum": 0,"version": 0,"createdate": "1921-03-09T00:00:00.000000","expirationdate": "1900-01-01T00:00:00.000000","dsnsec": false,"block_count_low": 0,"system_code": "IBM OS/VS 370","block_count_high": 0

},"HDR2": {

"recfm": "F","block_len": 3200,"lrecl": 80,"density": 4,"position": "0","jobid": "XMITAPE /COPYPS ","technique": " ","control_char": " ","block_attr": "B","devser": " 30001","dsnid": " ","large_block_len": " "

}},"PYTHON.XMI.PDS": {"HDR1": {

"dsn": "PYTHON.XMI.PDS","dsnser": "XMILIB","volseq": 1,"dsnseq": 2,"gennum": 0,"version": 0,"createdate": "1921-03-09T00:00:00.000000","expirationdate": "1900-01-01T00:00:00.000000","dsnsec": false,"block_count_low": 0,"system_code": "IBM OS/VS 370","block_count_high": 0

},"HDR2": {

"recfm": "V","block_len": 3220,"lrecl": 3216,"density": 4,"position": "0","jobid": "XMITAPE /COPYPO ","technique": " ","control_char": " ","block_attr": "S","devser": " 30001","dsnid": " ","large_block_len": " "

}}

}

1.5. Understanding Virtual Tape Files 27


The JCL used to move these two datasets to tape was

//XMITAPE JOB (01),'COPY TO TAPE',CLASS=A,MSGCLASS=H,NOTIFY=HERC01//* THIS JOB COPIES THE TEST FILES FOR XMILIB TO TAPES//* USE HETINIT TO GENERATE THE TAPES//* hetinit -d test_tape.aws XMILIB//* THEN SUBMIT THIS JOB//* AND ENTER: /devinit 480 test_tape.aws IN THE HERCULES CONSOLE//COPYPS EXEC PGM=IEBGENER,REGION=562K//SYSPRINT DD SYSOUT=*//SYSUT2 DD UNIT=TAPE,DISP=NEW,DSN=PYTHON.XMI.SEQ,// VOL=SER=XMILIB,LABEL=(01,SL)//SYSUT1 DD DSN=PYTHON.XMI.SEQ,DISP=SHR//SYSIN DD DUMMY//COPYPO EXEC PGM=IEBCOPY,REGION=562K//SYSPRINT DD SYSOUT=*//TAPE DD UNIT=TAPE,DISP=NEW,DSN=PYTHON.XMI.PDS,// VOL=SER=XMILIB,LABEL=(02,SL)//PDS DD DSN=PYTHON.XMI.PDS,DISP=SHR//SYSUT3 DD UNIT=SYSDA,SPACE=(80,(60,45)),DISP=(NEW,DELETE)//SYSIN DD *COPY INDD=PDS,OUTDD=TAPE/*//

1.5.5 The File Contents AWS/HET

After parsing the header records and any labels the actual file contents follow. If the file is a sequential dataset itseasy enough to detect the mime type using file and extract its content. If the file is a PDS then that means it was“unloaded” using IEBCOPY which is a little more complicated.

1.6 IEBCOPY File Format

IEBCOPY is an IBM program with multiple uses, but for the sake of this library we will talk about how it producesPDS unloads and their various controls records as well as the directory structure.

IEBCOPY is used to unload partitioned datasets to a tape, vitual tape, xmi, file etc. Its format is made up of controlsrecords (COPYR1 and COPYR2), a listing of members and their metadata followed by the PDS members file data.

1.6.1 COPYR1 Control Record

This record contains information about the file itself. It contains multiple items that can be used rebuild the PDS.

The COPYR1 record is 64 bytes long. Skipping the first 8 bytes this record has the eye catcher of 0xCA6D0F whichidentifies it as a COPYR1 record.

After the eye catcher is the following information:

• DS1DSORG - Dataset organization

• DS1BLKL - Block size

• DS1LRECL - Record length for all members

• DS1RECFM - Record format



• DS1REFD - Date last referenced

Multiple other fields are available in the COPYR1 record. Please refer to the following for more details: https://www.ibm.com/support/knowledgecenter/SSLTBW_2.2.0/com.ibm.zos.v2r2.idau100/u1322.htm

1.6.2 COPYR2 Control Record

Immediately following the COPYR1 control record is the COPYR2 record. This record contains information aboutthe Data Extent Block (DEB) for the original dataset. This information is collected by this library but largely unused.

1.6.3 Directory Information

Following the control records is the directory information. This section is vairable length and can span multiple blocks.It contains the member names and metadata about the member. This section begins with:

00 00 00 00 00 00 00 00

or for PDSE:

08 00 00 00 00 00 00 00

and the key length, the length of the directory and the name of the last opened member.

After the header multiple entries exist, one for each member (a member is essentially a file) in this PDS. Each entrycontains the following:

• Member name

• TTR, a pointer to the data for this member

• Number of notes attached to this member

• Alias flag, if enabled it means this member is an alias to another member

On top of this information optional information may be stored in the parameters field. This information is called “ISPFstats” since it is used mostly in ISPF when viewing and editing files. It can contain the following:

• The file version, which can be automatically incremented by ISPF

• Created date

• Last modified date (down to the microsecond)

• How many line the original file had

• How many lines have been added

• How many lines have been modified

• The owner of the file

1.6. IEBCOPY File Format 29




1.6.4 Member Data

After the directory block is the member data which can be broken down as:

• Flag (1 byte)

• Original “extent”, labeled as ‘M’ by IBM, (1 byte)

• Binary number (2 bytes)

• TTR

• Data length

Followed by the member data.

More information about member data here: https://www.ibm.com/support/knowledgecenter/SSLTBW_2.2.0/com.ibm.zos.v2r2.idau100/u1327.htm

1.6.5 Metadata IEBCOPY

So what does this all look like on actual PDS? Using this library we can export the metadata from the XMI filetest_pds_msg.xmi in the tests folder:

{"PYTHON.XMI.PDS": {"COPYR1": {

"type": "PDS","DS1DSORG": 512,"DS1BLKL": 27920,"DS1LRECL": 80,"DS1RECFM": "FB","DS1KEYL": 0,"DS1OPTCD": 32,"DS1SMSFG": 0,"file_tape_blocksize": 3120,"DVAOPTS": 12336,"DVACLASS": 32,"DVAUNIT": 15,"DVAMAXRC": 32760,"DVACYL": 10017,"DVATRK": 15,"DVATRKLN": 58786,"DVAOVHD": 0,"num_header_records": 2,"DS1REFD": "210067","DS1SCEXT": "b'\\x80m\\x10'","DS1SCALO": "b'P\\x00\\x00\\x02'","DS1LSTAR": "b'\\x00\\x02\\x02'","DS1TRBAL": "b'\\x9f>'"

},"COPYR2": {

"deb": "b'\\x01\\x00\\x00\\x00\\xff\\x00\\x00\\x00\\x8f\\x08\\x80\\x00\\x04\\x8b\\→˓x00'",

"extents": ["b'\\x01\\x00\\x00\\x00\\xff\\x00\\x00\\x00\\x8f\\x08\\x80\\x00\\x04\\x8b\\x00'","b'X\\xf4\\xe8X\\x00\\x00\\x01\\x0e\\x00\\x0b\\x01\\x0f\\x00\\x01\\x00\\x06'","b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\

→˓x00'",







"b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\→˓x00'",












"b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\→˓x00'"

]},"members": {

"TESTING": {"ttr": 8,"alias": false,"halfwords": 30,"notes": 0,"parms": "b'\\x01\\x00\\x00)\\x01!\\x06\\x7f\\x01!\\x06\\x7f\"S\\x00\\x02\\

→˓x00\\x02\\x00\\x00\\xd7\\xc8\\xc9\\xd3@@@@@@'","ispf": {

"version": "01.00","flags": 0,"createdate": "2021-03-08T00:00:00.000000","modifydate": "2021-03-08T22:53:29.000000","lines": 2,"newlines": 2,"modlines": 0,"user": "PHIL"

},},"Z15IMG": {

"ttr": 10,"alias": false,"halfwords": 0,"notes": 0,"parms": "b''","ispf": false,

}}

1.6. IEBCOPY File Format 31


Note: The actual raw member data has been omitted from this JSON output.

Notice that this file has a record format of Fixed Block (FB) and each line is 80 characters long. Also you can see thatthe first member TESTING contains ISPF information whereas the second file Z15IMG does not.

1.7 Support

Issues and help with this library should use Github Issues at https://github.com/mainframed/xmi/issues). If you’re go-ing to report an issue please include a debug log which can be enabled by passing logging.DEBUG to the loggingargument upon initialization:

import xmiimport logging

f = xmi.open_file(filename="/path/to/file.xmi", logging=logging.DEBUG)

or

import xmiimport loggingf = xmi.XMIT(filename="/path/to/file.xmi", logging=logging.DEBUG)f.open()


https://github.com/mainframed/xmi/issues

CHAPTER

TWO

INDICES AND TABLES

• genindex

• modindex

• search

33


34 Chapter 2. Indices and tables

PYTHON MODULE INDEX

xxmi, 3

35


36 Python Module Index

INDEX

Symbols__all_members() (xmi.XMIT method), 16__fix_circular_alias() (xmi.XMIT method), 17__get_file_mimetype_and_convert()

(xmi.XMIT method), 16__get_members_info() (xmi.XMIT method), 16__get_ttr() (xmi.XMIT method), 17__is_jcl() (xmi.XMIT method), 17__is_rexx() (xmi.XMIT method), 17__member_ttrs() (xmi.XMIT method), 17__process_blocks() (xmi.XMIT method), 16__text_units() (xmi.XMIT method), 17_get_clean_json() (xmi.XMIT method), 10_get_clean_json_no_text() (xmi.XMIT

method), 10_pprint() (xmi.XMIT method), 10

Cchange_outfile_date() (xmi.XMIT method), 13check_parsed() (xmi.XMIT method), 12convert_ebcdic() (in module xmi), 3convert_message() (xmi.XMIT method), 12convert_text_file() (xmi.XMIT method), 12

Ddump_xmit_json() (xmi.XMIT method), 10

Eextract_all() (in module xmi), 3extract_all() (xmi.XMIT method), 11extract_dataset() (xmi.XMIT method), 11extract_pds() (xmi.XMIT method), 11

Ffiletype_is_tape() (xmi.XMIT method), 11filetype_is_xmi() (xmi.XMIT method), 10

Gget_alias() (xmi.XMIT method), 10get_codec() (xmi.XMIT method), 8get_codecs() (xmi.XMIT method), 7

get_dataset_size() (xmi.XMIT method), 7get_dsorg() (xmi.XMIT method), 12get_file() (xmi.XMIT method), 7get_file_binary() (xmi.XMIT method), 9get_file_decoded() (xmi.XMIT method), 9get_file_info_detailed() (xmi.XMIT method),

8get_file_info_simple() (xmi.XMIT method), 8get_file_text() (xmi.XMIT method), 9get_files() (xmi.XMIT method), 7get_json() (xmi.XMIT method), 10get_last_modified() (xmi.XMIT method), 7get_member_binary() (xmi.XMIT method), 9get_member_decoded() (xmi.XMIT method), 9get_member_info() (xmi.XMIT method), 8get_member_info_simple() (xmi.XMIT method),

8get_member_size() (xmi.XMIT method), 9get_member_text() (xmi.XMIT method), 9get_members() (xmi.XMIT method), 8get_message() (xmi.XMIT method), 8get_num_files() (xmi.XMIT method), 8get_owner() (xmi.XMIT method), 7get_pds_info_simple() (xmi.XMIT method), 8get_recfm() (xmi.XMIT method), 12get_seq_decoded() (xmi.XMIT method), 9get_seq_raw() (xmi.XMIT method), 9get_tape_date() (xmi.XMIT method), 16get_tape_files() (xmi.XMIT method), 16get_tape_json() (xmi.XMIT method), 10get_total_size() (xmi.XMIT method), 7get_user_label() (xmi.XMIT method), 9get_volser() (xmi.XMIT method), 9get_xmi_files() (xmi.XMIT method), 13get_xmi_node_user() (xmi.XMIT method), 10get_xmit_json() (xmi.XMIT method), 10

Hhas_message() (xmi.XMIT method), 8has_tape() (xmi.XMIT method), 7has_xmi() (xmi.XMIT method), 7

37


Iiebcopy_record_1() (xmi.XMIT method), 16iebcopy_record_2() (xmi.XMIT method), 16is_alias() (xmi.XMIT method), 9is_file() (xmi.XMIT method), 10is_member() (xmi.XMIT method), 9is_pds() (xmi.XMIT method), 10is_sequential() (xmi.XMIT method), 10is_xmi() (xmi.XMIT method), 7ispf_date() (xmi.XMIT method), 12

Llist_all() (in module xmi), 3

Mmake_int() (xmi.XMIT method), 12module

xmi, 3

Oopen() (xmi.XMIT method), 5open_file() (in module xmi), 3

Pparse_INMR01() (xmi.XMIT method), 13parse_INMR02() (xmi.XMIT method), 14parse_INMR03() (xmi.XMIT method), 14parse_INMR04() (xmi.XMIT method), 14parse_tape() (xmi.XMIT method), 14parse_xmi() (xmi.XMIT method), 13print_details() (xmi.XMIT method), 11print_message() (xmi.XMIT method), 10print_tape() (xmi.XMIT method), 11print_xmit() (xmi.XMIT method), 11

Rread_file() (xmi.XMIT method), 7read_tape_file() (xmi.XMIT method), 7read_xmit_file() (xmi.XMIT method), 7

Sset_binary() (xmi.XMIT method), 6set_codepage() (xmi.XMIT method), 6set_file_object() (xmi.XMIT method), 6set_filename() (xmi.XMIT method), 6set_force() (xmi.XMIT method), 6set_modify() (xmi.XMIT method), 6set_output_folder() (xmi.XMIT method), 6set_overwrite() (xmi.XMIT method), 6set_quiet() (xmi.XMIT method), 6set_tape_file() (xmi.XMIT method), 6set_tape_object() (xmi.XMIT method), 6set_unnum() (xmi.XMIT method), 7

set_xmit_file() (xmi.XMIT method), 6set_xmit_object() (xmi.XMIT method), 6sizeof_fmt() (xmi.XMIT method), 12

Uunload_file() (xmi.XMIT method), 11unload_files() (xmi.XMIT method), 11unload_folder() (xmi.XMIT method), 11unload_pds() (xmi.XMIT method), 11unload_tape() (xmi.XMIT method), 11unload_xmit() (xmi.XMIT method), 11

Xxmi

module, 3XMIT (class in xmi), 4

38 Index

xmi library

Documents