MATLAB Webification Science Client User Guide Jet Propulsion Laboratory

Benjamin Mellman and Stephanie Oij

June 3rd, 2015

Version 0.6.0

Expires January 2016

Table of Contents

1 Introduction..............................................................................32 Environment Setup....................................................................3

2.1 Download Package.......................................................................32.2 Update Path.................................................................................3

3 CLI Set......................................................................................43.1 wslogin: Login to Repository.........................................................43.2 wsls: List Files..............................................................................53.3 wscd: Change Directory (Node).....................................................53.4 wspwd: Print Working Directory (Node).........................................63.5 wsget_file: Download Entire File....................................................63.6 wsget_array: Download Array........................................................7

4 GUI Set.....................................................................................74.1 wstree: Access Repository Tree.....................................................7

4.1.1 Download Arrays......................................................................................84.1.2 Download Entire File.................................................................................94.1.3 Changing URL to Navigate to a Different Repository..............................10

4.2 wsselect: Access Repository by Special Filtering..........................105 Data Visualization Examples....................................................14

5.1 Visual Inspection of Data on Globe..............................................145.1.1 wsglobe..................................................................................................145.1.2 wsglobe_tools.........................................................................................145.1.3 wsglobe_linked.......................................................................................15

6 Acknowledgments...................................................................167 Reference...............................................................................16

2

MATLAB Webification Science Client User GuideJet Propulsion Laboratory

1 IntroductionThe MATLAB Webification Science (w10n-sci) Client (Altinok, 2015) provides a few sets of tools for direct access and manipulation of scientific data on remote repositories.

The Command Line Interface (CLI) Set enables users to easily navigate through and access the data in the hierarchal repository tree using an UNIX-like commanding style directly in the MATLAB command window or in MATLAB scripts.

The Graphic User Interface (GUI) Set allows the user to easily click through the nodes and leaves of the hierarchal repository tree and directly load the desired data range to the MATLAB workspace.

The Data Visualization Example (DVE) are provided to show that visualizing data on a 3D globe is possible with using only MATLAB, and without an additional toolbox such as the mapping toolbox.

Please note that remote repositories must have been properly configured with an adequate version of w10n-sci service software, such as Pomegranate (Xing, 2010).

This user guide describes MATLAB Webification Science Client software version 0.6.0.

In this guide, user inputs are represented by Courier New font: Program Input.

2 Environment Setup

2.1 Download Package

To begin, download the MATLAB w10n-sci client package from the following URL:

URL_yet_to_be_assigned_by_JPL_Software_Release_Authority

to a desired directory.

2.2 Update Path

Open MATLAB and navigate to the directory containing the downloaded package.

Add all directories and subdirectories of the downloaded package to the working path with the command winstall.

>> winstall

Newly included folders will be printed.

3

Matlab path updated:/Users/SMAP/w10n/Users/SMAP/w10n/common/Users/SMAP/w10n/data/Users/SMAP/w10n/earth/Users/SMAP/w10n/tests

At this point, setup is complete.

3 CLI SetThe CLI Set can be used from the command window, or integrated into scripts. The commands wslogin, wsls, wscd, wspwd, wsget_array, wsget_file, and wslogout are included in the CLI Set and summarized in the table below:

Command Line Interface Setwslogin Login to a repository and start a sessionwsls View content information of nodewscd Navigation into a different nodewspwd Display URL of the current nodewsget_array Download leaf arraywsget_file Download entire filewslogout Logout of a repository

All commands, except for wslogin and wslogout check for an active session. If no active session was detected, then an error will be generated. To start a new session, login to the desired repository using wslogin.

Inputs to commands are strings, which must be quoted using single-quotes.

3.1 wslogin: Login to Repository

The wslogin command takes three string arguments:

>> wslogin('URL','username','password');

The URL is the location of the repository. The username and password are required for authentication to access in the event that the repository is password-protected. If the repository is not password-protected, you can enter anything into these two fields.

In this example the URL will be http://data.jpl.nasa.gov/earth/land/smap/, which is not a password-protected repository. Since any string can be used, the username and password will be username and password for this example, as shown in the figure below:

>> wslogin('http://data.jpl.nasa.gov/earth/land/smap','username','password')Login success.http://data.jpl.nasa.gov/earth/land/smap>>

4

The command window will report the status of your login attempt. If login is successful, the current node’s name will also be listed, as seen above.

Note: The active session persists over common MATLAB commands such as clear all, which will still be effective but do not end the active session.

3.2 wsls: List Files

The wsls command is similar to the UNIX ls command. Instead of listing files and directories in your current directory. the wsls command lists the leaves and sub-nodes under current node in the repository, e.g.,

>> wsls;SMAP_L3_SM_P_20010501_D04003_001.h5>>

Following the command with a semi-colon suppresses the return value, but not the listing, of the command. If the semi-colon is not used, MATLAB will print out the return value in the command window:

>> wslsSMAP_L3_SM_P_20010501_D04003_001.h5

ans =

name: 'SMAP_L3_SM_P_20010501_D04003_001.h5' kind: 'leaf' url: 'http://data.jpl.nasa.gov/earth/land/smap' attr: [3x1 struct] size: 7889408 webifiable: 1

>>

After listing the leaves and nodes available in the current node, the user can then use wscd to change directories into a node.

3.3 wscd: Change Directory (Node)

The wscd command is similar to the UNIX cd command for change directory. The wscd command allows navigation into a different node on the hierarchal tree of the repository. The usage is as follows:

>> wscd('node_name')

Path names can be provided as absolute or relative. For example, assuming the current node is at:

5

http://data.jpl.nasa.gov/earth/land/smap/SMAP_L3_SM_P_20010501_D04003_001.h5

to change to the Soil_Moisture_Retrieval_Data node, you can provide a relative path:

>> wscd('Soil_Moisture_Retrieval_Data')

http://data.jpl.nasa.gov/earth/land/smap/SMAP_L3_SM_P_20010501_D04003_001.h5/Soil_Moisture_Retrieval_Data>>

or the absolute path:

>> wscd('http://data.jpl.nasa.gov/earth/land/smap/SMAP_L3_SM_P_20010501_D04003_001.h5/Soil_Moisture_Retrieval_Data')

http://data.jpl.nasa.gov/earth/land/smap/SMAP_L3_SM_P_20010501_D04003_001.h5/Soil_Moisture_Retrieval_Data>>

The URL of the new location is then displayed to the screen. Note that the wscd command changes the current node for the active session in MATLAB, but does not return a value.

3.4 wspwd: Print Working Directory (Node)

The wspwd command is similar to the UNIX pwd command for print working directory. The wspwd command shows the URL of the current node.

>> wspwd

Assuming, the current node is http://data.jpl.nasa.gov/earth/land/smap/SMAP_L3_SM_P_20010501_D04003_001.h5/Soil_Moisture_Retrieval_Data, the wspwd will produce the following results:

>> wspwd

Current URL: http://data.jpl.nasa.gov/earth/land/smap/SMAP_L3_SM_P_20010501_D04003_001.h5/Soil_Moisture_Retrieval_Data

3.5 wsget_file: Download Entire File

To download an entire file, regardless of it being webifiable or not, use wsget_file. The usage of the command is as follows:

>> wsget_file('URL','local_filename')

The URL must be fully qualified and the local filename is the destination of where the downloaded data will be saved.

6

For example, download the file http://data.jpl.nasa.gov/earth/land/smap/SMAP_L3_SM_P_20010501_D04003_001.h5 using wsget_file,

>> wsget_file(' http://data.jpl.nasa.gov/earth/land/smap/SMAP_L3_SM_P_20010501_D04003_001.h5 ','MY_SMAP_L3_SM_P_20010501_D04003_001.h5')

The above example shows the return value 1 when the command completes with success. The file is saved in the directory displayed in the current directory window and named MY_SMAP_L3_SM_P_20010501_D04003_001.h5. Depending on the size of the file, it may take some time to download.

3.6 wsget_array: Download Array

To download an array from a leaf, use the wsget_array command. The usage of the command is as follows:

>> wsget_array('URL')

The URL of the data must be fully qualified, providing the full path to the array.

As an example, the soil_moisture array will be downloaded. Use the wsget_array command to pull one of the arrays into the workspace using the fully qualified URL for the array as shown below:

>> soil_moisture_array = wsget_array('http://data.jpl.nasa.gov/earth/land/smap/SMAP_L3_SM_P_20010501_D04003_001.h5/Soil_Moisture_Retrieval_Data/soil_moisture');>>

The command above saves the soil_moisture array from your current node into the current workspace under the name soil_moisture_array. This functionality is being used standalone in the current working session, but it is also possible to incorporate data into a script in this way.

4 GUI SetThe GUI Set includes wstree and wsselect. These are summarized in the table below:

Graphic User Interface Setwstree Interact with repository treewsselect Interact with repository by special filtering

4.1 wstree: Access Repository Tree

This GUI tool allows navigation through, and access to, the leaves and nodes of the repository to pull the desired data into your MATLAB workspace. To open the GUI, use wslogin to login to the repository and type wstree into the command window.

7

The GUI will open in a new window titled “Remote data sources” with the current node shown. If the node is double clicked, further sub-nodes appear. As these nodes are opened, their contents are revealed. Additionally, in the MATLAB workspace notifications are shown in the command window of the current location in the hierarchal tree, as shown below.

Navigation through the tree is also possible by typing directly into the URL box near the top of the GUI window.

4.1.1 Download Arrays As shown in the figure above, there are many arrays containing data. By left-clicking on them, the data is downloaded to the local machine and available in the MATLAB workspace:

This will download the entire array. By clicking the same data array again in the GUI tree, the command window shows that it is retrieving the data from the local path rather than

8

downloading it again from the URL. The package locally stores the data the first time it is downloaded. From then on, it knows to retrieve the data from local storage rather than downloading it again as some data arrays can be quite large and take an appreciable amount of time to download.

To only download a portion of the array, change the indices of the array by right-clicking on the array name in the GUI window. A new box will appear prompting for the desired range and step size:

The first number is the beginning array index for the range of interest. The second number specifies how many data points to be skipped between samples. The third number is the ending array index number for the range of interest. For example, entering [1,5,20] will take the 1st element of the array, skips the next five elements which are elements 2-6, then it takes element number 7, skips elements 8 – 12, accepts element 13 and so on.

When custom array indices are specified, the indices turn blue in the tree:

Left-click on the array name, now with the custom indices indicated in blue, and the custom array is downloaded again from the URL and saved to the workspace.

4.1.2 Download Entire File The GUI also allows downloading an entire file from a repository. After navigating to a file and left-clicking on it, the program asks where the file should be saved. Enter the desired output directory and click save.

9

4.1.3 Changing URL to Navigate to a Different RepositoryChange the URL to navigate to a different repository by simply clicking on the URL in the URL box near the top of the GUI window, entering in the new URL, and pressing enter:

After entering a new URL, a login prompt pops up. If it is not a secure repository and there is no username or password required, put anything in these fields and press “OK”. When the GUI window is closed, the program automatically logs out of the session.

4.2 wsselect: Access Repository by Special Filtering

The wsselect GUI allows users to select data products corresponding to orbits, dates, or times. Relative to wstree, wsselect also provides a user with the ability to easily download many data arrays, without having to click on each one. This GUI is designed for the repository https://smap4all.jpl.nasa.gov/data/product. In order for this GUI to work properly with a different repository, it must have the same directory structure.

To open the GUI, type wsselect into the command window.

>> wsselect

This will open a GUI with the repository of current session. In this example, it is https://smap4all.jpl.nasa.gov/data/product.

10

The top bar contains the current data repository. The bar below the top bar, on the left, contains all of the product names with the associated repository. The bar to the right will change corresponding to the selection made in the bar on the left. This bar on the right contains the contents of the product selected. This is displayed in the diagram below.

When these bars are selected, the first box is populated with either orbits or files. If the product selected is a level 1 or level 2 product, the first box will contain orbits. If the product selected is a level 3 product, the first box will contain files.

Click and drag the mouse over the desired files or orbits to select multiple files or orbits. To select individual orbits or files, hold CTRL (or Command for Mac) and click on the desired files or orbits. The entire list can be selected using CTRL + a (or Command + a for Mac). Once all of the desired orbits or files are selected, right click to populate the DATE box.

Content of Product LeafData Products

Current Data Repository

11

By default, both the “Asc” and “Dsc” (ascending and descending) check boxes are selected. The selection of these boxes determines the dates listed in the DATE box. Below shows only the ascending orbits listed because the “Dsc” box is un-selected.

Multiple dates can be chosen in the same fashion as choosing multiple orbits or files. Once the selection is made, right click to populate the TIME box. Select individual or multiple times and right click to populate the SEQ box. Select individual or multiple sequences to populate the DATA FIELDS box. A window will open saying “Retrieving fields” and the DATA FIELDS box will be populated.

12

The boxes can be updated by changing the selection and right clicking. Any preceding box will also have to be updated.

Select individual or multiple data fields and right click to view details or download. After right-clicking a window will open displaying the size of the data fields.

By selecting details, information about the data fields selected will be printed in the command window.

Click details:

3.04 KB : float64 : 2014/11/11/SMAP_L1A_RADAR_01021_A_20141111T120258_T11251_001.h5/Health_and_Status_Data/beam_index_crossing_time3.04 KB : uint16 : 2014/11/11/SMAP_L1A_RADAR_01021_A_20141111T120258_T11251_001.h5/Health_and_Status_Data/hsd_status_flag3.04 KB : uint16 : 2014/11/11/SMAP_L1A_RADAR_01021_A_20141111T120258_T11251_001.h5/Health_and_Status_Data/loopback_hh3.04 KB : float64 : 2014/11/11/SMAP_L1A_RADAR_01022_A_20141111T134123_T11251_001.h5/Health_and_Status_Data/beam_index_crossing_time3.04 KB : uint16 : 2014/11/11/SMAP_L1A_RADAR_01022_A_20141111T134123_T11251_001.h5/Health_and_Status_Data/hsd_status_flag3.04 KB : uint16 : 2014/11/11/SMAP_L1A_RADAR_01022_A_20141111T134123_T11251_001.h5/Health_and_Status_Data/loopback_hh>>

By selecting yes, a new window will open asking to store a copy in the workspace.

By clicking yes the data fields downloaded will populate the workspace with variable names corresponding to the filename. In addition, the data fields will be downloaded to the “data” directory as .mat files with the same directory structure as the data tree. Anything downloaded will remain in the “data” directory unless the user manually moves or deletes the files.

13

If no is selected, the data will only be downloaded into the “data” directory and not added to the workspace.

5 Data Visualization ExamplesThis GUI Set includes wsglobe , wsglobe_tools, and wsglobe_linked. These commands are summarized in the table below:

Data Visualization Exampleswsglobe Visualize data on globewsglobe_tools Adjust data transparency on globewsglobe_linked View four globes with data

5.1 Visual Inspection of Data on Globe

The items described in this section provide example code from which a user can build from and alter to produce scripts to visualize data in the MATLAB environment.

These globe functions are provided to exemplify different ways you can visualize downloaded data on a 3D rotating earth within MATLAB. The code is well annotated to assist the user in utilizing the code for their individual purposes. Below describes each of these three functions.

5.1.1 wsglobeThe wsglobe function simply takes an Earth image, stored in the Earth folder as earth.mat, wraps it around a sphere, and populates it with the data. The code provided with this example demonstrates how to align the data with the sphere. Typing wsglobe into the Command Window will open the GUI with example data.

14

5.1.2 wsglobe_toolsThe wsglobe_tools function calls the same globe as wsglobe, but adds a few items to the globe window. It shows you how to add a slider to the window to change the transparency of each of the layers in the window. Call this function by typing wsglobe_tools into the Command Window.

Notice how as the slider is moved to the left (left image), the data layer becomes more transparent giving way to the globe layer. As the slider is moved to the right (right image) the globe layer becomes more transparent giving way to the data layer.

It is possible to take this code and expand on it to having multiple data layers on a single globe with multiple sliders for each of the data layers.

5.1.3 wsglobe_linkedThe wsglobe_linked function populates the screen with 4 identical windows with Earths and data in them. The four Earths are linked together so as you rotate or zoom one of the globes, the other three globes are identically modified. Type wsglobe_linked into the Command Window to open the four linked globes.

15

It should be simple to extend this function so that each globe displays different data with its own slider, etc.

In the current implementation of this function, when you close one of the four linked windows the link between all the windows will break and the globes become independent.

6 AcknowledgmentsThis user guide was developed under a short task supported by the Soil Moisture Active Passive (SMAP) Project and Task “Webification of Earth Science Data” funded by NASA ESDIS Program Office.

7 ReferenceAltinok, A., 2015, “MATLAB Webification Science Client”. JPL New Technology Report #49858. In the process of being open sourced.

Xing, Z., 2010, “Pomegranate – Web Access of NetCDF and HDF File”. JPL New Technology Report #47546. Open source software: http://pomegranate.nasa.gov

16