ReadmeHPC Image Rendering v2

Lab version: 2.0.0

Last updated: 5/18/2023

Contents

OVERVIEW................................................................................................................................................. 3

GETTING STARTED................................................................................................................................... 5Task 1 - Creating a Hosted Service in Windows Azure.........................................................................5

Task 2 - Inspecting the Image Rendering Solution...............................................................................5

Task 3 - Inspecting the Run.cmd File..................................................................................................11

Task 4 - Preparing the Image Rendering Solution for Deployment....................................................12

Task 5 - Deploying the Image Rendering Package to Windows Azure...............................................13

Task 6 - Uploading Instruction Files to the Blob Storage...................................................................14

DEPLOYMENT.......................................................................................................................................... 17Task 1 - Deploying the Parametric Sweep Application to Windows Azure Nodes.............................17

RUNNING THE CLIENT............................................................................................................................ 19Task 1 - Verifying Cluster State..........................................................................................................19

Task 2 – Running the Client Application............................................................................................19

SUMMARY................................................................................................................................................ 25

Overview

The HPC Image Rendering sample demonstrates how to run a parametric sweep application that renders images according to instruction files.

This sample includes several types of applications, as shown in Figure 1:

Figure 1Architecture of the Image Rendering sample

Note: Blue arrows signify application / service calls; orange arrows signify data transfer to/from blobs.

1. Windows Phone 7 (WP7) client. The client creates the HPC job, monitors the job’s status, and shows the output video file.

2. HPC Web Access website. A Windows Azure web role that manages the creation of the HPC parametric sweep job using the representational state transfer (REST) interface of the HPC Pack web features.

3. Windows HPC Server 2008 R2 SP2 cluster with Windows Azure nodes. The head node controls the Windows Azure nodes that run the image-rendering parametric sweep application.

4. Windows Azure blob storage. The blob storage contains the input and output rendering files, as well as the output video created by combining all the outputted images.

5. Video encoding service. A Windows Azure web role that downloads all rendered images and creates a video file from them.

6. Push notification service. A Windows Azure web role that uses WP7 Push Notifications to inform the client about the status of the job.

This sample uses the Aqsis application to render 3D images according to the RenderMan® standard, and the FFmpeg utility to create a video file from a set of images.

Note: To use this sample application, download the sample RenderMan data files, extract them to the labs folder, and copy them to Windows Azure blob storage as described later on in the document.

This sample demonstrates some of the new features offered by Windows HPC Server 2008 R2 Service Pack 2 (SP2). Refer to the What's New in Windows HPC Server 2008 R2 Service Pack 2 article on TechNet for the complete list of new features offered in this version.

Key Features

This sample demonstrates the following features:

Uploading a parametric sweep application package to Windows Azure nodes.

Uploading and downloading files to Windows Azure blob storage.

Creating and running a parametric sweep job in a Windows HPC Server 2008 R2 SP2 cluster.

Controlling jobs with the REST interface of the HPC Pack web features.

Getting Started

The following prerequisites are required for running the sample:

On the HPC Cluster’s head node:

HPC Pack 2008 R2 SP2

HPC Pack web features

Note: When configuring the web components according to the Installing the Microsoft HPC Pack Web Components article, follow the steps for using the Basic authentication method.

On the development machine:

Microsoft Visual Studio 2010

Windows Azure SDK and Windows Azure Tools for Microsoft Visual Studio (March 2011)

Windows Phone Developer Tools RTW

Windows Phone Developer Tools January 2011 Update

HPC Pack 2008 R2 SP2 Client Utilities

To run this sample, you need to have administrative access to your HPC cluster’s head node, and you must have a valid Windows Azure account, a Windows Azure worker node template defined in your head node, and several Windows Azure worker nodes in the HPC cluster that are started and online. Follow the Deploying Windows Azure Worker Nodes in Windows HPC Server 2008 R2 Step-by-Step Guide on TechNet for further information.

Task 1 - Creating a Hosted Service in Windows Azure

In this task, you will create a hosted service in the Windows Azure Platform Management Portal. Hosted services are responsible for hosting applications such as Windows Azure web roles and worker roles. A hosted service has a special URL address that you can use in order to access the applications you deploy to it, and you can use it to define the geographic location of newly deployed servers.

The image-rendering sample includes several Windows Azure web roles that need to be deployed to a Windows Azure hosted service; before deploying these roles, you will need to create a hosted service for them. Follow the How to Create a Hosted Service walkthrough in MSDN in order to create the new hosted service using the Management Portal. At this point, you are only creating a hosted service, without deploying packages to it. Also, write down the DNS prefix you used for the URL, as you will need to use it in later tasks.

Task 2 - Inspecting the Image Rendering Solution

In this task, you will inspect the ImageRendering solution to see the various projects included in this sample.

1. Open Microsoft Visual Studio 2010 from Start | All Programs | Microsoft Visual Studio 2010 | Microsoft Visual Studio 2010.

2. Open the ImageRendering.sln solution file located in the ImageRenderingSP2\Source folder.

3. Examine the projects tree in the Solution Explorer window. The solution tree includes several Windows Azure projects, class libraries, and a console application, as shown in Figure 2:

Figure 2The ImageRendering projects tree

The following list describes the purpose of each project in the ImageRendering solution:

AzureDeployment

◦ ImageRenderingAzure. The project contains the service configuration files that control how to deploy the Windows Azure web roles to the hosted service. Three web roles are configured in this deployment project: HpcWebGate, WcfAzureAgent, and WcfRoleNotification.

Client

◦ HpcRestClient. The project contains a library that supports the REST interface of the HPC Job Scheduler’s web service, which is a part of the HPC Pack web features. Using this library, clients can send REST requests over HTTP to the HPC Job Scheduler’s web service in order to create new parametric sweep jobs and get information about active and finished jobs.

You can inspect the HpcClient.cs file in this project to see how the REST interface is used and which data structures it receives and returns.

◦ HpcWebGate. The project contains a web application that enables Windows Phone 7 client devices to create new HPC jobs. This project uses the HpcRestClient library to send job creation requests over HTTP to the HPC Job Scheduler’s web service using the REST interface.You can inspect the HpcControl.ascx.cs file in this project to see how the web application uses the HpcRestClient library to create the new parametric sweep job.

◦ WindowsPhone7. The project contains a WP7 application that does the following:

i. Enables the user to start a new parametric sweep job for the image rendering process using the web application from the HpcWebGate project. The user enters the account information of an HPC administrator and the number of frames to render, and then starts the job according to that information, as shown in Figure 3:

Figure 3The HPC Mobile Web Access page

4. While the HPC job runs, the device reports to the user which frames have been constructed and the percentage of total work completed, as shown in Figure 4:

Figure 4Reporting the progress of the job

5. After the job completes, the user will be able to see the rendered video file on the device, as shown in Figure 5:

Figure 5Viewing the output video created by the HPC job

CommandLine

◦ AzureBlobCopy. The project contains a command-line utility that enable users to upload and download files from Windows Azure blob storage. The utility also enables sending notifications to WP7 devices after uploading files to the blob.

◦ AzureUtilities. The project contains a library for handling the uploading and downloading of files from blobs. The AzureBlobCopy and the HpcAzureAgent projects make use of this library for that purpose.

You can inspect the BlobUtilities.cs file in this project to see how the library uses the Microsoft.WindowsAzure.StorageClient assembly to communicate with blobs.

Services

◦ WcfAzureAgent. The project contains a WCF service that exposes an HTTP REST interface, which receives a notification whenever a rendered image is loaded into the output blob. Each Windows Azure worker node in the HPC cluster calls the service after it processes an image, and the service is responsible for downloading all the rendered images from the blob. After the images are downloaded, the service generates the video file and notifies the WP7 device of the newly generated video.You can inspect the AzureAgent.svc.cs file in this project to see the service’s implementation: how it downloads images from the blob, renders the video file, and sends notifications to WP7 devices using the WcfRoleNotification service.

◦ WcfRoleNotification. The project contains a WCF service that enables WP7 devices to subscribe to notifications. The service receives notifications from the WcfAzureAgent service whenever there is a change in the HPC job’s state, and pushes those notifications to the WP7 devices. The service keeps a dictionary of all subscribed clients and their addresses, and is able to send notifications about specific jobs to the client that started the job.You can inspect the NotificationService.svc.cs file in this project to see how the notification message is generated and sent to the subscribed WP7 clients (look for the SendToast and SendMessage methods).

Task 3 - Inspecting the Run.cmd File

In this task, you will inspect the contents of the run.cmd file to see which commands run when the parametric sweep job executes in each Windows Azure compute node.

1. In the Solution Explorer window, right-click the solution and select Open Folder in Windows Explorer.

2. In the Windows Explorer window navigate to the aqsis\bin folder, locate the run.cmd file, and view the file’s contents:

RUN.CMD

REM Use the input parameter as a frame index.set frame=%1set total=%2set device=%3

REM Setup the executable, input, and output folders.set root=%CCP_PACKAGE_ROOT%\AqsisSP2set inputdir=%CCP_WORKDIR%\%CCP_JOBID%\%CCP_TASKID%\inputset outputdir=%CCP_WORKDIR%\%CCP_JOBID%\%CCP_TASKID%\outputif not exist %inputdir% mkdir %inputdir%if not exist %outputdir% mkdir %outputdir%

REM Pull input data from blob storage. %root%\bin\AzureBlobCopy.exe -Action Download -BlobContainer input2 -LocalDir %inputdir% -FileName %frame%.zip

REM Unzip the input file, run the executable, and create output data.%root%\bin\rar.exe e -y %inputdir%\%frame%.zip %outputdir%cd %outputdir%%root%\bin\aqsis.exe -shaders:"%root%\displacement:%root%\shaders\imager:%root%\shaders\light:%root%\shaders\surface:%root%\shaders\volume" -displays="%root%\bin" %outputdir%\%frame%.rib

REM Upload the output files to blob storage.%root%\bin\AzureBlobCopy.exe -Action Upload -BlobContainer output2 -LocalDir %outputdir% -FileName %frame%.tif

REM send client notification %root%\bin\AzureBlobCopy.exe -action webget -query /addframe?frame=%frame%^&total=%total%^&device=%device%

REM remove local filesdel /Q %inputdir%\%frame%.zipdel /Q %outputdir%\%frame%.ribdel /Q %outputdir%\%frame%.tif

3. The run.cmd file executes the following commands:

a. Runs the AzureBlobCopy utility to download the current rendering instruction file from the input blob and decompress it.

b. Runs the aqsis 3D rendering application.

c. Runs the AzureBlobCopy utility to upload the generated .tiff image to the output blob and notify the user that a new image is ready.

Task 4 - Preparing the Image Rendering Solution for Deployment

In this task, you will make the necessary adjustments to the projects’ configuration files so you can deploy them to the Windows Azure hosted service you created in Task 1.

1. In the Solution Explorer window, expand the Client folder, expand the HpcWebGate project, and then open the Web.config file.

2. In the Web.config file, locate the <appSettings> section, and set the values of the following keys:

a. HpcWebServiceUrl. This application setting contains a URL that points to the location of the HPC Job Scheduler’s HTTP web service. For example, if your HPC Job

Scheduler’s machine name is MyHeadNode, and the HPC cluster’s name is MyCluster, then the application setting will be:

XML

<add key="HpcWebServiceUrl" value="https://MyHeadNode/WindowsHpc/MyCluster"/>

b. GroupName. This application setting holds the name of the Windows Azure node group. If you have a different group that contains Windows Azure worker nodes and would prefer to use it for this sample, change the value of this setting to the name of your group as it is specified in your HPC cluster’s node configuration.

3. In the Solution Explorer window, expand the Client folder, expand the WindowsPhone7 project, and open the App.xaml file.

4. In the App.xaml file, locate the WebGateServer string resource, and change the DNS prefix to the prefix of the hosted service you created in Task 1. For example, if the DNS prefix you used is ImageRenderingHPC, then the resource code should look like this:

XAML

<sys:String x:Key="WebGateServer">ImageRenderingHPC.cloudapp.net:8080</sys:String>

Note: This URL is used to connect to the HpcWebGate web application, which is hosted on port 8080, so do not remove the port number from the URL.

5. Still in the WindowsPhone7 project, open the ServiceReferences.ClientConfig file and locate the notification service endpoint under <system.serviceModel>|<client>|<endpoint>. In the address attribute, change the DNS prefix to that of your new hosted service.

6. In the Solution Explorer window, expand the CommandLine folder, expand the AzureBlobCopy project, and open the App.config file. The appSettings section contains three keys: StorageAccountName, StorageKey, and AzureServiceUrl.

a. Change the StorageAccountName and StorageKey settings to match your Windows Azure storage account name and storage account primary key, respectively.

b. In the AzureServiceUrl setting, change the DNS prefix to that of your hosted service.

7. In the Solution Explorer window, expand the Services folder, expand the WcfAzureAgent project, and then open the Web.config file. The appSettings section contains two keys: StorageAccountName and StorageKey. Change the settings of these keys to match your Windows Azure storage account name and storage account primary key, respectively.

8. Still in the Web.config file of the WcfAzureAgent project, locate the notification service endpoint under <system.serviceModel>|<client>|<endpoint>. In the address attribute, change the DNS prefix to that of your new hosted service.

9. Save all the changed files and build the solution.

Task 5 - Deploying the Image Rendering Package to Windows Azure

In this task, you will deploy the image-rendering package to the hosted service you created in Task 1.

To deploy your packages to Windows Azure, follow the instructions in the Publish a Windows Azure Application section of the Getting Started with the Windows Azure Tools for Visual Studio article on MSDN. The Windows Azure project that contains the roles is the project named ImageRenderingAzure, which is located in the Solution Explorer window under the AzureDeployment folder.

Note: In step four of the instructions (selecting the deployment environment), select the production environment of the hosted service you created in Task 1.

Task 6 - Uploading Instruction Files to the Blob Storage

In this task, you will upload the instruction files required by the parametric sweep application to a blob in your Windows Azure storage account.

Uploading, downloading, and browsing files in blobs is an easy task if you install one of the blob storage browsing applications, such as CloudBerry Explorer for Azure Blob Storage, or the Azure Storage Explorer. The following steps are for the CloudBerry Explorer application; you can use the same techniques with Azure Storage Explorer, but the steps may differ.

1. Download and install CloudBerry Explorer for Azure Blob Storage.

2. Open the CloudBerry Explorer for Azure Blob Storage from Start | All Programs | CloudBerryLab | CloudBerry Explorer for Azure Blob Storage | CloudBerry Explorer for Azure Blob Storage.

3. Open the File menu and select Azure Blob Storage Accounts. The Account Registration dialog will appear, as shown in Figure 6:

Figure 6

The Account Registration dialog

4. Click the Add button and input the display name of the storage account, the storage account name, and the shared key (primary access key) of the account. Use the same storage account settings you used in Task 4 for the AzureBlobCopy project.

5. Click the Test Connection button and wait for the approval message. Close the approval message, click OK to add the storage account, and then close the Azure Blob Storage Accounts dialog.

6. You should now see your blob storage in the left pane of the application, and your machine (“My Computer”) in the right pane.

7. Create a new container in the blob by clicking on the New Container button in the left pane, as shown in Figure 7:

Figure 7Creating a new blob container

8. In the Create New Container dialog, set the container name to input2, select the Full public read access option from the Access control options as shown in Figure 8, and click OK.

Figure 8The Create New Container dialog

Note: If the container already exists, it is preferable that you delete it and create it again using the above settings.

9. Locate the newly created container in the list of containers and double-click its name to see its content (it should be empty for now).

10. In the right pane, navigate to the labs folder, and locate the RenderMan\SampleInput folder.

11. Select all the files (40 .zip files) from the SampleInput folder and click the Copy button. Click Yes in the confirmation message that appears, and then wait for the copy procedure to complete.

Note: The content of the folder is about 76MB, so this may take a couple of minutes, depending on your network bandwidth.

12. After the upload completes, close the CloudBerry Explorer application.

Deployment

In order to run this sample, you will need to deploy the parametric sweep application to Windows Azure. To copy applications to each of the Windows Azure nodes, you will use the hpcpack command. This command can be used for packaging applications, uploading them to Windows Azure storage, and deploying packages to every Windows Azure worker node in the cluster, as shown in Figure 9:

Figure 9Using the hpcpack command

Task 1 - Deploying the Parametric Sweep Application to Windows Azure Nodes

In this task, you will deploy the aqsis parametric sweep application to your Windows Azure nodes.

Note: Step 1 of Figure 9 was already executed when the image-rendering solution compiled in Task 4 of the previous section.

1. Open the command prompt window from Start | All Programs | Microsoft Visual Studio 2010 | Visual Studio Tools | Visual Studio Command Prompt (2010).

2. Navigate to the ImageRenderingSP2 labs folder and run the following command to upload the deployment package to the Windows Azure package storage (this step is illustrated in step 2 of Figure 9):

CMD

hpcpack upload Source\aqsisSP2.zip /nodetemplate:"Azure node template" /relativePath:aqsisSP2

Note: Change the value of the nodetemplate parameter to the name of your Windows Azure node template.

3. If you already have Windows Azure nodes started in the HPC cluster, you need to copy the new package to them. To sync the Windows Azure nodes with the new packages stored in the application packages blob, run the following command (this step is illustrated in step 3 of Figure 9):

CMD

clusrun /nodegroup:AzureNodes hpcsync

Running the Client

Task 1 - Verifying Cluster State

In this task, you will verify that the nodes in your cluster are online.

1. Open the HPC 2008 R2 Cluster Manager application from Start | All Programs | Microsoft HPC Pack 2008 R2 | HPC Cluster Manager.

2. In the Cluster Manager application, enter the Node Management section and verify that the Windows Azure nodes in the cluster are online, as shown in Figure 10:

Figure 10Verifying the state of the Windows Azure nodes

Task 2 – Running the Client Application

In this task, you will run the Windows Phone 7 client application that creates the parametric sweep job.

1. Open Microsoft Visual Studio 2010 from Start | All Programs | Microsoft Visual Studio 2010 | Microsoft Visual Studio 2010.

2. Open the ImageRendering.sln solution file located in the ImageRenderingSP2\Source folder.

3. Expand the Client folder, right-click the WindowsPhone7 project, and select Set as StartUp Project.

4. From the Debug menu, select Start Without Debugging, and wait for the Windows Phone 7 application to start and display the screen shown in Figure 11:

Figure 11The Image Rendering welcome screen

5. Click on the Open HPC WEB Access button and wait for the web access page to display, as shown in Figure 12:

Figure 12The HPC Mobile Web Access page

6. Enter the user name and password of an administrative user in the HPC cluster, select the number of frames to render, and click the Start Rendering button.

Note: The user name must include the domain name of the user, in the format domain\user.

7. After the job submits, you will begin seeing notifications about the images being generated, as shown in Figure 13:

Figure 13The Image Rendering job’s status

8. While you wait for the application to finish rendering, you can also inspect the status of the parametric sweep job using the HPC 2008 R2 Cluster Manager utility, as shown in Figure 14:

Figure 14Status of the parametric sweep rendering job

9. Wait a couple of minutes for the application to complete.

10. After the job is completed, you will see the rendered video in the Windows Phone 7 application, as shown in Figure 15:

Figure 15The resulting video created by the HPC job

Summary

After running the Image Rendering sample, you should have learned the following:

How to work with Windows Azure blob storage.

How to use the HPC Pack REST interface.

How to deploy Windows Azure web roles to a hosted service.

How to package an application for Windows Azure.

How to deploy an application to Windows Azure nodes.

How to submit a parametric sweep job from a Windows Phone 7 client application.

How to check a running job’s status.