fully automated updating of arbitrarily complex excel

Fully Automated Updating of Arbitrarily Complex Excel WorkbooksDavid OesperLands’ End

WIILSUJune 28, 2017

The Challenge: Automation

• A coworker left the company and I inherited a set of his programs that he ran manually at the end of every fiscal month.

• He generated a series of data sheets which he then manually cut and pasted into existing Excel workbooks.

• After all that, he manually emailed the revised Excel workbook to several recipients.

The Challenge: Excel

• The Excel workbooks my coworker was updating were not simple grids of rows and columns.

• Our clients who received these Excel workbooks designed them and preferred not to make any changes to the design.

• It was evident to me that SAS alone would not be able to reproduce these workbooks exactly (at least not without a lot of work!).

The Challenge: A SAS Solution?

• I knew how to add hidden data sheets and add references to them on the main worksheet.

• But could SAS, running on Linux, make a “strategic strike” on an existing Excel workbook, and update or replace existing data sheets?

• If I could update the workbook, then emailing the revised workbook and/or uploading it to a Windows directory would be easy.

Five Step Approach

• Create a static version of your Excel workbook to use as a “model” for SAS, and

• Add one or more hidden data sheets

• Clone a copy of your Excel workbook for editing

• Replace your data sheets “on the fly”

• Save or email to the destination(s) of your choice

What You’ll Need

• SAS 9.4 TS1M2 or later

• SAS/ACCESS to PC Files

• Microsoft Excel (to prepare your model Excel workbook with data sheets)

Step #1

Create a static version of your Excel workbook to use as a “model” for SAS

Excel Workbook to UpdateU.S. Census Divisions – Population by State

Add one or more hidden data sheets

Step #2

Excel Workbook to UpdateU.S. Census Divisions – Population by State

Excel Workbook to UpdateOur Data Sheet

proc print data=sashelp.us_data;title 'sashelp.us_data';var statename population_2010

division;run;

Excel Workbook to UpdateU.S. Census Divisions – Population by State

Data!B4

Excel Workbook to UpdateOur Data Sheet

proc print data=sashelp.us_data;title 'sashelp.us_data';var statename population_2010

division;run;

Data!B4

Excel Workbook to UpdateU.S. Census Divisions – Population by State

Clone a copy of your Excel workbook for editing

Step #3

Copy Excel Workbook for EditingPC SAS

/* Today's date in yyyymmdd format: example 20170503 */%let tdate = %sysfunc(today(),yymmddn8.);

/* Path and File Name of model Excel workbook */%let xlinpath = C:\Users\David\Documents\SASIN;%let xlinfile = U.S. Census Divisions.xlsx;

/* Path and File Name of output Excel workbook */%let xloutpath = C:\Users\David\Documents\SASOUT;%let xloutfile = U.S. Census Divisions &tdate..xlsx;

Copy Excel Workbook for EditingPC SAS

/* Copy the Excel Workbook */options noxwait noxsync;data _null_;

call system(%unquote(%bquote(')copy "&xlinpath\&xlinfile"

"&xloutpath\&xloutfile"%bquote(')));

run;options xwait xsync;

Copy Excel Workbook for EditingLinux SAS

%let userid = daoespe;

/* Today's date in yyyymmdd format: example 20170503 */%let tdate = %sysfunc(today(),yymmddn8.);

/* Path and File Name of model Excel workbook */%let xlinpath = /sasdevr/devr180/&userid;%let xlinfile = U.S. Census Divisions.xlsx;

/* Path and File Name of output Excel workbook */%let xloutpath = /sasdevr/devr7/&userid;%let xloutfile = U.S. Census Divisions &tdate..xlsx;

Copy Excel Workbook for EditingLinux SAS

/* Copy the Excel Workbook */data _null_;

/* change to input directory */call system("cd &xlinpath");/* copy model Excel workbook to output directory */call system("cp –f %unquote(%bquote(')

&xlinfile%bquote(')) &xloutpath");/* change to output directory */call system("cd &xloutpath");/* rename excel file on output directory */call system("mv %unquote(%bquote(')

&xlinfile%bquote('))%unquote(%bquote(')

&xloutfile%bquote('))");run;

Replace your data sheets “on the fly”

Step #4

Getting Our Updated Census DataScoop .csv file off a web site: filename url

2016 U.S. population estimate

Getting Our Updated Census DataScoop .csv file off a web site: filename url

filename newpop url"https://www2.census.gov/programs-surveys/popest/datasets/2010-2016/state/asrh/scprc-est2016-18+pop-res.csv";

2016 U.S. population estimate

You could also use PROC HTTP to get the data.

Getting Our Updated Census Data

Existingdata sheet

SAS data set for new data sheet

Excel row #s off by 1 -Column headers take a row

Create an Updated Excel WorkbookUpdate Working Copy of Excel File!

/* use LIBNAME XLSX access method */libname xlout xlsx "&xloutpath/&xloutfile";

/* update data sheet with a merge */data xlout.'Data'n;

merge xlout.'Data'n sasuser.newpop;by statename;

run; This is the Key!!LIBNAME XLSX

+Simple DATA Step MERGE!

SAS Analytics Pro – PC SASDoes Not Include SAS/ACCESS to PC Files

/* update data sheet with a merge */data xlout.'Data'n;

merge xlout.'Data'n sasuser.newpop;by statename;

run;

SAS LOGlibname xlout xlsx "&xloutpath/&xloutfile";

ERROR: The product with which the engine XLSX is associated is either not licensed for your system or the product license has expired. Please contact your SAS installation representative.ERROR: Error in the LIBNAME statement.

Updated Excel Workbook2016 Population Estimates

Data sheet was hidden in the original workbook – remains hidden

Save or email to the destination(s) of your choice

Step #5

Copy Updated Excel WorkbookFrom Linux to Windows

/* filename same as output libname */filename xlout "&xloutpath/&xloutfile";

/* filename for FTP to Windows drive */filename ftpout ftp

"DIRMARKET124/BA/Staff/Dave/SASOUT/&xloutfile"recfm=s host='********' user='********‘pass='********';

Copy Updated Excel WorkbookFrom Linux to Windows

/* copy the file byte-for-byte */data _null_;

length filein 8 fileid 8;filein = fopen('xlout','I',1,'B');fileid = fopen('ftpout','O',1,'B');rec = '20'x;do while(fread(filein)=0);

rc = fget(filein,rec,1);rc = fput(fileid,rec);rc = fwrite(fileid);

end;rc = fclose(filein);rc = fclose(fileid);

run;

Email Updated Excel WorkbookFrom Linux

/* output filename using the email accessmethod - notice we are emailing the Excelworkbook we created on Linux */

filename eml emailto = ('Dave.Oesper@landsend.com')cc = ('DaveOesper@mac.com')type = 'text/plain‘emailid = 'Dave.Oesper@landsend.com'attach = ("&xloutpath/&xloutfile"ct="application/octet-stream" ext="xlsx")subject = 'Updated U.S. Census Data';

Email Updated Excel WorkbookFrom Linux

/* Send email, but only if no errors */%macro sendemail;%if %length(&syserrortext) ne 0 %then %return;data _null_;

file eml;put "Here are the updated population estimates

for each state from the U.S. CensusBureau." /;

put 'Best Regards,' /;put 'Dave';

run;%mend sendemail;%sendemail;

Email Updated Excel WorkbookFrom Linux

Email Updated Excel WorkbookFrom Linux

Email Updated Excel WorkbookFrom Linux

Summary

• We used the LIBNAME XLSX engine, a part of SAS/ACCESS to PC File Formats, to update data sheets in an Excel workbook using a simple DATA step MERGE.

• We used FILENAME URL to “scoop” a .csv file off of a web site to get our updated data.

• We used FILENAME FTP to copy our updated Excel workbook from a Linux server to a Windows directory.

• We used FILENAME EMAIL to email our updated Excel workbook to a recipient list.

Thank You!

David OesperLands’ EndDave.Oesper@landsend.com

WIILSUJune 28, 2017