using python to facilitate python facilitating code refactoring
TRANSCRIPT
![Page 1: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/1.jpg)
Python Facilitating Code Refactoring
Using Python to Facilitate
Code Refactoring
Using Python to Facilitate
Code RefactoringCode Refactoring
11
Code RefactoringCode Refactoring
Ben ChristensonAssociate Scientist – Process Optimization
Engineering Sciences – Core R&D
Dow Chemical Company
7/14/2011
Process Automation Legacy Migration
PALM
Yahya NazerM&E Consultant
Engineering Solutions Technology Center
Dow Chemical Company
![Page 2: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/2.jpg)
Background
Code Refactoring is a systematic way of restructuring code without changing the intent of the code.
![Page 3: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/3.jpg)
Background
Code Refactoring is a systematic way of restructuring code without changing the intent of the code.
� Historically code refactoring has been used for:
� Compilers
� Military
� Finance
� Communication� Communication
� Process Control
![Page 4: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/4.jpg)
Background
Code Refactoring is a systematic way of restructuring code without changing the intent of the code.
� Historically code refactoring has been used for:
� Compilers
� Military
� Finance
� Communication� Communication
� Process Control
� PALM worked on Process Control for:
� Translation
� Simulation
� Transition Logic
� Data Mining
� Code Analysis
� Migration
![Page 5: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/5.jpg)
Background
Process Control, within Dow, refers to using computers to control the equipment for the purpose of manufacturing chemicals.
![Page 6: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/6.jpg)
Background
Process Control, within Dow, refers to using computers to control the equipment for the purpose of manufacturing chemicals.
� Dow has been developing the MOD process control systems since 1960
� The MOD system is a state based system that looks similar to FORTRAN
� The MOD system is cutting edge with how it manages process automationprocess automation
� http://www.controlglobal.com/articles/2006/029.html
![Page 7: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/7.jpg)
Background
Process Control, within Dow, refers to using computers to control the equipment for the purpose of manufacturing chemicals.
� Dow has been developing the MOD process control systems since 1960
� The MOD system is a state based system that looks similar to FORTRAN
� The MOD system is cutting edge with how it manages process automationprocess automation
� http://www.controlglobal.com/articles/2006/029.html
Unfortunately the MOD system is also:
� Not object oriented
� Contains hardware tricks for math and logic
� Is based on the VAX system
![Page 8: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/8.jpg)
EM
This will make the EM Finder that is
used to specify the EM and enablement
logic.
Inputs: *.dtn
Outputs: *.em.xlsb
Routines:
Remove_Non_Ascii
Parse_Out_Glossary
Remove_Line_Number
Setup_Database
Add_Const_Calc
Add_Sequence
Parse_Out_Comments
Combine_Continued_Lines
Associative_Term
Truncate_Constants
DK_AK_ZERO
Digsum
FNG
NOT_in_ALARM
Replace_Strings
Associative_Variables
DEV_Function
IMPORT_Function
� PALM as a research project investigate 30 scenarios containing over 200 transformations.
� Each of these scenarios constituted a mode in the PALM engine which was made from the listed transformation routines.
IMPORT_Function
PFS_Function
ABS_Function
-Bit_Shift_Function
Convert_Variables
Negative_Numbers
Convert_Not
Negative_Numbers
Convert_Not
Basic_IF
Remove_Irregular_text
AND_Prescedence
Trim_Variables
Irrelevant_Parentheses
Timer_Function
Adjust_Scale_Factor
Unravel_Formula
Dividing_By_Zero
VB_Syntax
SST_NOT
Overflow_Function
Floating_Point
-XOR_Syntax
…
![Page 9: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/9.jpg)
EM
This will make the EM Finder that is
used to specify the EM and enablement
logic.
Inputs: *.dtn
Outputs: *.em.xlsb
Routines:
Remove_Non_Ascii
Parse_Out_Glossary
Remove_Line_Number
Setup_Database
Add_Const_Calc
Add_Sequence
Parse_Out_Comments
Combine_Continued_Lines
Associative_Term
Truncate_Constants
DK_AK_ZERO
Digsum
FNG
NOT_in_ALARM
Replace_Strings
Associative_Variables
DEV_Function
IMPORT_Function
� PALM as a research project investigate 30 scenarios containing over 200 transformations.
� Each of these scenarios constituted a mode in the PALM engine which was made from the listed transformation routines.
� A Transformation refers to a code refactor that handles a specific function.
IMPORT_Function
PFS_Function
ABS_Function
-Bit_Shift_Function
Convert_Variables
Negative_Numbers
Convert_Not
Negative_Numbers
Convert_Not
Basic_IF
Remove_Irregular_text
AND_Prescedence
Trim_Variables
Irrelevant_Parentheses
Timer_Function
Adjust_Scale_Factor
Unravel_Formula
Dividing_By_Zero
VB_Syntax
SST_NOT
Overflow_Function
Floating_Point
-XOR_Syntax
…
def Associative_Term(code):
""" This will replace TERM(NNN) = cond
with STEP(NNN+1) = cond + " and STEP(NNN) or STEP(NNN+1)"
"""
try:
for i in xrange(len(code)):
line = code[i].text
if(line.startswith("TERM")):
index = line[5:].split(')',1)[0]
term = line[:5+len(index)+1].strip()
current_step = 'STEP('+index+')'
new_step = 'STEP('+str(evl(index)+1)+')'
line = line.replace(term,new_step)
if(line.find(current_step) == -1):
line += " AND " + current_step
code[i].text = line + " OR "+new_step
# end if
# end for
except: exception()
![Page 10: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/10.jpg)
def add_Signal(glos,um_dtn,links):
""" This will add the signal type
If I6B / I4B / I5B / I10B, then put 4..20mA
V5B = 0..5 VDC
V10B = 0..10 VDC
V20B = 0..20 VDC
"""
try:
global signal_types
add_column(glos,'Signal Type')
signal_types = {'I4B':'4-20mA','I5B':'4-20mA','I6B':'4-20mA','I10B':'4-20mA', \
'V5B':'0.5 VDC','V10B':'0.10 VDC','V20B':'0.20 VDC', \
'N2FB':'200 C nickel RTD','N4FB':'400 C nickel RTD', \
'P2DB':'200 C platinum RTD','P4DB':'400 C platinum RTD', \
'P8DB':'800 C platinum RTD','P8HD':'800 C platinum RTD', \
'P8HDB':'800 C platinum RTD', \
'T2JB':'200 C type J','T8JB':'400 C type J', \
'T2EB':'200 C type E TC','T4EB':'400 C type E TC', \
'T8EB':'800 C type E TC','T10KB':'1000 C type K TC', \
Data Mining Transformations
'T8EB':'800 C type E TC','T10KB':'1000 C type K TC', \
'T13KB':'1300 C type E TC','T15SB':'1500 C type S TC', \
'T4TB':'400 C type T TC'}
for k in signal_types.keys():
lines = line_find(um_dtn,'; CALL '+k+'(','Signal Type')
for line in lines:
ln = line.split(';')[0].strip()
for arg in parse_args(line):
head,index = arg.split('_')
stype = signal_types[k]+' ('+k+')'
index = evl(index) - 1
index = index - index%10 + 1
for i in range(index,index+10):
var = head+'_'+('0000'+str(i))[-4:]
glos[var][-1] = stype
links.append(['Signal Type',var,ln])
# end for
# end for
# end for
except: exception()
![Page 11: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/11.jpg)
Transition Logic
� In MOD we have these global variables that tell what mode the process control is in.
� For migration these modes assignment statements need to be converted into transition logic between any two given modes.
� Transition logic were created by:
algebraic expansion and replacement
T3400_MWMaintance Wait
T3400_MW_PW T3400_PW_MW
� algebraic expansion and replacement
of the logic
� removing any logical paradoxes
� breaking the logic up into transitions
� And compressing the logic
T3400_PWProcess Wait
T3400_FILLFill
T3400_HEATHeat
T3400_RUNRun
T3400_CSHDNControl ShutDown
M
M
T3400_PW_FILL T3400_FILL_PW
T3400_FILL_HEAT
T3400_HEAT_RUN
T3400_HEAT_CSHDN
T3400_RUN_CSHDN
T3400_CSHDN_PW
![Page 12: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/12.jpg)
COM & Excel
By using win32com and pythoncom, it was a very simple matter to control excel through python.� http://sourceforge.net/projects/pywin32/files/pywin32/Build%202
14/pywin32-214.win32-py2.3.exe/download
� http://snippets.dzone.com/posts/show/2036
� This creates a COM connection to Excel
def connect(self,visible=0):def connect(self,visible=0):
""" This will simply connect to the excel already running """
try:
if(self.excel == None):
cc('importing com')
from win32com.client import Dispatch
from win32com.client import GetActiveObject
import pythoncom
cc('Initilaizing python com')
pythoncom.CoInitialize()
cc('Dispatching Excel')
self.excel = Dispatch("Excel.Application")
self.excel.DisplayAlerts = False
if(DEBUG > 3 or visible): self.excel.Visible = 1
else: self.excel.Visible = 0
except: exception()
![Page 13: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/13.jpg)
COM & Excel
� This opens an excel workbookdef open_excel(self,file,ws='',clear=1):
""" This will open a com interface to excel and open the file
it will then store that connection in a global variable
clear will close all worksheet already open in excel
"""
try:
local_echo = 100
self.connect()
if(clear): self.clear()
self.sheets = []
self.modules = {}
self.created_macro = []
self.file = os.path.basename(file)
self.working_directory = os.path.split(file)[0]+'\\' self.working_directory = os.path.split(file)[0]+'\\'
cc('Excel Opening file '+file,local_echo)
self.name = os.path.basename(file)
self.workbook = self.get_workbook(self.name)
if(self.workbook == None):
self.excel.Workbooks.Open(file)
self.workbook = self.get_workbook(self.name)
if(ws != ''):
self.active_sheet(ws)
self.excel.Visible = 1
self.excel.DisplayAlerts = False
self.reconnect()
except: exception({'file':file})
![Page 14: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/14.jpg)
COM & Excel
� This will create a VBA macro within Exceldef create_macro(self,module,code):
""" This will create a VBA excel macro within the given module name """
try:
i = 0
code = code.replace('function ','Sub ').replace('Function ','Sub ').replace('sub ','Sub ')
for sub in code.split('Sub'):
if(sub != ''): self.created_macro.append(sub.split('(')[0].strip())
if(not self.modules.has_key(module)):
for j in xrange(self.workbook.VBProject.VBComponents.Count):
wb = self.read_workbook().VBProject.VBComponents(j+1)
if(wb.name == module):
self.modules[module] = wb
break
# end if# end if
# end for
self.modules[module] = self.read_workbook().VBProject.VBComponents.Add(1)
self.modules[module].name = module
while(code[0].strip() == ''): code = code[1:]
buffer = len(code[0]) - len(code[0].lstrip())
if(buffer < 0): buffer = 0
cc("removing buffer of size "+str(buffer),40)
for i in range(len(code)):
assert(code[i][:buffer].strip() == '')
code[i] = code[i][buffer:]
code.append('')
for i in range(0,len(code),500):
code_block = '\n'.join(code[i:i+500]).replace('\t',' ')
self.modules[module].CodeModule.AddFromString(code_block)
except: exception({'module':module,'i':i})
![Page 15: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/15.jpg)
Variable Watch Table VBA Translation Simulation
![Page 16: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/16.jpg)
Simulated Variable value with Original Code
![Page 17: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/17.jpg)
PipeTran Simulation
![Page 18: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/18.jpg)
Clock AC_2010
1 100.000
2 102.000
3 104.000
4 106.000
5 108.000
6 110.000
7 112.000
8 114.000
9 116.000
10 118.000
11 120.000
12 122.000
13 124.000
14 126.000
15 128.000
16 130.000
17 132.000
18 134.000
Predefined SimulationInputs
Historical Plotsof both Inputs and Outputs
![Page 19: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/19.jpg)
� Process Automation refers to the enablement logic that turns on or off the process control.
� By clustering the inputs and outputs and then tracing the genealogy of the inputs to the outputs, the enablement logic can be identified.
Code Analysis
Genealogical Trace
Enablement Logic
![Page 20: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/20.jpg)
� The process control code will need to be broken into blocks of code.
� Then one or more blocks can be replaced with an object.
Migrating to Object Oriented
� Translated code for the blocks can then be sent to the commercial vendor for mapping to their objects.
![Page 21: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/21.jpg)
Next Step
� More research in:
� Code Clone Detection
� Control Module (Object) Refactoring
� Equipment Module (Object with Localized State Based Variables)
� Logic verification on migration
� The PALM project has successfully researched the proof of concept enough to begin implementation of a commercial product.
![Page 22: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/22.jpg)
Lessons Learned
� Python� Development was Fast and readable
� String manipulation was very helpful
� Speed of execution was not a problem
� UI was not created
� Py2exe was problematic
� Win32COM� Versatile and easy to use
� Reliable minus Excel limitations
� Slow
� Not a substitute for Python UI
� PyGraphViz� Easy to Use
� Graphs were not very helpful
� Beyond Compare� Great at comparing Text files
� Bad at comparing Excel files
![Page 23: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/23.jpg)
Python Tricks 1
� Replacing Visual Basic with Python
� Excel calls python as a shell command.
� Python then creates a COM interface back into Excel and executes what ever script the user asks for.
def main():
""" This will call the first argument as a function with the
remaining arguments as the argument for the function """
try:
import sys
if(len(sys.argv) == 1): return
command = sys.argv[1]+'('+','.join(sys.argv[2:])+')'
exec(command)
except: error()
![Page 24: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/24.jpg)
Python Tricks 2
ProxyController.sourceforge.net
![Page 25: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/25.jpg)
Python Pitfalls
� Lists or Dictionaries as default arguments
� def function (arg = []):
� Setting python script time stamp to 0
![Page 26: Using Python to Facilitate Python Facilitating Code Refactoring](https://reader030.vdocuments.mx/reader030/viewer/2022020922/61fb23542e268c58cd5a9832/html5/thumbnails/26.jpg)
Python Pitfalls
� Lists or Dictionaries as default arguments
� def function (arg = []):
� Setting python script time stamp to 0
XKCD.com
EPOCHFAIL