web engineering at ucl
TRANSCRIPT
![Page 1: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/1.jpg)
Web Engineering @ UCL
Research Programming Social – Oct 14 2015Ian Sillitoe (Darwin 627)
![Page 2: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/2.jpg)
About me• Bashing my head against various web technologies
for > 20 years
• What I’ve learned so far:• There is no single, correct way to do things• There are lots of bad ways to do things
![Page 3: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/3.jpg)
![Page 4: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/4.jpg)
![Page 5: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/5.jpg)
Orengo Group• Prof Christine Orengo• Room 627, Darwin• 6 Post docs, 6 PhD students
Computational Biology
Algorithms
Databases
PipelinesWeb
services
![Page 6: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/6.jpg)
Linux Services
• Version Control SVN, Git• Web Tools Apache, Varnish, FCGI, Solr, Tomcat
• Databases PostgreSQL, Oracle, MySQL
• Software Dev Perl, Python, Java, C++, Javascript
• Sys Dev VM, Puppet, CentOS, Ubuntu
20Desktops /
Laptops
6GeneralServers
3Big VM Servers
50 Local HPC
nodes
![Page 7: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/7.jpg)
![Page 8: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/8.jpg)
Background• Technology• HTML / CSS / Javascript
• Concepts• MVC• Web application• Virtualisation
![Page 9: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/9.jpg)
HTML (HyperText Markup Language)
• Provides semantic meaning (markup) to documents• <h1>Title</h1>• <p>Paragraph</p>• <table>…</table>
• HTML5 provides a rich, standardised vocabulary• <footer>…</footer>• <section>…</section>• <audio>…</audio>• <video>…</video>
https://developer.mozilla.org/en-US/docs/Web/HTML
“MDN HTML”
![Page 10: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/10.jpg)
<h1>Title</h1><p>Some text and a <a href="http://www.ucl.ac.uk">link</a></p>
TitleSome text and a link
HTML (HyperText Markup Language)
![Page 11: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/11.jpg)
HTML5<!DOCTYPE html><html lang="en"> <head> <meta charset="utf-8"> <title>Title</title> <link rel="stylesheet" href="style.css"> <script src="script.js"></script> </head> <body> <!-- page content --> <p>Some text and a <a href="http://www.ucl.ac.uk">link</a> </p> </body></html>
![Page 12: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/12.jpg)
CSS (Cascading Style Sheets)
• Language to specify presentation of the document
• Separates data from presentation
• Consistency across many pages
https://developer.mozilla.org/en-US/docs/Web/CSS
“MDN CSS”
![Page 13: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/13.jpg)
CSS (Cascading Style Sheets)
<h1>Title</h1><p>Some <span class="example">text</span> and a <a href="http://www.ucl.ac.uk">link</a></p>
TitleSome text and a link
HTML
CSS
![Page 14: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/14.jpg)
CSS (Cascading Style Sheets)
<h1>Title</h1><p>Some <span class="example">text</span> and a <a href="http://www.ucl.ac.uk">link</a></p>
TitleSome text and a link
p .example {color: red;
}
HTML
CSS
![Page 15: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/15.jpg)
Javascript• Dynamic language• Commonly used to negotiate interaction between
user and HTML• Standard is supported by all modern browsers• JS Frameworks make it easier to use• jQuery, Underscore, Prototype, …
https://developer.mozilla.org/en-US/docs/Web/JavaScript
“MDN Javascript”
![Page 16: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/16.jpg)
Background• Technology• HTML / CSS / Javascript
• Concepts• MVC• Web application• Virtualisation
![Page 17: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/17.jpg)
MVC
Model View
Controller
Database, Filesystem, … HTML, Excel, PDF, …
Glue
![Page 18: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/18.jpg)
MVC
Model View
Controller
Database, Filesystem, … HTML, Excel, PDF, …
Glue
![Page 19: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/19.jpg)
Crowd Computing• Web Service
http://api.ucl.ac.uk/RandomNumberAndColour/2
• Models• RandomNumber• RandomColour
• Views• to_json• to_html• to_html( language => ‘Spanish’ )
![Page 20: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/20.jpg)
MVC
Model View
Controller
Colours: [“red”, “green”]Numbers: [42, 15]
[ { colour: “red”, number: 42}, { colour: “green”, number: 15}]
to_json, to_html, to_pdf, …
![Page 21: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/21.jpg)
Crowd Computing• Web Service
http://api.ucl.ac.uk/RandomNumberColour/2
• Returns
[ { colour: “red”, number: 42}, { colour: “green”, number: 15}]
![Page 22: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/22.jpg)
Crowd Computing• Web Service
http://api.ucl.ac.uk/RandomNumberColour/2
• Returns
red green05
1015202530354045
Number
![Page 23: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/23.jpg)
Crowd Computing• Web Service
http://api.ucl.ac.uk/RandomNumberColour/2
• Returns
red green
![Page 24: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/24.jpg)
Challenge 1• Provide web application that is:• Efficient not CGI.pl• Robust no downtime, no bad links• Flexible encourage new features• Scalable deal with traffic, future proof• Maintainable > 1 developer
![Page 25: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/25.jpg)
Efficient• Use MVC web application (not CGI.pl)
• e.g.• Perl Catalyst, Dancer, Mojolicious, …• Python Django, CherryPy, …• Ruby Rails, …
![Page 26: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/26.jpg)
Robust• Continuous Integration (CI)• Everything gets put into group code repository
• code, crontabs, tests, etc• All code has its own tests• Tests run automatically all the time• If tests PASS - shared code gets updated• If tests FAIL - we get an email
• e.g. Jenkins CI, Travis CI, …
![Page 27: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/27.jpg)
Flexible• Lots of (good) tests gives you… • Confidence to make changes• Confidence to accept changes from others
• Continuous Integration lets you…• See your own changes in “beta” sites
• MVC web applications make it easy to…• Create powerful, interactive web pages
![Page 28: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/28.jpg)
Scalable• Move web code to Virtual Machine (VM)• SVN all common code in central repo• Puppet maintains machine requirements• Varnish web cache, load balance, failover
![Page 29: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/29.jpg)
VM server
Code Repo (SVN)
orengobuild64
orengocatalyst01
orengocatalyst02
orengocatalyst03
User 2
User 1
Develop, Build / Test, Deploy
DEVELOP BUILD / TEST
DEPLOY
![Page 30: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/30.jpg)
VM server
Web server
Varnish
DataSource
A
DataSource
B
WEB
orengobuild64
orengocatalyst01
orengocatalyst02
orengocatalyst03
Cache, Load Balance, MVC
CACHE
LOAD BALANCE(and FAILOVER)
DataSource
C
![Page 31: Web Engineering at UCL](https://reader035.vdocuments.mx/reader035/viewer/2022070514/587c83321a28ab93568b5ac3/html5/thumbnails/31.jpg)
Challenge 2• Monitor SGE jobs (qstat without the typing)
• https://github.com/sillitoe/sge-monitor