EECS E6893 Big Data AnalyticsHW4: Data visualization with web dashboard

Hritik Jain, hj2533@columbia.edu

111/24/2020

Agenda● Introduction to Web Development

○ HTML, CSS and Javascript○ 2 important things to know: SVG and DOM

● D3.js demo for data visualization● HW4 Part 1(35% of HW4)

○ Short-answer Questions○ Draw a simple barchart

● Introduction to Django● D3.js demo for data visualization● HW4 Part 2 (65% of HW4)

○ Dashboard of HW3○ Required results from HW2

2

I should preface the tutorial material by saying that if you are primarily interested in web development, this tutorial is hardly enough. What we really focus on is visualizing the results we obtained in our previous two HWs on a web application. So to have the required background, we will review the basic front end technologies - HTML, CSS, Javascript and then two key ideas of SVG and DOM. Then we will talk about d3 that you will need spend time on a little bit beyond this tutorial to learn the core concepts.

Web Application

3Reference: https://hackr.io/blog/web-application-architecture-definition-models-types-and-more

So any website you want to visit on the web, you enter a url in the address bar, the browser requests for that particular web address. The server that is identified through the address then sends files to the browser as a response to the request. The browser then executes those files to show the requested page.

The code that is returned to the browser and responds to some user input is the client side code, the end-user can see it as well as change it.

Now, the developers of the web application decide as to what the code on the server will do with respect to the code in the browser. The server-side code is responsible for creating the page that the user requested as well as storing different types of data, including user profiles and user input, business logic etc. The end-user can’t see it.

Client-side of Web● HTML, CSS and JS are the parts of all websites that users directly interact

with. ● HTML provides the basic structure of sites, which is enhanced and modified

by other technologies like CSS and JavaScript.● CSS is used to control presentation, formatting, and layout.● JavaScript is used to control the behavior of different elements.

4Reference: https://tutorial.techaltum.com/webdesigning.html

Html css are used on the client side for how the webpage looks. Javascript is a scripting language, it’s a very powerful tool that can do many things for a website. Like it enables you to add interactivity. You can build rich UI components such as image sliders, pop-ups, site navigation mega menus, form validations, tabs, accordions, and much more.

HTML

5Reference: https://www.w3schools.com/html/default.asp

HTML describes the structure of a Web page. It consists of a series of elements - these elements tell the browser how to display the content. All that you see here - heading 1, body, paragraph these are elements. An HTML element is defined by a start tag, some content, and an end tag, some don’t always require that. They can be nested as well. Then there are elements - all HTML elements can have attributes. attributes provide additional information about elements and they are added in the tags.

HTML DOM

6

Document Object Model (DOM) is important: With the HTML DOM, JavaScript can access and change all the elements of an HTML document

Reference: https://developer.mozilla.org/en-US/docs/Web/API/HTML_DOM_API

We can think of the HTML representing a web page as a document that has a hierarchical structure, you know with nested elements. So Document Object Model or DOM represents such a document with a logical tree. Each branch of the tree ends in a node, and each node contains objects. DOM methods allow programmatic access to the tree. With them, you can change the document's structure, style, or content. That’s where javascript comes in.

Javascript with HTML DOM

7

● With the HTML DOM, JavaScript can access and change all the elements of an HTML document

● HTML DOM methods are actions you can perform (on HTML Elements).

● HTML DOM properties are values (of HTML Elements) that you can set or change.

8

Javascript with HTML DOM

getElementById is the method and innerHTML is the property

SVG in HTML

9

● SVG stands for Scalable Vector Graphics. It is used to define vector-based graphics for the Web

● Every element and every attribute in SVG files can be animated● SVG integrates with other W3C standards such as the DOM and XSL

Reference: https://developer.mozilla.org/en-US/docs/Web/SVG

Now, Scalable Vector Graphics (SVG) are an XML-based markup language for describing two-dimensional based vector graphics it's a text-based standard for describing images that can be rendered at any size and are designed specifically to work well with other web standards including CSS, DOM, JavaScript. We’ll talk about DOM in a bit. SVG is, essentially, to graphics what HTML is to text. The HTML <svg> element is a container for SVG graphics.

What if we create a SVG elements and use DOM in Javascript to access its attributes?

10

If we draw a series of SVG and texts based on data, and use DOM to control their attributes, then we get a simple charts!

- D3.js, a library to do this in a simple way

This is an essence of how you add interactivity to a webpage right.

D3.js

11

D3.js is a JavaScript library for manipulating documents based on data. It helps you bring data to life using HTML, SVG, and CSS. It provides a data-driven approach to DOM manipulation.

Visit https://d3js.org for more tutorials!

So d3 provides a data-driven approach to DOM manipulation. Again, you would want to look into the tutorials. Here’s a quick illustration on the important ideas.

12

Declaration of data and variables

D3 provides operating on arbitrary sets of nodes called selections. You can manipulate individual nodes and set the attributes

Data joins. Once you bound data with selection, each element in the data array is paired with the corresponding node in the selection. Tricky part of D3: If there are fewer nodes than data, you can use enter() to append nodes.

Again, please visit https://d3js.org for more tutorials!

On the top you see declaration of variables. Next, the idea of selections, with selections, it lets us to grab a set of one or more elements of the webpage, ie the nodes of a DOM. You will use selections to modify, append, or remove elements in a relation to the predefined dataset.

That leads to the concept of data binding. Ultimately you are visualizing data right. The power of d3 really lies in binding data to elements, svg elements in particular.You see this data() operator here in the third code snippet, you are passing our dataset that we defined above in a list to the operator (in the hw you will read this data from bigquery). And you are using that on a selection of DOM elements, all the svg elements of the type rectangle. Doing this you join both sets of items - the svgs and the numbers in the list with each other item by item, it’s called a data join and then you do something with them.

But the interesting this is you can do this data join even when you have more data items than the selected dom items. That’s where it gets a little tricky. When the dataset contains more items than there are available DOM elements, like through the interactivity allowed through javascript you can remove svg elements from the DOM. So the surplus data items are stored in a subset of this selection called the enter selection. Then you call append to create new dom elements.

Similarly, there is an exit selection for the case when you more dom elements in the selection than the joined dataset. You can call remove() to clean up the DOM elements.

13

Example: Simple Bar Chart

The power of d3 really lies in binding data to elements, svg elements in particular.

Django

14

● Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design.

● Takes care of much of the hassle of Web development

● Free and open source.

https://www.djangoproject.com

https://docs.djangoproject.com/en/3.1/intro/tutorial01/

Now when you develop web applications, there is the server side, and there are overhead associated with common activities performed, for example, a database, database access, session management, etc

Web frameworks aim to automate these activities for you. Django is one such popular framework.

We will django in this HW for part 2 to build the dashboard application. I will be uploading another tutorial soon which is more like exact step by step instructions for setting up a django app, that you can refer to when you begin. Most of this code for setting up a django app will be given to you. For completing the exercise, your primary task here is to write or modify the d3 javascript code that will display the visualizations of previous results on a webpage.

That said, you may want to consider frameworks like django and flask for your project if you build a web application, so it’s encouraged that you read up the documentations in detail by yourself, for these exercises you will only need a basic familiarity.

HW4

● Part 1 (35%)○ 4 short-answer questions (5% each)○ Draw a simple bar chart as per instructions (15%)

● Part 2 (65%)○ Create a Django project○ Query the HW3 data from bigquery○ Draw a Dashboard based on the data, showing a bar chart and a pie chart (20% each)○ Process the results from HW2 and store the required data to BigQuery○ Create a simple visualization to show connections between nodes of a connected component

15

References● https://d3js.org● https://developer.mozilla.org/en-US/docs/Web/SVG/Tutorial● https://www.w3.org/Graphics/SVG/IG/resources/svgprimer.html ● https://developer.mozilla.org/en-US/docs/Web/SVG/Attribute● https://jtr13.github.io/d3book/index.html

16

Thanks!

17