cs 430p/530: internet, web, & cloud systems

APIs/abstractions

Previously

Abstractions for infrastructure to ease operations (Ops) Manage complexity in deploying software systems

Examples covered… VMs, containers

Infrastructure as a Service

Platform as a Service

Functions as a Service

Infrastructure as Code

Portland State University CS 430P/530 Internet, Web & Cloud Systems

But, what about the apps themselves?

Emerging software abstractions for building functionality

Example: Your homework (e.g. Flask) Common web application abstractions as a framework

No need to program sockets, HTTP, cookies, sessions, routing, HTML

generation, database communication, etc.

Imagine writing it in C (from scratch!)


Historically

From Unix to Windows in 90s/00s


To this, 20 years later

What happened?


Currently this…on the front-end

Mobile apps (Android/iOS SDKs)


Web apps (client-side JS frameworks)

Or both Progressive web apps

Seamless off-line/on-line experience

Native/hybrid applications


Meanwhile…on the backend

Server-side web and API frameworks

Data and event management


Now this…

Software as a service via APIs

Comprehensive lists at https://rapidapi.com/


Even traditional businesses are building them…


https://www.companynewshq.com/company-news/personal-finance-utilities-company-news/wells-fargo-surpasses-one-billion-api-calls/

https://news.lendit.com/2020/02/12/apis-become-core-to-wells-fargo-strategy/

Others…

Payments (PayPal, Venmo)

Social media (Twitter, Facebook, Instagram, Pinterest, Spotify)


Multi-service instant

messaging and

communications (Twilio,

Nexmo) Communicate with

customers using whatever

they have!


IoT (LiFX, GCP IOT Core, AWS IOT Core)


Food and nutrition APIs (Nutritionix)


Spoonacular


On GCP…

Maps API, Search API, Knowledge Graph API, Analytics, Ads, Voice

transcription, Image analysis, Video analysis, Translation, etc.


Accessing APIs

#1: Python packages

(if they exist)

PyPI

Python Package Index Packages installable via pip (as we have been doing all quarter)

Most popular web APIs come with supported Python packages

Many are auto-generated via Swagger

Note Code examples are written for simplicity

NEVER specify API keys within source code


Example: Giphy

pip install giphy_client


import giphy_client

# Create app and API_KEY at https://developers.giphy.com# Note insecure pattern: Should supply from environment variablesAPI_KEY = os.environ.get('YOUR_GIPHY_API_KEY')

api_instance = giphy_client.DefaultApi()api_response = api_instance.gifs_search_get(API_KEY,

'clapback', limit=1, rating='g', lang='en', fmt='json')if len(api_response.data) > 0:

print(api_response.data[0].embed_url)

https://developers.giphy.com/

Example: Spotipy (Spotify)

pip install spotipy


import spotipyfrom spotipy.oauth2 import SpotifyClientCredentials

# Obtain credentials via https://developer.spotify.com# Pass to client via environment variables that are not committed to gitCLIENT_ID=os.environ.get('SPOTIFY_CLIENT_ID')CLIENT_SECRET=os.environ.get('SPOTIFY_CLIENT_SECRET')

# Return a spotify playlist given querycredentials = SpotifyClientCredentials(client_id=CLIENT_ID,

client_secret=CLIENT_SECRET)spotify = spotipy.Spotify(client_credentials_manager=credentials)results = spotify.search('Foo Fighters', limit=1, type='playlist')if len(results['playlists']['items']) > 0:print(results['playlists']['items'][0]['external_urls']['spotify'])

https://developer.spotify.com/

#2: API discovery

OpenAPI specifications

OpenAPI initiative (Linux Foundation) Formerly known as Swagger (swagger.io)

"Standard, language-agnostic interface to REST APIs which allows

humans and computers to discover and understand the capabilities fo the

service without access to source code, documentation, or through

network traffic inspection"

Specify REST API and its semantics in standard way Code parses specification to automatically build calls that use it


https://github.com/OAI/OpenAPI-Specification

Google's API Discovery format

API discovery format Supports both REST APIs with OpenAPI specifications as well as gRPC's

generated API clients Recall gRPC (open-source RPC framework) supporting zero-copy, binary data

transfer with specifications written in Protocol Buffers language

More info at https://googleapis.github.io/

OpenAPI/Swagger tools here


https://googleapis.github.io/

https://github.com/APIs-guru/google-discovery-to-swagger

Example

In-platform access of storage API via apiclient.discovery Avoid use of key management for accesses done completely within GCP

pip install google-api-python-client google-auth


import apiclient.discoveryimport google.auth

# Obtain credentials and instantiate APIcredentials, project_id = google.auth.default()gcs_api = apiclient.discovery.build('storage', 'v1', credentials=credentials)

# Obtain credentials and instantiate APIgcs_api.objects().insert(bucket='wuchang-bucket',

name='myfile.txt',media_body='./myfile.txt').execute()

YouTube API via apiclient.discovery Access from anywhere, but requires API key


# https://developers.google.com/youtube/v3/docs/search/listimport apiclient.discoveryAPI_KEY = os.environ.get("API_KEY")

# Build the request for YouTube search contentyoutube = apiclient.discovery.build("youtube",

"v3",developerKey=API_KEY)

search_response = youtube.search().list( q = "Grumpy Cat",part = "id, snippet",maxResults = 1, ).execute()

for search_result in search_response.get("items", []):if search_result["id"]["kind"] == "youtube#video":print("%s" % search_result["snippet"]["title"])print("%s" % search_result["id"]["videoId"])

https://developers.google.com/youtube/v3/docs/search/list

Knowledge Graph API via apiclient.discovery

Translation API via apiclient.discovery


import apiclient.discovery, os

API_KEY = os.environ.get('API_KEY')

kgsearch = apiclient.discovery.build('kgsearch', 'v1',developerKey=API_KEY)

request = kgsearch.entities().search(query='Portland State University',limit=1)

# Returns a dict of search result inforesult = request.execute()

import apiclient.discovery, os

translate = apiclient.discovery.build('translate', 'v2',developerKey=os.environ.get("API_KEY"))

request = translate.translations().list(source = 'en',target = 'fr',q=['flower','car'] )

result = request.execute()

#3: HTTP requests to REST API

Language support for HTTP

Most languages with native support for sending HTTP requests and

parsing HTTP responses Typically, GETs and POSTs with JSON being returned

Packages are wrappers for the underlying REST API call over HTTP

Python Requests (My) preferred package for HTTP

Most examples will use this


Example: Google Places API (REST via Python)


import requests

def post(self):""" Sends search box information to Google Places API and returns response """api_key = os.environ.get('API_KEY')keyword = request.form['search']

# Base URL for Google Places APIurl = 'https://maps.googleapis.com/maps/api/place/nearbysearch/json?'

# Lat/Long of Portland, ORlocation = 'location=45.5122,-122.6587&radius=1500'

# Places type variableplace = 'restaurant'

# API responseresponse = requests.get(url + location + '&type=' + place +

'&keyword=' + keyword + '&key=' + api_key)r = response.json()return render_template('home.html', response=r['results'])

Example: Knowledge Graph API (REST via Python)


import requests

# Grab Knowledge Graph API Key from APIs & Services:Credentials and# set environment variableapi_key = os.environ.get('API_KEY')

term = 'Portland State University'

# Use term and api_key string to formulate a URL for hitting the# Knowledge Graph API endpoint. Note f-strings are Python 3.6url_string = f'https://kgsearch.googleapis.com/v1/entities:search?query={term}&key={api_key}&limit=1&indent=True'

# Take given response, parse it as a json object, and return a dictresponse = requests.get(url_string)

result_dict = response.json()

Example: Yelp API


import requestsdef yelpSearch(self, title):

# Craft Yelp's API endpoint URL, set location to Portland and limit resultsurl = 'https://api.yelp.com/v3/businesses/search?term=' + title +

'&location=portland&limit=3'

# Declare the api header with api key (Fill in your own key)headers={'Authorization': "Bearer YOUR_API_KEY"}

# Query the API endpoint, parse JSON response and return dictresponse = requests.get(url, headers=headers)return response.json()

Side-by-side (Google Maps)

pip install googlemaps

pip install requests


# Requires enabling geocoding API and generation of an API key in Google Cloudimport googlemapsimport os

API_KEY = os.environ.get('API_KEY')gmaps = googlemaps.Client(key=API_KEY)boba_query = 'Mo Cha Tea House, Portland, OR'boba_shops = gmaps.geocode(boba_query)for shop in boba_shops:print(shop['formatted_address'])

import requestsimport jsonimport os

API_KEY = os.environ.get('API_KEY')boba_query = 'Mo Cha Tea House, Portland, OR'resp = requests.get( f'''https://maps.googleapis.com/maps/api/geocode/json?address={boba_query}&key={API_KEY}''')boba_shops = json.loads(resp.text)for shop in boba_shops['results']:

print(shop['formatted_address'])

Side-by-side (Nutritionix)

pip install nutritionix nutritionix-client

pip install requests


from nutritionix import NutritionixClientrecipe = 'Pasta Salad'

# Get the credentials from environment variables (more secure than in code)APP_ID=os.environ.get('NIX_APP_ID')API_KEY=os.environ.get('NIX_API_KEY')

nut_info = NutritionixClient(application_id=APP_ID, api_key=API_KEY)result = nut_info.search(q=recipe, limit=3, offset=0)

def nutritionix(self, ingredient):url = 'https://trackapi.nutritionix.com/v2/natural/nutrients'# Declare the api header with api key (Fill in your own App ID and Key)headers = {"Content-Type":"application/json", "x-app-id":"YOUR_APP_ID",

"x-app-key":"YOUR_APP_KEY"}

# Declare POST query as a dictionary, set query field to ingredientbody = {"query":ingredient,"timezone": "US/Eastern"}

# Make request to API endpoint, parse JSON response and return dictresponse = requests.post(url, headers = headers, json = body)return response.json()

AWS API Gateway

Building and deploying APIs

Types of Web APIs

RESTful APIs (request-response) Online transactions

Weather application

Blogs, social media

Via HTTP methods PUT, GET, POST, DELETE, etc.

Take in and output HTML, XML, or JSON

WebSocket APIs (bi-directional) Chat applications

Real-time alerts and notifications


API Gateway

Make it easy to scale and manage versions of an API Fully managed, serverless API platform

REST and WebSocket support

Lifecycle management support for APIs


API Gateway overview

https://www.alexdebrie.com/posts/api-gateway-elements/


1. Authorization

and access

control

Routing to

endpoint,

access control

Packaging

request to API

logic in

integration

Transform response, format

HTTP response headers

2. Request flow

3. Response flow



1. Authorization and access control

User access management to API IAM-based, resource-based policies

AWS Cognito

Custom (e.g. Authorizer built in a Lambda)

OAuth/SAML

User API key generation and management

Usage-plans for billing and throttling Per-API key metering

Usage per second or daily quota

Configuration of external site access via Cross-Origin Resource

Sharing (CORS) Restrict sites that can access API via cross-domain requests

TLS certificate management for endpoint


2. Request Flow

Endpoint configuration, request routing based on HTTP method

Method request Validation of request parameters

Validation of request bodies Done via pre-defined models (data structure for specifying request and response

payloads in a JSON Schema format)

Integration request Transformation to format method request into backend integration

request (via request mapping templates)

Integration invocation to perform API logic

Integration Implementation of API logic (EC2, ECS, Lambda, etc)


Example model and mapping templates

Cost calculator request model that contains a price, size, and unit API Gateway ensures requests supply all parts of the model

Mapping templates Transform from one model to another

For requests, translate method request model into integration request

model for use by backend integration


3. Response Flow

Transform integration responses and return values back into HTTP

response Done similarly to request flow via response mapping templates and

method response models


API Gateway miscellany

Caching support (via CloudFront) Endpoint types

Edge (Latency optimized via deployment in CloudFront) Regional (For requests and responses residing in the same AWS region) Private (Exposed only to services and resources inside your VPC)

Stages Snapshots or versions of the API available for clients to call Create an API deployment and associate it with a stage (stage settings include

caching enabled, request throttling, logging config) SDK generation

Automatically generate language SDKs (Java, JavaScript, Objective-C, Swift, and Ruby)

CloudWatch monitoring built-in (metrics collected per minute) Count Integration latency CacheHitCount CacheMissCount HTTP 400 and 500 errors


Labs

Leverage APIs to build applications (Slack/Knowledge Graph)

Manually deploy APIs (Guestbook) A lot of work!

Next week IAC + API deployment

ML APIs

Backend as a Service APIs


cs 430p/530: internet, web, & cloud systems

Documents