Create a Custom Street View Using Google Maps APIs and Google Cloud

Platform

Whitepaper Last updated: October 2016

Contents

About this document Obtaining spherical panoramas

Camera details Photos details (resolution and quantity)

Create panorama tiles Download ImageMagick software to manage panoramas Edit photos

Map the panoramas path Work with different resolutions Map the path Call panoramas using the Google Maps API

Follow the “pancake” Function A: Create a custom ellipse Function B: Generate horizon parameters Function C: Simulate the click Function D: Simulate movement Function E: Update horizontal attributes

Create a panorama management tool (optional) Store floorplans in Google Cloud Storage Store processed panoramas in the Google Cloud Storage

Build the web application

2

About this document This document describes how to generate the panorama files necessary to implement Custom Street View using Google Maps APIs and Google Cloud Platform. While there are many hardware options to build these panoramas yourself, you can also contact a local trusted photographer or agency under the Google Business View project. Note: Google Business View should only be used for external applications—all panoramas are published on maps.google.com for public access. If you intend to build an internal application, you should consider purchasing a Google Maps license per our Terms of Service (9.1.1 General Rules; Public Access - No Firewall), but you also can leverage the trusted photographer or agency to help to generate the panorama. The information and recommendations in this document were gathered through our work to implement an indoor map at SENAC University in Brazil and also a variety of clients and environments in the field. We thank our customers and partners for sharing their experiences and insights, but especially Geoambiente Sensoriamento Remoto (partner based in Brazil) that helped us to put together their feedback about this implementation; special thanks to Alex Vidotto, Lucas Tenório and Mateus Pontes. The details in this document are specific to SENAC and describe their environment to build and process these panoramas.

Prerequisite Resources ● Access to a Google Maps API license ● Access to the appropriate Google Developers console ● Access to Google App Engine ● Access to Google Cloud Storage or Google Drive

Prerequisite Software The following software are required to manage panoramas/tiles and develop the application. ● IDE of your preference ● ImageMagick®—Open Source software that can be downloaded

here. This is a software studio to create, edit, compose, or convert bitmap images, and also can be easily configured by command line or your favorite language.

3

Camera details

The hardware selected for the SENAC project was Ladybug with the following details: ● Camera: Ladybug 5 ● Panorama photos that were taken: about 6,000 photos

The agency placed the camera in an adapted cart (see figure below) to navigate around the campus and Ladybug generated a movie that was automatically transformed as 360 degree panoramas. In this case, the panorama didn't come with georeferenced coordinates, and it was manually set for each environment (room, laboratory, and so on).

Figure 1. Capturing panoramas photos with an adapted cart

Photos details (resolution and quantity)

As result, the following panorama details were noted: ● 1,100 panorama photos were created. ● For the first phase of the project, it covered 3 SENAC campus sites = Santo Amaro, Campos do Jordão

and Águas de São Pedro. ● Each tile has 512 x 512 pixels. ● Each panorama follows the specification based on Google Maps API documentation; for example, 2:1

proportion.

Create panorama tiles To improve the performance and guarantee a seamless user experience, you need to standardize the resolution and generate panorama tiles.

4

Download ImageMagick software to manage panoramas

As noted earlier, to manage these panoramas, we used ImageMagick software primarily because it's possible to automate the generation of these tiles through the command line. It works standalone and was run on a Windows® 7 Pro, Core i3, 4 GB RAM machine.

Edit photos

To maintain the same resolution for all panoramas, you need to use a normalized script; for example: Resize resolution Ex.: High Resolution const string CONVERT_RESIZE_COMMAND = @"""{0}"" -resize 7168x3584 -quality 50 ""{1}""" ; Ex.: Medium Resolution const string CONVERT_RESIZE_COMMAND2 = @"""{0}"" -resize 4096x2048 -quality 50 ""{1}""" ; Ex.: Low Resolution const string CONVERT_RESIZE_COMMAND3 = @"""{0}"" -resize 2048x1024 -quality 50 ""{1}""" ; where the quality parameter represents the resolution of the photos. Experiment with the resolution to see what's better for your user, because sometimes there are problems with quality of the end user internet and the download can take more time than expected. Generate Tiles This example was developed on an engine that iterates with the photos stored in the folder and for each interaction, it runs the following command line on ImageMagick:

const string CONVERT_CROP_COMMAND = @"""{0}"" -crop 512x512 -set filename:tile

""%[fx:page.y/512+1]_%[fx:page.x/512+1]"" +repage +adjoin

""{1}\{2}\tile_%[filename:tile].jpg""" ;

string.Format ( CONVERT_CROP_COMMAND, tempPath, Path.GetDirectoryName(tempPath),

Path.GetFileNameWithoutExtension(tempPath)) ;

where: {0} = "C:\PanoramasPhotos\panorama1.jpg" - original file {1} = “C:\PanoramasPhotos” - root folder {2} = “panorama1” - original filename that will store the tiles for an specific panorama

5

Figure 2. Result of generated tiles for the same panorama (id=000060)

Map the panoramas path This section explains how to map the panorama’s path, including information where the tiles were stored and should be read for the JavaScript® (Google Maps API). For example: http://…../panoramas/resol/Tile_1_1.jpg Each panorama has a specific ID (automatically incremented) with its respective tiles.

Work with different resolutions

For the same folder, two kinds of imagery were created for the same tile: high and low resolutions. The goal here is to guarantee that the user has a good navigation experience, whether it’s in a browser or on a mobile device. Note: You can configure this setup directly on JavaScript by identifying the client's resolution (browser, mobile).

6

More details can be found on the topic 'Call panoramas using the Google Maps API'.

Map the path

To map two important parameters, consider the relation between these paths:

● Relation between location (environment: room, gym, and so on) and the panorama (ID) ● Relation between panoramas and tiles (arrows linking both of them)

This JavaScript Object Notation (JSON) can be stored directly on the user's browser each time the user opens the website or it can be downloaded on demand based on each panorama transition. The following example just shows the mapping of a specific panorama that repeats every time a new panorama is added to the solution: // Panorama Array: var panoramaSource = [ { // Panorama that is currently showing "PanoId": "348_ladybug_panoramic_000102", "Description": "Main Hall", // Autocomplete Tags "Tags": [

"University Path", "Entrance", "CAP"

], // Array of Coordinates "Latitude": "-22.596848", "Longitude": "-47.880689", // List of all the panoramas that can be navigated from the current imagery (angle

attribute will be better explained on pancake topic) "Links": [

{ "PanoId": "349_ladybug_panoramic_000114", "Angle": -2

}, { "PanoId": "369_ladybug_panoramic_000016", "Angle": -168 }

], //campus is an ID for each campus "Campus": "3", //It was defined a north direction for each panorama "Heading": 56 }, ... ];

7

Call panoramas using the Google Maps API

Based on Google Maps API documentation, the following example was adapted to represent all previous configurations on the files scheme:

function initialize() {

// Checks client resolution var Layout = {}; Layout.worldSizeX = "7168"; if (goog.userAgent.product.ANDROID || goog.userAgent.product.IPHONE) { Layout.worldSizeX = "2048"; Layout.txtPesq.toggleClass("miniTXT"); } else if (goog.userAgent.product.IPAD)

Layout.worldSizeX = "4096";

Layout.worldSize["7168"] = new google.maps.Size(7168, 3584); Layout.worldSize["4096"] = new google.maps.Size(4096, 2048); Layout.worldSize["2048"] = new google.maps.Size(2048, 1024);

// Set up Street View and initially set it visible. Register the // custom panorama provider function. Set the StreetView to display // the custom panorama 'reception' which we check for below. var panoOptions = { pano: 'reception', visible: true, panoProvider: getCustomPanorama }; var panorama = new google.maps.StreetViewPanorama(

document.getElementById('map-canvas'), panoOptions); } // Return a panorama image given the panoID function getCustomPanoramaTileUrl(pano, zoom, tileX, tileY) {

var picName = Layout.worldSizeX + '__Z__' + pano + '__Z__tile_' + (tileY + 1) + '_' + (tileX + 1) + '.jpg';

return "http://storage.googleapis.com/example_directory/" + picName; } function getCustomPanorama(pano) {

var valido = false; panoKeys = $.map(panoramaSource, function (panoObj) { return panoObj.PanoId; }); for (var i = 0; i < panoKeys.length; i++) {

8

if (panoKeys[i] == pano) { currentPano = pano; valido = true; break; } } if (pano == panoInit) {

currentPano = pano; valido = true;

} if (!valido) {

pano = currentPano; } objPanorama = panoramaSource.where(function (x) { return x.PanoId == pano; })[0]; return { location: { pano: objPanorama.PanoId, description: objPanorama.Descricao, latLng: new google.maps.LatLng(objPanorama.Latitude, objPanorama.Longitude) }, links: [], copyright: '<a href="http://www.geoambiente.com.br" target="_blank" style="text-decoration: none; cursor: pointer; color: rgb(68, 68, 68);">' + panorama.getPano().substr(0, panorama.getPano().indexOf("_")) + ' | Geoambiente</a>', tiles: { tileSize: new google.maps.Size(512, 512), worldSize: Layout.worldSize[Layout.worldSizeX], getTileUrl: getCustomPanoramaTileUrl } };

}

9

Follow the pancake There is no out-of-box interface in the Google Maps API to give the same experience as Street View to allow users to view the path and select the following panorama as shown below. The white arrow is called the pancake and it helps the user follow a specific direction on a custom Street View.

Figure 3. Public Street View with arrow inside the pancake

10

Figure 4. Custom Street View with arrows and no pancake To create the pancake:

1. Create the custom Scalable Vector Graphics (SVG) element that represents the pancake.

For this example, the jQuery library is selected because it has Document Object Model (DOM) manipulation functions. The pancake is typically an SVG element (ellipse) attached with a mousemove event and click event (Function A).

2. Set the variables of the pancake (Function B). 3. Attach the mousemove event:

For each triggered mousemove event attached by Function D, the SVG properties are updated, simulating the mouse movement and the experience of "horizon": “ry” (vertical radius), “rx” (horizontal radius), “cy” (vertical center), and “cx” (horizontal center). The update of the ellipse properties (“ry”, “rx”, "cy", “cx”) is done based on a pov_changed event triggering Function E.

4. Attach the click event (Function C): The pancake is HTML-based and each time the user clicks the object, the following logic is applied:

a. Identify the degree (using pov.heading attribute) based on a North position where the click has happened (we recommend to set the North position as the panorama’s center).

b. Using Function C, identify which link (arrow), available in the current panorama, has the approximate degree from the degree of the last step. (Give the user the perception to keep walking on the same direction.)

5. At this step, using the Developer Tools available in the browser that you're developing (for example, Chrome DevTools), inspect the elements and locate the original Google Maps Street View ellipse. You get the original SVG object (see Figure 5).

11

Figure 5. Inspecting the DOM elements

6. Retrieve the ellipse (dynamically created in Step 1) and append it to the original SVG.

Function A: Create a custom ellipse

$svg = $("<svg id=\"ellipse\" version=\"1.1\">" + "<ellipse fill=\"white\" fill-rule=\"evenodd\" visibility=\"visible\" " + "cx=\"0\" cy=\"0\" rx=\"0\" ry=\"0\" fill-opacity=\"0.4\"></ellipse></svg>");

Function B: Generate horizon parameters

$ellipse = $svg.find('ellipse'); $panel = $("#map-canvas"); $status = $("#status"); $pov = $("#pov"); $eventpan = $("#eventpan"); windowY = $(window).height();

maxw = $panel.width();

maxy = $panel.height();

limitClickToGo = (0.30 * maxy + $(window).height() / 2);

currentLimitClickToGo = limitClickToGo; limitHorizont = (0.03 * maxy + $(window).height() / 2);

currentHorizont = limitHorizont;

Function C: Simulate the click

$ellipse.on('mousedown', function () { oldPov = panorama.getPov();

}).on('mouseup', function (evt) { var pov = panorama.getPov(), nextPano = null;

action = (oldPov.heading == pov.heading && oldPov.pitch == pov.pitch && oldPov.zoom

== pov.zoom) ? "CLICK" : "PAN";

if (action != "CLICK") return; if (!(evt.clientY < currentLimitClickToGo)) return;

nextPano = getNearestPano();

panorama.setPano(nextPano.pano)

/* *

*/

function getNearestPano() { var pos = false, h = pov.heading,

g = null, pos = pov.heading > 0,

links = panorama.getLinks();

/*

12

*

*/

for (var i = 0; i < links.length; i++) { if (!g || Math.abs((!pos && !links[i]['heading'] ? 360 : links[i]['heading']) -

h)

< Math.abs((!pos && !g.heading ? 360 : g.heading) - h)) {

g = links[i];

}

}

return g; }

})

Function D: Simulate movement

// sets the variables that will be used on mousemove event and pov_changed var entryPanoId = null, maxw = 0,

maxy = 0,

oldPov = null, windowY = 0,

limitClickToGo = 0,

currentLimitClickToGo = 0,

limitHorizont = 0,

currentHorizont = 0,

pancakeLimit = 0,

$svgPath = null, $line = null, $horizont = null, $pov = null, $status = null, $ellipse = null, $panel = null, $svg = null, panoInit = "eLfLrs3ASmb0vicaYOjIZg", currentPano = null, // calculations to define what will be the size of the ellipse

mousemoveHandler = function (evt) { var diff = ((evt.pageY - pancakeLimit) / 2); if (diff <= 0) { $ellipse.attr('visibility', 'hidden'); return; } if ((evt.pageY - maxy) <= 20) { $ellipse.attr({ 'ry': diff / 5, 'rx': diff + 20, cy: evt.pageY - 55 }); }

$ellipse.attr({ visibility: 'visible', 'cx': evt.pageX }); },

13

Function E: Update horizontal attributes

// function to update the panorama pov_changed object // ex: google.maps.events.addListener(panorama, ‘pov_changed’, pov_changed) pov_changed = function(){

var pov = panorama.getPov(), windowY = $(window).height(),

temp = Math.floor(limitClickToGo + pov.pitch * 5),

tempHorizont = limitHorizont + pov.pitch * Math.ceil(windowY / 60);

if (temp < maxy / 2 + windowY / 2 && temp > windowY / 2 - maxy / 2) { currentLimitClickToGo = limitClickToGo + pov.pitch * 5;

$line.css({ top: currentLimitClickToGo });

}

if (tempHorizont < maxy / 2 + windowY / 2 && tempHorizont > windowY / 2 - maxy / 2)

{

currentHorizont = tempHorizont;

$horizont.css({ top: tempHorizont });

pancakeLimit = (tempHorizont - windowY / 2 - maxy / 2) + maxy;

}

}

Create a panorama management tool (optional) There’s no out-of-box interface to create and map a JSON (arrayPanoramas) on Google Maps API. One suggestion is to create an specific interface that can do that.

● In this step, we will create an interface that's responsible to read a standard file (it can be *.txt, *.xml) telling the link between the panoramas (for each environment = room, gym, and so on) for example, image 1 to image 2, and so on.

● This tool will read this information, display on an user friendly interface, and allow the administrator to manually update the tiles. It's possible to save and automatically generate the JSON file.

14

Figure 6. Interface to manage, link, and define the panoramas paths

To manage the panorama interface (administrator):

1. Select a specific panorama, which is displays in the center of the screen. 2. Figure 6, the red arrow represents the direction that the user navigates. Select one of the panoramas

tile on the list (on the right) and click Add link. 3. A navigation link is then generated between these 2 panoramas. 4. When done, click the Get heading option that the JSON is generated.

15

Store floorplans in Google Cloud Storage This section describes how to store and visualize the floorplan (displayed at top right). In Figure 7, the floorplan is a single image above the map, named as overlay . There's a class on Maps API specifically used to manage overlays.

Figure 7. Overlay example

For each campus, store an image. In this case, 3 campi = 3 images.

Figure 8. Campi of SENAC As a first step, you need to store the images either locally or in the cloud. For the cloud, you have many vendors or options; for example, Google Drive or Cloud Storage. Google Drive would be simpler, but once you set up a Google Cloud Platform account, you can check all other products Google offers.

16

To store floorplans in Google Cloud Storage:

1. Create your account in Google Cloud Platform. a. Click Free Trial (blue button). b. Select Google Cloud Storage as the product.

Keep this account handy for building an application using Google App Engine.

2. Activate Google Cloud Storage: Follow these steps.

3. Upload files by using the Developers Console (see steps below and in link.) (You could also use gsutil, a command-line tool).

a. Create a bucket. b. Upload objects into the bucket. c. Create a folder in the bucket. d. Download an object. e. Share an object publicly. f. Delete objects.

Instead of only working with folders, Google Cloud Storage works with the concept of buckets, a higher hierarchical structure that can contains files or folders.

4. Access your data.

Because the data is public and doesn't need authentication, you can access the images by using the link you created. Learn more about accessing public data.

Store processed panoramas in the Google Cloud Storage To store all tiles in Google Cloud Storage, you simply create folders inside a bucket, upload all tiles, and share them publicly. As a good practice, create a folder for each panorama, because multiple panoramas have several tiles. In this example, each campus has its own folder and each corresponding panorama also has its own folder.

Figure 9. Interface to manage, link, and define panoramas paths

17

Figure 10. Directory of all rooms by specified campus

Build the web application In this specific example, the developer opted to use a more traditional model; that is, the application is hosted on a regular web server calling the images stored in Google Cloud Storage. (The development framework used was Microsoft .NET®.)

Web Server Google Cloud Storage

As a viable tool, Google offers App Engine, which is part of the Cloud Platform portfolio. It works under the concept of Platform as a Service (PaaS), where the physical server, memory, performance, availability, and scalability needs are addressed. To start learning App Engine, use one or all of these options:

● Select your preferred programming language: ● Python® ● Java® ● PHP® ● Go®

● Access the App Engine guided quick start guide. ● Ask technical questions on Stack Overflow using the official tag google-app-engine.

18

© 2015 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other company and product names may be trademarks of the respective companies with which they are associated.

19