compositie filtering of application datasets

Composite Filtering of Datasets- - -

Simplify Data Filtering Functions & Increase Performance

Coffee ShopsServes Cold Brew?

Yes NoServes what food?

Pastry Sandwiches Fruit

Reviews

0 5

A Simple App for filtering coffee shops on a map

With 0 filters applied, the complete set of coffee shops appear

Data points in “map-space”




Reviews

0 5


With 0 filters applied, the complete set of coffee shops appear

Data points in “data-space”




Reviews

0 5


When the Cold Brew filter applied, we get a “filtered set” of data points.

How do we translate a user-interaction (e.g., click) into a filtered-set?




Reviews

0 5


How does an application translate a user-interaction (e.g., click) into a filtered-set?




Reviews

0 5

“DB-centric” filtering approach

1

Adding a single filter1) User click’s on “cold brew” filter




Reviews

0 5


1

{“cold brew”: true }

2

Adding a single filter1) User click’s on “cold brew” filter2) Cold Brew filter criteria sent to database




Reviews

0 5


1

{“cold brew”: true }

2

3

Adding a single filter1) User click’s on “cold brew” filter2) Cold Brew filter criteria sent to database3) Filtered set of coffee shops returned




Reviews

0 5

1

{ “cold brew”: true “food” : sandwiches }

2

3

Adding an additional filter 1) User click’s on “food” filter2) Cold Brew and Food filter-criteria sent to database.3) Filtered set of coffee shops returned





Reviews

0 51

{ “cold brew”: true “food” : sandwiches “reviews”: [3,5] }

2

3

Adding a 3rd filter 1) User click’s on “food” filter2) Cold Brew, Food, Review Range filter-criteria sent to database3) Filtered set of coffee shops returned

“Stateless” filtering approach




Reviews

0 51

{ “cold brew”: true “food” : sandwiches “reviews”: [3,5] }

2

3

Inefficiency identified This approach does not leverage the DB processing performed on previous

user interactions


Details1) Each user interaction adds single criteria, but ALL criteria sent and processed

after each interaction

2) The DB filter operation gets more expensive as the number of potentially submitted filters increases

3) “Unselecting” a filter requires another database operation, even though the application had the resultant dataset prior to toggling this filter on

Inefficiency identified This approach does not leverage the DB processing performed on

previous user clicks



Additional downsides of this DB-centric approach:1) A single “catch-all” DB function gets increasingly complicated as number and

types of possible filters grow

2) A single DB function has to handle “null” filter-critera - it needs to be ready to accept values for all filter types - but all might not be sent

Example: Cold-brew filter sent, but food and reviews filter values are undefined; a single DB function needs to be ready to process food and review even though they aren’t in the incoming request

Alternative: “Stateful” filtering approach

1) Store the result of each individual filter set on browser2) Move set intersections from DB to the browser

{ “coffeeShops”:{ totalSet: [1,2,3,…,n] }

}

Angular service object (Keep track of the ID set associated with each individual filter)

On App load: retrieve the total set, store it as ID array

total-set

“Stateful” filtering




Reviews

0 5

{ “coffeeShops”:{ totalSet: [1,2,3,…,n], finalSet:[1,2,3,…,n] }

}

When no filters: total set = final set

total-set






Reviews

0 5

{ “coffeeShops”:{ totalSet: [1,2,3,…,n], filteredSets: { coldBrew:{resultSet: [2,4, …, n] } }

}

Serves Cold Brew?



Reviews

0 5

Coffee Shops

Add a filter

coldbrew-set



{ “coffeeShops”:{ totalSet: [1,2,3,…,n], filteredSets: { coldBrew:{resultSet: [2,4, …, n] } finalSet:[2,4,…,n] }

}

single filter result-set = final set

coldbrew-set



Serves Cold Brew?



Reviews

0 5

Coffee Shops

{ “coffeeShops”:{ totalSet: [1,2,3,…,n], filteredSets: { coldBrew:{ resultSet: [2,4, …, n], food: { resultSet: [4,5, … n] } }

}

Add additional filter

coldbrew-set

food-set

Serves Cold Brew?



Reviews

0 5

Coffee Shops


This set stored during previous user interaction


{ “coffeeShops”:{ totalSet: [1,2,3,…,n], filteredSets: { coldBrew:{ resultSet: [2,4, …, n], food: { resultSet: [4,5, … n] }, finalSet: [4,7 … n] }

}

Angular service object

coldbrew-set food-set

INTERSECT ID ARRAYS IN THE ANGULAR SERVICE!

Intersection of the two arrays = final set


Serves Cold Brew?



Reviews

0 5

Coffee Shops

{ “coffeeShops”:{ totalSet: [1,2,3,…,n], filteredSets: { coldBrew:{ resultSet: [2,4, …, n], food: { resultSet: [4,5, … n] }, finalSet: [2,4 … n] }

}

Angular service object

coldbrew-set food-set

Removing a filter does not require another round trip to DB

Remove a filter


Serves Cold Brew?



Reviews

0 5

Coffee Shops

Coffee ShopsAttributes:

Serves Cold Brew?Yes No

Serves what food?Pastry Sandwiches Fruit

Reviews

0 5


Angular program flow for “stateful filtering”.

1. User clicks on Cold Brew “yes”, fires an ng-click with info about the filter.

2. Controller tells service what type of filter this is (cold brew) and the value (yes)

3. Service checks a state object to see if this particular filter already has a value and a result set associated with it (e.g., previous UX had cold brew : no filter); if so it clears that value

4. Service determines which API endpoint to request a the filtered set from, and passes on the relevant request data { coldbrew: true} onto the endpoint

5. Endpoint returns a integer array of IDs meeting the filter-criteria to the service

6. The service completes the intersection of result sets from all currently toggled on filters

7. The service broadcasts the final intersected ID set to an listeners (e.g. a map controller the uses the ID array to show or hide map points).


1) Distributes processing across user interactions2) Eliminates DB trip for “filter-removal” event3) More performant - each DB operation is less complex; Intersections on the

browser are very quick (e.g, intersecting 4 10000 element integer arrays took 0.033s)

4) Simplifies DB function development5) Allows for a more customized set of DB filters.

Example Application instance “AAA”: food filter operates like an AND (pastries AND sandwiches) Application instance “BBB”: the filter operates like and OR (pastries OR sandwiches)You can easily have separate simple API endpoints/DB functions that handle each of the above cases for this type of filter instead of multiple large, complicated DB functions that handle all possible filter combinations in a simple request.

Benefits

compositie filtering of application datasets

Software