delicious data: automated techniques for complex reports
DESCRIPTION
Delicious Data: Automated Techniques for Complex Reports. Presented by Jeff Godin and Benjamin Shum. How NOT to run reports. Planning the framework. SQL Query Report. Export Results to CSV. Schedule as Cron Job. Convert CSV to XLS. Email XLS to Library. Some Strategies. Schedule as - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/1.jpg)
Delicious Data:Automated Techniques
for Complex Reports
Presented byJeff Godin and Benjamin Shum
![Page 2: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/2.jpg)
How NOT to run reports...
![Page 3: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/3.jpg)
Planning the framework
SQL Query SQL Query ReportReport
Export ResultsExport Resultsto CSVto CSV
ConvertConvertCSV to XLSCSV to XLS
Email XLSEmail XLSto Libraryto Library
Schedule asSchedule asCron JobCron Job
![Page 4: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/4.jpg)
Some Strategies
Schedule asSchedule asCron JobCron Job
Use our experience setting up crontab
![Page 5: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/5.jpg)
Use our existingSQL reports
SQL Query SQL Query ReportReport
Export ResultsExport Resultsto CSVto CSV
CSV is a standardformat to work with
![Page 6: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/6.jpg)
ConvertConvertCSV to XLSCSV to XLS
Email XLSEmail XLSto Libraryto Library
Find a converter
Find an email client
![Page 7: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/7.jpg)
Implementation
•Can find online at http://evergreen-ils.org/dokuwiki/doku.php?id=scratchpad:automated_sql_reports
![Page 8: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/8.jpg)
Future of the work
•Additional formatting
•Dealing with reports with zero results
•Adding new reports
•Implementing more refinements
![Page 9: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/9.jpg)
No database access?
•Create scheduled reports
•Completion notification via e-mail
•Script pulls report data
•Import and process
![Page 10: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/10.jpg)
Don’t do this unless you have to
![Page 11: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/11.jpg)
Reports as Exports
•Create template that outputs raw data
•Schedule report
•Enable e-mail notification to special address
![Page 12: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/12.jpg)
Act on the e-mail
•fetchmail -> procmail -> perl
•Report data is automatically downloaded
•Within a minute or so
![Page 13: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/13.jpg)
act-on-mail.pl
#!/usr/bin/perl# called by procmail, and receives the mail message on stdinuse strict; use warnings;my $url;while (<>) { if ($_ =~ /https:\/\/reporter/) { $url = $_; last;}}system("/home/stats/bin/fetch-report.pl", "-u", $url);
![Page 14: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/14.jpg)
fetch-report.pl logic
•Fetch URL like:https://host/reporter/[...]/report-data.html
•Automatically log in if needed (keep a cookie)
•Download report-data.csv and save to output dir
![Page 15: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/15.jpg)
imports.ini example 1
[general]dir=/home/stats/reportsstate_dir=/home/stats/state[group holdratio]name=Hold Ratiofile1=holds_outstanding.csvfile2=holdable_copies.csvfile3=bib_info.csvcommand=psql -f /home/stats/sql/holds_init.sql#post-command=/home/stats/bin/mail-holdratio.pl
![Page 16: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/16.jpg)
imports.ini example 2
[general]dir=/home/stats/reportsstate_dir=/home/stats/state[group items_for_export]name=Items for Exportfile1=items_for_export.csvcommand=psql -f /home/stats/sql/items_for_export.sqlpost-command=/home/stats/bin/export_bibs.pl -f /home/stats/output/bre_ids.csv
![Page 17: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/17.jpg)
Import script overview
•Runs from cron
•Have all input files been updated since the last time I ran this import?
•Run import
•Update timestamp
•Run post-import command
![Page 18: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/18.jpg)
The import process
•drop table
•create table
•import data
•process to output
![Page 19: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/19.jpg)
The post-import
•Send the report output somewhere
•E-mail notify that “report’s ready at someurl”
•Trigger something like a bib export of the records that were output
![Page 20: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/20.jpg)
One more thing...
•Don’t forget to delete your scheduled report output
![Page 21: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/21.jpg)
Custom reporting views exposed in
the IDL
![Page 22: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/22.jpg)
SQL views in the IDL
•Exposed via reporting interface
•Good for on-demand reporting
•Don’t break the IDL
•xmllint is your friend
•pick a class id unlikely to collide
![Page 23: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/23.jpg)
<class id="rlcd" controller="open-ils.cstore open-ils.pcrud open-
ils.reporter-store"oils_obj:fieldmapper="reporter::last_copy_deleted"oils_persist:readonly="true"reporter:core="true"reporter:label="Last Copy Delete Time" >
![Page 24: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/24.jpg)
<oils_persist:source_definition>
SELECT b.id, MAX(dcp.edit_date) AS last_delete_date FROM biblio.record_entry b JOIN asset.call_number cn ON (cn.record = b.id) JOIN asset.copy dcp ON (cn.id = dcp.call_number)
WHERE NOT b.deleted
GROUP BY b.id
HAVING SUM( CASE WHEN NOT dcp.deleted THEN 1 ELSE 0 END) = 0
</oils_persist:source_definition>
![Page 25: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/25.jpg)
<!-- continued --><fields oils_persist:primary="id"
oils_persist:sequence="biblio.record_entry"><field reporter:label="Record ID" name="id"
reporter:datatype="id"/><field reporter:label="Delete Date/Time"
name="last_delete_date" reporter:datatype="timestamp"/></fields><links>
<link field="id" reltype="has_a" key="id" map="" class="bre"/>
</links>
![Page 26: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/26.jpg)
<!-- continued --><permacrud xmlns="
http://open-ils.org/spec/opensrf/IDL/permacrud/v1"><actions>
<retrieve/></actions>
</permacrud></class>
![Page 27: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/27.jpg)
The Future
•General third party reporting frameworks?
•Library Dashboards?
•Your Ideas?
![Page 28: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/28.jpg)
Photo CreditsSurrounded by papers by Flickr user suratlozowick
http://www.flickr.com/photos/suratlozowick/4522926028/
![Page 29: Delicious Data: Automated Techniques for Complex Reports](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815822550346895dc5892d/html5/thumbnails/29.jpg)
Thanks!