circumnavigating the antarctic with python and django during ace 2016

Python and Django during the AntarcticCircumnavigation Expedition (ACE)

PyCon UK, 2017. Carles Pina i Estany (carles@pina.cat)

What do we do for work?

CarlesC++/Qt software engineer at MendeleySome Python (scripts at work, evenings and weekends, Python code Dojos, etc.)Using GNU/Linux for a long timeLong time ago: system administrator / network administrator

2 / 45

What do we do for work?

CarlesC++/Qt software engineer at MendeleySome Python (scripts at work, evenings and weekends, Python code Dojos, etc.)Using GNU/Linux for a long timeLong time ago: system administrator / network administrator

JenWorked for 3.5 years at the British Antarctic Survey (BAS) as a data managerNow works for the Swiss Polar Institute, following up on the data management of ACEMore into wildlife and science (especially birds) than programming

2 / 45

What were our roles on the ship?As a team we:

managed the data of all 22 science projects

set up the data management recording system (Django);set up the data storage and back-ups;liaised with scientists to back up data, record metadata in a database and createmetadata records;

telecommunications setup:

onboard network;e-mail system;file uploader / Dropbox equivalent;

connecting science equipment.

3 / 45

What is ACE?

Antarctic Circumnavigation Expedition

4 / 45

Copyright: Swiss Polar Institute 5 / 45

R/V Akademik Tryoshnikov

6 / 45

7 / 45

8 / 45

Why is ACE different?Usually science expeditions focus on a small area of the Antarctic

9 / 45

Why is ACE different?Usually science expeditions focus on a small area of the AntarcticOnly other similar expedition was Discovery II in the 1950's

9 / 45

Why is ACE different?Usually science expeditions focus on a small area of the AntarcticOnly other similar expedition was Discovery II in the 1950'sSampled and studied marine and terrestrial habitats

9 / 45

Why is ACE different?Usually science expeditions focus on a small area of the AntarcticOnly other similar expedition was Discovery II in the 1950'sSampled and studied marine and terrestrial habitatsFirst expedition of the Swiss Polar Institute

9 / 45

Why is ACE different?Usually science expeditions focus on a small area of the AntarcticOnly other similar expedition was Discovery II in the 1950'sSampled and studied marine and terrestrial habitatsFirst expedition of the Swiss Polar InstituteIt was organised very quickly

9 / 45

Science

10 / 45

Copyright Swiss Polar Institute 11 / 45

12 / 45

13 / 45

14 / 45

Data management

15 / 45

What is it?Backup all the collected dataKeep a log of what's happening:

ultimate goal - to be able to describe where, when, by whom and for what reason thedata were collected

Keep a database of all samples collectedCreate metadata records of all data sets so they can be discovered

16 / 45

What is it?Backup all the collected dataKeep a log of what's happening:

ultimate goal - to be able to describe where, when, by whom and for what reason thedata were collected

Keep a database of all samples collectedCreate metadata records of all data sets so they can be discovered

Why?Backup to avoid losing data and to archive it properly for the futureThe data can be made publicly available after the 2 year embargoMake data easier to discover and usable by others in the future

16 / 45

We had many IT tasksSetup the data storage (2 NAS, 64 TB, 2 UPS, etc.)

17 / 45

We had many IT tasksSetup the data storage (2 NAS, 64 TB, 2 UPS, etc.)Create utilities to backup from USB hard disks and network-attached computers

17 / 45

We had many IT tasksSetup the data storage (2 NAS, 64 TB, 2 UPS, etc.)Create utilities to backup from USB hard disks and network-attached computersCreate an intranet webpage for the data entry, utilities and to disseminate information onboard

17 / 45

We had many IT tasksSetup the data storage (2 NAS, 64 TB, 2 UPS, etc.)Create utilities to backup from USB hard disks and network-attached computersCreate an intranet webpage for the data entry, utilities and to disseminate information onboardDesign and implement an e-mail system that works with an unstable 128 kbit connectionfor 80 people

17 / 45

We had many IT tasksSetup the data storage (2 NAS, 64 TB, 2 UPS, etc.)Create utilities to backup from USB hard disks and network-attached computersCreate an intranet webpage for the data entry, utilities and to disseminate information onboardDesign and implement an e-mail system that works with an unstable 128 kbit connectionfor 80 peopleCreate a way to upload videos for the news (we had journalists on board)

17 / 45

We had many IT tasksSetup the data storage (2 NAS, 64 TB, 2 UPS, etc.)Create utilities to backup from USB hard disks and network-attached computersCreate an intranet webpage for the data entry, utilities and to disseminate information onboardDesign and implement an e-mail system that works with an unstable 128 kbit connectionfor 80 peopleCreate a way to upload videos for the news (we had journalists on board)Download data (ice images, weather models) for the scientists

17 / 45

We had many IT tasksSetup the data storage (2 NAS, 64 TB, 2 UPS, etc.)Create utilities to backup from USB hard disks and network-attached computersCreate an intranet webpage for the data entry, utilities and to disseminate information onboardDesign and implement an e-mail system that works with an unstable 128 kbit connectionfor 80 peopleCreate a way to upload videos for the news (we had journalists on board)Download data (ice images, weather models) for the scientistsHelp scientists to connect different equipment (like snow flake counter)

17 / 45

We had many IT tasksSetup the data storage (2 NAS, 64 TB, 2 UPS, etc.)Create utilities to backup from USB hard disks and network-attached computersCreate an intranet webpage for the data entry, utilities and to disseminate information onboardDesign and implement an e-mail system that works with an unstable 128 kbit connectionfor 80 peopleCreate a way to upload videos for the news (we had journalists on board)Download data (ice images, weather models) for the scientistsHelp scientists to connect different equipment (like snow flake counter)Fix a winch (software part)

17 / 45

We had many IT tasksSetup the data storage (2 NAS, 64 TB, 2 UPS, etc.)Create utilities to backup from USB hard disks and network-attached computersCreate an intranet webpage for the data entry, utilities and to disseminate information onboardDesign and implement an e-mail system that works with an unstable 128 kbit connectionfor 80 peopleCreate a way to upload videos for the news (we had journalists on board)Download data (ice images, weather models) for the scientistsHelp scientists to connect different equipment (like snow flake counter)Fix a winch (software part)...

17 / 45

science-data-managementJen said: Can you help me build a database?

18 / 45

science-data-managementJen said: Can you help me build a database?

I answered: yes! (and thought of MySQL and create table...)

18 / 45

science-data-managementJen said: Can you help me build a database?

I answered: yes! (and thought of MySQL and create table...)

Jen also meant a web system to enter data in the database!

18 / 45

19 / 45

Django

20 / 45

I knew:

Python

21 / 45

I knew:

PythonFlask

21 / 45

I knew:

PythonFlaskMySQL

21 / 45

I knew:

PythonFlaskMySQLSQLAlchemy

21 / 45

I knew:

PythonFlaskMySQLSQLAlchemyMustache

21 / 45

I knew:

PythonFlaskMySQLSQLAlchemyMustache

But... How could I create the forms to enter data into the database?

21 / 45

I knew:

PythonFlaskMySQLSQLAlchemyMustache

But... How could I create the forms to enter data into the database?

I asked Fran... and he said...

21 / 45

I knew:

PythonFlaskMySQLSQLAlchemyMustache

But... How could I create the forms to enter data into the database?

I asked Fran... and he said...

Django!

21 / 45

I knew:

PythonFlaskMySQLSQLAlchemyMustache

But... How could I create the forms to enter data into the database?

I asked Fran... and he said...

Django!(I thought... nooooo...)

21 / 45

I knew:

PythonFlaskMySQLSQLAlchemyMustache

But... How could I create the forms to enter data into the database?

I asked Fran... and he said...

Django!(I thought... nooooo...)

Fran created an example... and I loved it!

21 / 45

One week of DjangoI went back home and really enjoyed learning Django:

Writing Django models is easier than MySQL create table (with foreign keys, etc.)Easier to change the models and migrateWe needed authentication of usersPermissions for tables were usefulTemplates are very well integrated with modelsAll the admin part that we used has a good default UIEasy to extend

22 / 45

One week of DjangoI went back home and really enjoyed learning Django:

Writing Django models is easier than MySQL create table (with foreign keys, etc.)Easier to change the models and migrateWe needed authentication of usersPermissions for tables were usefulTemplates are very well integrated with modelsAll the admin part that we used has a good default UIEasy to extend

And it has very good off-line documentation!

22 / 45

Django modelsclass StorageCrate(models.Model): name = models.CharField(max_length=255, unique=True) location = models.CharField(max_length=255) description = models.CharField(max_length=255, null=True, blank=True) comment = models.TextField(null=True, blank=True)

23 / 45

Django modelsclass StorageCrate(models.Model): name = models.CharField(max_length=255, unique=True) location = models.CharField(max_length=255) description = models.CharField(max_length=255, null=True, blank=True) comment = models.TextField(null=True, blank=True)

It creates the forms automaticallyIf the model didn't exist: creates the tables in the databaseIf the model already existed: creates the migration codeDeals with foreign keys, relations, etc.It creates relational database models like we would have done

23 / 45

science-cruise-data-managementhttp://github.com/cpina/science-cruise-data-management

24 / 45

25 / 45

26 / 45

27 / 45

28 / 45

29 / 45

In retrospectDjango was a very good tool to do what we had to do:

Without Django we would have taken much longer!Django didn't get in the way as much as I had thoughtDjango commands were very easy to implement and useful for accessing the models fromcommand line parameters, widely used for many of our scriptsDjango developers are very nice and helpful! Thanks for the concise error messages:

main.Project.title: (fields.E120) CharFields must define a 'max_length' attribute.

30 / 45

In retrospectDjango was a very good tool to do what we had to do:

Without Django we would have taken much longer!Django didn't get in the way as much as I had thoughtDjango commands were very easy to implement and useful for accessing the models fromcommand line parameters, widely used for many of our scriptsDjango developers are very nice and helpful! Thanks for the concise error messages:

main.Project.title: (fields.E120) CharFields must define a 'max_length' attribute.

Django documentation is very well written!Admin forms were easy to adapt and change

30 / 45

31 / 45

What is Django?From the Django website:

Django makes it easier to buildbetter Web apps more quickly andwith less code.

32 / 45

What is Django?From the Django website:

Django makes it easier to buildbetter Web apps more quickly andwith less code.

Data manager thinks:

We need to build a database

32 / 45

What is Django?From the Django website:

Django makes it easier to buildbetter Web apps more quickly andwith less code.

Data manager thinks:

We need to build a database

Django for data managers?32 / 45

Very useful Django features for usAdmin forms were easy to adapt and change (we could work around something if itwasn't possible)commands are fantastic!

33 / 45

Very useful Django features for usAdmin forms were easy to adapt and change (we could work around something if itwasn't possible)commands are fantastic!

Django things that were not so helpful...Some modules use resources (CSS, js, etc.) from the Internet (e.g. debug module). Thisdidn't work for usHard to change (only via changing the template?) the main admin page

33 / 45

pip and rsyncpip wasn't "bad connection friendly":

No progress barWe didn't see how to easily see the URIs being download (e.g. apt-get --show-uris todownload them overnight)When bad networking: error messages weren't clear

34 / 45

pip and rsyncpip wasn't "bad connection friendly":

No progress barWe didn't see how to easily see the URIs being download (e.g. apt-get --show-uris todownload them overnight)When bad networking: error messages weren't clear

man rsync, --progress:

This option tells rsync to print information showing theprogress of the transfer. This gives a bored user something towatch.

I'd say "it helps my impatience".

34 / 45

pip and rsyncpip wasn't "bad connection friendly":

No progress barWe didn't see how to easily see the URIs being download (e.g. apt-get --show-uris todownload them overnight)When bad networking: error messages weren't clear

man rsync, --progress:

This option tells rsync to print information showing theprogress of the transfer. This gives a bored user something towatch.

I'd say "it helps my impatience".

Python documentationMore comprehensive off-line documentation could have been better (we relied on examplesand source code)

34 / 45

Django closing the circleWe wanted to thank Django somehow for all the fantastic help.

I thought of fixing at least one bug that we experienced during the expedition

35 / 45

Django closing the circleWe wanted to thank Django somehow for all the fantastic help.

I thought of fixing at least one bug that we experienced during the expeditionBug #28120 reported and fixed! (thanks for the quick code review, merge...)

35 / 45

Python/Django ideas if you go onan expedition

Take lots of code with you (e.g. Calibre, other projects, etc.). This was more useful thanbooks and documentation

36 / 45

Python/Django ideas if you go onan expedition

Take lots of code with you (e.g. Calibre, other projects, etc.). This was more useful thanbooks and documentationUse an IDE like Pycharm that allows you to navigate code of your libraries easier

36 / 45

Python/Django ideas if you go onan expedition

Take lots of code with you (e.g. Calibre, other projects, etc.). This was more useful thanbooks and documentationUse an IDE like Pycharm that allows you to navigate code of your libraries easierDownload the offline Django documentation: you will feel almost online!

36 / 45

Python/Django ideas if you go onan expedition

Take lots of code with you (e.g. Calibre, other projects, etc.). This was more useful thanbooks and documentationUse an IDE like Pycharm that allows you to navigate code of your libraries easierDownload the offline Django documentation: you will feel almost online!

Actually also download Wikipedia!

36 / 45

Appendix: Internet on the ship

37 / 45

IntroductionWe had 2 Iridium systemsInitially one for calls, another one for InternetThis is, 128 kbits for (officialy) 80 peopleThe Iridium connections are very unstable

38 / 45

L. Wood, SaVi: satellite constellation visualization, First Annual CCSR Research Symposium(CRS 2011), Centre for Communication Systems Research, 30 June 2011.http://arxiv.org/abs/1204.3265

39 / 45

Iridium66 Iridium satellites

40 / 45

Iridium66 Iridium satellitesIridium name: originally supposed to be 77 satellites: atomic number of Iridium

40 / 45

Iridium66 Iridium satellitesIridium name: originally supposed to be 77 satellites: atomic number of IridiumThere are spare satellites for when there are problems

40 / 45

Iridium66 Iridium satellitesIridium name: originally supposed to be 77 satellites: atomic number of IridiumThere are spare satellites for when there are problemsSignal on the north and south pole! But VERY slow and unstable: it connects anddisconnects because satellites are not geostationary

40 / 45

Iridium66 Iridium satellitesIridium name: originally supposed to be 77 satellites: atomic number of IridiumThere are spare satellites for when there are problemsSignal on the north and south pole! But VERY slow and unstable: it connects anddisconnects because satellites are not geostationarySatellites are around 780 Km above the Earth (GPS and Inmarsat around 20,000 Km)

40 / 45

Iridium66 Iridium satellitesIridium name: originally supposed to be 77 satellites: atomic number of IridiumThere are spare satellites for when there are problemsSignal on the north and south pole! But VERY slow and unstable: it connects anddisconnects because satellites are not geostationarySatellites are around 780 Km above the Earth (GPS and Inmarsat around 20,000 Km)Sometimes the satellites are visible: Iridium flares!

40 / 45

Iridium66 Iridium satellitesIridium name: originally supposed to be 77 satellites: atomic number of IridiumThere are spare satellites for when there are problemsSignal on the north and south pole! But VERY slow and unstable: it connects anddisconnects because satellites are not geostationarySatellites are around 780 Km above the Earth (GPS and Inmarsat around 20,000 Km)Sometimes the satellites are visible: Iridium flares!They are launching new satellites

40 / 45

41 / 45

42 / 45

43 / 45

Thank you!Questions?

44 / 45

ContactCarles Pina i Estany (carles@pina.cat)Jen Thomas (jenny.thomas@epfl.ch)

License

This work is licensed under a Creative Commons Attribution 4.0 International License.

SlidesSlides have been written using Markdown and rendered using remarkjs (https://remarkjs.com)

Slides: https://github.com/cpina/ace-it-presentation / https://cpina.github.io/ace-it-presentation/

ACE: http://spi-ace-expedition.ch/

45 / 45