Versions Compared

Old Version 6

changes.mady.by.user Michael Rans

Saved on

compared with

New Version 7

changes.mady.by.user Michael Rans

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Background

...

HDX has a RESTful API largely unchanged from the underlying CKAN API which can be used from any programming language that supports HTTP GET and POST requests. The HDX Python API provides a simple interface that communicates with HDX using the CKAN Python API, a thin wrapper around the CKAN REST API. It is a mature library that supports Python 2.7 and 3 with tests that have a high level of code coverage. The major goal of the library is to make pushing and pulling data from HDX as simple as possible for the end user. HDX objects, such as datasets and resources, are represented by Python classes. The scrapers we will discuss here use this library to communicate with HDX. 

Current Platform

The current platform is an external service called ScraperWiki. It has a web based user interface that enables the status of scrapers to be viewed. Through the UI, it is possible to create a new environment on which to run new scrapers. The environment is an Amazon Web Services virtual server with various packages included on it like Python. The UI allows the user to obtain a url for a server. The user can ssh into that server and set up the scraper by, for example, git cloning its code onto the server and then setting up a virtualenv with the required Python packages. Cron is used to execute the scrapers according to the desired schedule.

...

Rather than each scraper executing within a Python virtualenv as currently, they will each be in a Docker container. The scraper's Docker image will build upon (inherit) a base image owned by OCHA IT. The draft base image is here. It inherits from unocha/alpine-base:3.8 and contains a Python 3 environment suitable for running scrapers - it includes HDX Python API library, awesome-slugify and Pandas (including its dependencies on Scipy and Numpy). The libraries that HDX Python API depends on are all open source. An example scraper that inherits this base image is the FTS scraper.

There is some private information that is needed by the scrapers to run. Currently it resides in a private OCHA GitHub repository, but it will be moved to Ansible.

The setup will comply with OCHA IT's Hosting in Shared Infrastructure: Project Requirements.