...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

Introduction

Scrapers collect data from websites that is pushed into HDX or in the case of the dap ones into CPS as indicators and from there into HDX. The data is extracted programmatically from spreadsheets or html. The status of the dap scrapers is reviewed here. This document considers the others.

Current Status on ScraperWiki

Notes:

Scrapers with “Dragon -” as a prefix were created by Dragon, all others are created by “UN OCHA” - finding specifically who for these requires looking for and then at the source code

Working but not in HDX

A number of FTS scrapers work but their output cannot be seen in HDX. FTS refers to the OCHA Financial Tracking Service. It is "a centralized source of curated, continuously updated, fully downloadable data and information on humanitarian funding flows."

...

The FTS Appeals and FTS Emergency collectors are working. It is not obvious where they output. There are similarly named datasets in HDX but they do not have any resources - see https://data.hdx.rwlabs.org/dataset/fts-appeals and https://data.hdx.rwlabs.org/dataset/fts-emergencies. FTS Appeals in fact outputs to a number of datasets ending “ humanitarian contributions” eg. Senegal 2015 humanitarian contributions according to https://docs.google.com/spreadsheets/d/1RdHFRn3s8uxRfbmJ6aZCghpt85mFk3aR8eDZAgwLRKs/edit?usp=sharing. This is not readily apparent in the code. Since these collectors are coded in R and rely on CPS, they should be rewritten in Python and their dependence on CPS removed.

The FTS Ebola and FTS Ebola Coverage scrapers are working but rely on CPS to get data to HDX. These collectors are written in R and should be recoded in Python without the reliance on CPS. FTS Ebola Coverage (https://data.hdx.rwlabs.org/dataset/fts-ebola-coverage) was last updated in HDX four months ago. It is not clear which dataset FTS Ebola is updating - from the code there should be one called "FTS Ebola Indicators" but it does not come up in a search on HDX.

Working but not run recently

It is not easy to find out the frequency with which scrapers are supposed to be running as this information is not in the ScraperWiki GUI. The frequency tells us whether a scraper should have run in a certain period or not. For example, if a scraper has not run for 5 months, but its frequency is annual, then it is probably ok, but if it has not run for a week and its frequency is daily then it is is broken.

...

The table above uses the arbitrary cutoff of 4 months to try to point to scrapers that should be examined in more detail.

UNHCR Mediterranean collector

The UNHCR Mediterranean collector had not run for 6 months. Connecting to the machine where it resides by ssh and running it worked fine, and afterwards it displayed as recently run on ScraperWiki.

...

The scraper's data taken from: http://data.unhcr.org/api/stats/mediterranean/monthly_arrivals_by_country.json is put into a ScraperWiki sqlite database and relies on CPS to get it into HDX. It is not clear what dataset is being updated in HDX.  This would be clearer if the scraper was rewritten to use the CKAN API to HDX.

FTS Nepal Earthquake Collector

The FTS Nepal Earthquake collector has not run for eight months but since it was designed to collect data around a specific crisis, this is most likely not a problem. The dataset in HDX is here: https://data.hdx.rwlabs.org/dataset/response-plan-coverage-nepal-earthquake

FTS Collector

The FTS collector has not run for nine months. It does not have any scheduling set up and the datasets in HDX do not have any resources (https://data.hdx.rwlabs.org/dataset/fts-clusters, https://data.hdx.rwlabs.org/dataset/fts-appeals and https://data.hdx.rwlabs.org/dataset/fts-emergencies). It was likely designed to replace the two scrapers written in R with the same output datasets in HDX. It resides in a GitHub repository in Reuben's public area and should be moved to a central well known location if it is to be maintained (assuming it works).

Failed Scrapers

UNHCR Real-time API

This failed with the error: “Error in library(sqldf) : there is no package called ‘sqldf’”

...

Sqldf is an R library that appears to be missing. The code for this scraper is written in R and put directly in ScraperWiki (“code in your browser”). It would be advisable to rewrite it in Python as supporting multiple languages is a maintenance headache and make it standalone (“code your own tool”) in ScraperWiki.

HDX Repo Analytics

This scraper has the same issue as the UNHCR Real-Time API.

UNOSAT Product Scraper

This scraper failed because one of the urls that it scrapes from a webpage was a dead link and it has no check for this possibility. It was difficult to find this fault because the scraper is written in R with limited logging and error handling. After fixing it, although the scraper claims to have run successfully, the datasets on HDX seem to remain unchanged, hopefully because the data is assessed as unmodified. This requires further study. The metadata modification date is generally 24/11/2015 but a few datasets are newer (from Luis spreadsheet). Ultimately, the scraper should be rewritten in Python with detailed logging and error handling.

UNOSAT Flood Portal

Running this scraper produced this output:

...

Although it shows as successful on ScraperWiki, given these errors, it is not clear if it is working and further analysis is required. The metadata modification dates

IFPRI Dataverse Collector

When running this scraper, the error it produces is: “Service Temporarily Unavailable. The server is temporarily unable to service your request due to maintenance downtime or capacity

...

It tries to connect to this url: https://dataverse.harvard.edu/api/dataverses/IFPRI/contents which fails similarly in a browser.

...

This seems to be a problem with the website and needs to be followed up with the maintainer.

WFP VAM API Scraper

This scraper runs for several hours and then fails. It posts a form to a number of urls of this form: http://reporting.vam.wfp.org/API/Get_CSI.aspx along with a set of parameters. It is likely one of these that is timing out. The script is quite complicated so will require some time to debug.

Overall Condition of Scrapers

Looking at the scrapers as a whole rather than individually, an assessment can be made of their general health. The unfortunate conclusion is that they are not in a good state even those which are currently running regularly. It was raised previously in the DAP scraper review before the recent departure of a long serving data scientist that there is “significant key man risk as knowledge about them is not distributed and they are built and looked after in varying ways by different authors. As the number of scrapers grows, it will be difficult to maintain and if there are staff changes, this will present serious challenges.” That is the situation we are in now.

...

The update frequency of a dataset in HDX is not tied to the schedule of scrapers in ScraperWiki which are set up using crontab. For example, a collector may operate every week in ScraperWiki but its corresponding dataset in HDX may have update frequency “never” (particularly if CPS sits in between), so determining if data in HDX is current is not obvious as it may be that the same data is being downloaded and then not updated in HDX. Whether CPS refreshes datasets in HDX if the files have not changed is not clear and it needs to be confirmed that the activity stream in HDX always shows when public datasets are updated. Some scrapers are run manually on demand eg. WFP food prices and those should be documented.

Next Steps

As was mentioned for dap scrapers, for those that are not used or for which it is decided that data is unimportant, they should be deleted so that it is clear what needs to be properly maintained and tested.

...