| Table of Contents |
|---|
...
Issues that were already identified
The following issues have been identified:
archiving of "old" datasets (where data is from a long time ago, but is as up to date as it can be eg. ebola 2013) - this is done
tag cleanup - dataset tags should not be freeform, they should be from a fixed list with the facility to request to add more - some of this is done
fixing data URLs so that charts, reports etc. don't break - USAID have asked for this and there is a Jira Epic for it (the Fixed Data URLs Idea goes further than what USAID have asked for)
a workflow that tries to alert a contributor when an update to a resource they are making has unexpected field names, data type changes, etc. - USAID have asked for this and there is a Jira Epic for it
a system whereby automated users (and maybe normal users as well) can register to receive important information about a dataset they are using eg. a breaking change to the format, no longer being updated etc. - USAID have asked for this and there is a Jira Epic for it
we need to be able to distinguish data resources from auxiliary ones - helpful to DP's work on QAing datasets
distinguishing API resources from others
resources can't keep growing indefinitely - we need a way to split a resource once it grows beyond a certain size
keeping a history of data (versioning) - newly added data may contain errors so it may be helpful to be able to fall back to a previous version of the data eg. if a dashboard cannot load latest/xxx.csv, it could try 1/xxx.csv
a service whereby if a contributor uploads data in a particular format/structure that we specify, then the data is served out disaggregated in multiple ways
finding data by API needs to be simpler. Currently the limitation there is the capabilities of the CKAN API search. It can be helped by adding more metadata into the dataset for example the list of fields and HXL tags in the data
more generally a search tailored to what users want to search for eg. if a user types "wheat price kandahar" they would like to get back that price.
How data is currently structured
...
The ideas below would require technological leaps and involve creating new systems outside of CKAN (athat could still link back into CKAN). They require research and discussion to flesh out.
...
Meta Service/API that links to other APIs
The idea is to have a meta service/API that would link to other services/APIs and allow the easy download of data given a standard set of input parameters. On top of such a meta service/API would be a user interface which would allow setting of those parameters and download by non technical users. Using the "wheat price kandahar" example, I could imagine a meta service/API user choosing parameters like "service": "prices", "type": "commodities", "provider": "WFP", "country": "Afghanistan", "adm1": "Kandahar", "Commodity": "wheat", "date": "08/03/2022".
...
Resources should have a field "is_queryable" which can be True or False
Resources should have a field "queryable_desc_url" which is a link to a webpage that describes the parameters a user can to the URL to filter or transform the data
There should be an explanation of what queryable means in the dataset edit dialog
There should be a checkbox in the dataset edit dialog to mark a resource as queryable
There should be a url field for adding a link to a webpage that describes the parameters a user can to the URL to filter or transform the data
Queryable resources should be flagged in the dataset view and search UIs (work has been done on this)
The link to a webpage describing the parameters should be next to or below the queryable resource
...
Data Cube
Typically we make data available in different forms like by indicator or by country by making separate datasets with their own data. This makes accessing the data fast but complicates scrapers which must do the breakdowns so mostly data is just broken down by country on HDX. A data cube enables data to be modelled and viewed in multiple dimensions. Is there a way that data could be stored in some sort of data cube so that it can be viewed in different ways like by country, admin 1 or indicator without keeping multiple copies of the data?
...