The International Organization for Migration (IOM) is the leading inter-governmental organization in the field of migration and works closely with governmental, intergovernmental and non-governmental partners. It is dedicated to promoting humane and orderly migration for the benefit of all and works to help ensure the orderly and humane management of migration, to promote international cooperation on migration issues, to assist in the search for practical solutions to migration problems and to provide humanitarian assistance to migrants in need, including refugees and internally displaced people.
The Displacement Tracking Matrix (DTM) continually tracks and monitors displacement across countries allowing IOM to identify the locations to which IDPs have chosen to settle. The location and population of these IDPs are recorded and further in-depth assessments are conducted to identify the multi-sectorial needs of the displaced.
Sources
(Unless otherwise noted, all datasets are shared by IOM and consist of a single DTM round.)
HDX dataset | Date | Notes |
|---|---|---|
2016-04-30 | ||
2015-10-31 | Raw DTM (with some sensitive fields removed). Much-more granular than the other examples, with 175 columns, and structured as a survey. This lets us see what the DTM originally looks like, before it’s digested into the high-level reports that are more typically shared publicly. Includes ADM1, ADM2, “Payam”, and “Village”, as well as site id and lat/lon. Lists number of IDPs and households. | |
2015-11-25 | ||
2015-04-30 | ||
2016-01-10 | ||
2016-01-20 | Multiple rounds. | |
2015-12-30 | Multiple rounds. | |
2016-01-28 | Multiple rounds. | |
2015-11-30 | ||
2016-03-01 | ||
2015-11-03 | ||
2015-11-30 | ||
2014-09-14 | Shared by OCHA Iraq. | |
2014-08-07 | Shared by OCHA Iraq. | |
2014-08-24 | Shared by OCHA Iraq. | |
2014-09-01 | Shared by HDX team. Multiple rounds. | |
2014-11-25 | Shared by OCHA Iraq. | |
2014-07-02 | Shared by OCHA Iraq. | |
2015-05-20 | Shared by OCHA ROSA. | |
2015-03-06 | Shared by OCHA ROSA. | |
2015-03-06 | Shared by OCHA ROSA. | |
2015-05-31 | Shared by OCHA Mali. |
Data Structure
Some spreadsheets have "master list" in the title and some do not.
Taking the latest master list for a few countries, unfortunately we see that the DTMs do not have a fixed structure.
Below are examples of different headings for various DTM spreadsheets:
Libya: rd4_DTM_Master_List_Jun2016.xlsx
This is at an aggregated rather than survey level, has several sheets and 2 rows of headers for the main sheet entitled "1-DTM Round 4 Dataset":
| Shabiya_Name_EN | Baladiya_ID | Baladiya_Name_EN | Baladiya_Name_AR | Lat | Long | is area assessed by DTM? (Y,N) | IDP Households | IDP Individuals | IDP households displaced in 2011 | Type of Displacement 2011 | Baladiya of Origin 2011 | IDP households displaced 2012- mid-2014 | Type of Displacement 2012- mid-2014 | Baladiya of Origin 2012- mid-2014 | IDP households displaced after mid-2014 | Type of Displacement after mid-2014 | Baladiya of Origin after mid-2014 | Migrant Individuals in Baladiya | Migrant Individuals in Detention Centers in Baladiya | Migrant Individuals crossing Baladiya | Returnee Households | Returnee Individuals | households displaced by general violence reasons | households displaced by special security reasons | households displaced by economic Reasons | Area have IDPs in Rented_House_Paid | Area have IDPs in Rented_House_NotPaid | Area have IDPs with Host community - relatives | Area have IDPs with Host community - non-relatives | Area have IDPs in schools | Area have IDPs in Public_Building | Area have IDPs Squatting | Area have IDPs in Unfinished_Building | Area have IDPs in Abandoned_Resorts | Area have IDPs in Collective_NonFormal settlements | Area have IDPs where shelter type is unknown |
| ADM2_Shabiya_Name_EN | ADM3_Baladiya_ID | ADM3_Baladiya_Name_EN | ADM3_Baladiya_Name_AR | Latitude | Longitude | Area assessed by DTM | IDPs In Baladiya_HH | IDPs In Baladiya_IND | IDPs In Baladiya HH_2011 | Origin Type 2011 | Origin 2011 | IDPs In Baladiya_HH 2011_2014 | Origin Type 2011_2014 | Origin 2011_2014 | IDPs_In_Baladiya_HH 2014+ | Origin Type 2014+ | Origin 2014+ | Migrants in Baladiya | Migrants in Detention Center | Crossing Migrants | Returnees HH | Returnees Ind | Displacement for violence | Displacement for Security | Displacement for Economic | Rented accommodation (self-pay) | Rented accommodation (paid by others) | Host families who are relatives | Host families who are not relatives | Schools | Other public buildings | Squatting on other people’s properties (e.g. in farms, flats, houses) | In unfinished buildings | In deserted resorts | In Informal Settings (e.g. tents, caravans, makeshift shelters) | Unknown |
Nigeria: rd10_DTM_Master_List_Jun2016.xlsx
This is at an aggregated level and has just one sheet "Sheet1" with a single line header:
| LGA | WARD | STATUS | LATITUDE | LONGITUDE | SITE MANAGEMENT AGENCY (SMA) | SMA TYPE | REGISTRATION ACTIVITY | SUPPORT WASH | SUPPORT HEALTH | SUPPORT SHELTER/NFI | SUPPORT FOOD | SUPPORT PROTECTION | SUPPORT EDUCATION | SUPPORT LIVELIHOOD | SITE CLASSIFICATION | SITE TYPE | LAND OWNERSHIP | COMMON SHELTER TYPE | NO OF HOUSEHOLDS | INFANTS MALE | INFANTS FEMALE | CHILDREN MALE | CHILDREN FEMALE | YOUTH MALE | YOUTH FEMALE | ADULT MALE | ADULT FEMALE | ELDERLY MALE | ELDERLY FEMALE | TOTAL NUMBER OF IDPS | PREVIOUSLY BEEN DISPLACED | INTENDED RETURN AREA | WHY NOT RETURN | NO SHELTER | TENTS | MAKESHIFT | INDOORS | ACCESS ELECTRICITY | ACCESS SAFE COOKING | HAVE PRIVATE AREAS | HAVE MOSQUITO NETS | MOST NEEDED NFI | WATER SOURCE LOCATION | DRINKING WATER SOURCE | WATER CONSUMPTION | DRINKING WATER POTABLE | DRINKING WATER QUALITY COMPLAINTS | LATRINE CONDITION | FUNCTIONING TOILET | GARBAGE DISPOSAL | SOLID WASTE PROBLEM | HAND WASHING STATIONS | HYGIENE PROMOTION CAMPAIGN | OPEN DEFECATION | ACCESS TO FOOD | ACCESS TO MARKET | DISTRIBUTION FREQUENCY | OBTAINING FOOD | MALNUTRITION SCREENING | MOST PREVALENT HEALTH PROBLEM | ACCESS TO MEDICINE | ACCESS TO HEALTH FACILITY | LOCATION HEALTH FACILITY | HEALTH FACILITIES PROVIDER | ACCESS TO EDUCATION | EDUCATION LOCATION | ATTENDING SCHOOL | MAJORITY OCCUPATION | ACCESS TO INCOME | LIVESTOCK | CULTIVATION |
Iraq: rd_50_DTM_Master_List_July2016.xlsx
This is at an aggregated level and has two sheets. The main sheet is "DTM DATASET" with a two line header:
| Location of Displacement | Governorate of origin | Shelter type | Period of displacement | Link for Map | ||||||||||||||||||||||||||||||||||||||||||||
| Place id | Governorate | District | Location name in English | Location name in Arabic | Latitude | Longitude | OCHA admin 1 | OCHA admin 2 | OCHA PCode | Families | Individuals | Anbar | Babylon | Baghdad | Basrah | Dahuk | Diyala | Erbil | Kerbala | Kirkuk | Missan | Muthanna | Najaf | Ninewa | Qadissiya | Salah al-Din | Sulaymaniyah | Thi-Qar | Wassit | Camp | Host families | Hotel/Motel | Informal settlements | Other shelter type | Religious building | Rented houses | School building | Unfinished/Abandoned building | Unknown shelter type | Pre-June14 | June-July14 | August14 | Post September14 | Post April15 | Post March16 | Open Street Map | Google Map | Bing Map |
Yemen: r4_DTM_Master_List_Apr2016.xlsx
This has aggregated level data in different sheets. Sheet "Returnee hybrid data Locationlv" which has a single line header:
| AreaAssessmentID | AssessmentRound | Governorate | GovernoratePCode | DistrictEN | DistrictPCode | RetIDPsFromDistrict_HH | OfficialPlaceEN | PCode | Returnee_ConflictHH | Returnee_DisasterHH | Returnee_Accessibility | Returnee_WomenPer | Returnee_MenPer | Returnee_FemChildPer | Returnee_MaleChildPer | Latitude | Longitude |
Sheet "IDPs Raw data" has this single line header:
| AreaAssessmentID | Assessed Governorate | Governorate PCode | Assessed District | District PCode | Interview Date | Assessment Round | Site Name | Site Name A | Site PCode | Latitude | Longitude | Site Type | HH IDPs in District | Individual IDPs in District | Arrival Year | Arrival Month in 2015 | Site Estimated Conflict HHs | Site Estimated Disaster HHs | Site Estimated IDPs HHs | Family Size | Accessibility | Women % | Men % | Female Children % | Male Children % | Women Estimeted # | Men Estimeted # | Female Children Estimeted # | Male Children Estimeted # | Camps | Using rented accomodation | With host families who are relatives (no rent fee) | With host families who are not relatives (no rent fee) | Using schools, Health facilities, religious building | Using private or public building | In informal settlement (grouped families) in urban areas | In informal settlement (grouped families) in rural areas | Out of settlement (isolated families) |
Common Fields
From the headers above, there are certain fields which are common to all. These are:
- Latitude
- Longitude
- Location Name (ward, district etc.)
- Families/Households
- Type of shelter (although this is represented in different ways eg. individual columns for each type or one column with the type)
Conclusion
Given the lack of any consistency between the DTM Master List spreadsheets, writing a one size fits all automated data checker and cleaner for all of them is challenging. It would involve placing a great deal of "intelligence" into the cleaning program with the possibility that errors are introduced during cleaning for example, by accidentally matching the wrong column heading when making an algorithm to match a very diverse range of names. It may be possible to write cleaners per country but the effort involved would be large.
A better approach is to try to encourage the different offices to use a similar template for their spreadsheets, as from this starting point, writing a cleaner would not be too onerous. If such a template were introduced, it could be HXLated from day one. How easy it would be to invent a template that covers the range of needs is debatable but it is likely to be much easier than trying to process greatly varying spreadsheet formats.