The International Organization for Migration (IOM) is the leading inter-governmental organization in the field of migration and works closely with governmental, intergovernmental and non-governmental partners. It is dedicated to promoting humane and orderly migration for the benefit of all and works to help ensure the orderly and humane management of migration, to promote international cooperation on migration issues, to assist in the search for practical solutions to migration problems and to provide humanitarian assistance to migrants in need, including refugees and internally displaced people.

...

Country DTMs are not well-standardised, though IOM in Geneva is starting an initiative to define a common core set of columns. Some countries use databases to manage their DTMs, while others are entirely spreadsheet based. Note that as of 2016-09-23, most of the DTMs on HDX are operationally out-of-date (shared once but not updated regularly).

...

Conclusion

Given the lack of any consistency between the DTM Master List spreadsheets, writing a one size fits all automated data checker and cleaner for all of them is challenging. It would involve placing a great deal of "intelligence" into the cleaning program with the possibility that errors are introduced during cleaning for example, by accidentally matching the wrong column heading when making an algorithm to match a very diverse range of names. It may be possible to write cleaners per country but the effort involved would be large.

A better approach is to try to encourage the different offices to use a similar template for their spreadsheets, as from this starting point, writing a cleaner would not be too onerous. If such a template were introduced, it could be HXLated from day one. How easy it would be to invent a template that covers the range of needs is debatable but it is likely to be much easier than trying to process greatly varying spreadsheet formats.

Appendix: IOM Global DTM information

The following files contain a data dictionary and partial sample for the in-progress global DTM, as supplied by IOM in Geneva.

This Google Sheet contains proposed HXL hashtags for the global DTM: https://docs.google.com/spreadsheets/d/1gifTnrz9A2fZ8Tuwg-EClsFvu4QEbC4OUtdgb61dXGs/edit?usp=sharing