Proper Methodology and Methods of Collecting and Analyzing Slavery Data : An Examination of the Global Slavery Index

The Global Slavery Index aims to, among other objectives, recognize the forms, size, and scope of slavery worldwide as well as the strengths and weaknesses of individual countries. An analysis of the Index’s methods exposes significant and critical weaknesses and raises questions into its replicability and validity. The Index may prove more valuable in the future if proper methods are implemented, but the longer improper methods are used the more damage is done to the public policy debate on slavery by advancing data and policy that is not based on sound methodology. To implement proper methods, a committee of sophisticated methodologists needs to develop measurement tools and constantly analyze and refine these methods over the years as data is collected.


Introduction
The difficulty in generating reliable, replicable, and scientifically sound primary data makes it hard to monitor change and policy effectiveness that is often required to secure and maintain attention and funding for the issue of slavery.Both policymakers and activists have a common interest in ensuring better data (Dottridge, 2003, p. 47).This paper analyzes the Global Slavery Index produced by Walk Free for proper and adequate methodologies and methods (Walk Free Foundation, 2013).Where inadequate methodology and methods are found then proper methodologies and methods are discussed to help the antislavery field progress.By examining the strengths and limitations of this first index of its kind, i.e. the Global Slavery Index, this paper aims to enhance and contribute to the perennial debate of "what is good research" and ultimately support the curbing of slavery.
Much antislavery research is based on relatively small samples of survivors, usually identified by law enforcement agencies or persons assisted by NGOs or international organizations (Laczko, 2007, p. 39).This often leads to biased estimates constructed by "expert" and NGO opinions that are not subjected to methodological scrutiny by external actors.Numbers are given weight not based on methods, but on the authority of the person or organization that provided the estimate.Even if different groups have similar findings, respondents operating in this area do not work in a vacuum and are often influenced by one another.It is essential then that the use of secondary sources are limited if re-liable findings are to emerge.Instead, the focus should be on primary data collection and analysis.The Index's goal to collect primary data is admirable and will take enormous amounts of resources, but this was not done in the initial Index and needs doing in the future if the Index is to enjoy credibility.
The Index is a quantitative study using three weighted factors of estimated prevalence (95%), in-and out-migration (2.5%), and child marriage (2.5%) to reach an estimate that 29.8m people are currently living in modern slavery.How these figures are constructed for each country is unclear from the information given in the Index and the estimate is significantly higher than the well-accepted estimate of 20.9m people by the International Labour Organization (ILO) (2012).However, the critique of the Index is not meant to diminish the efforts put forth by Walk Free, but to point out areas that must be strengthened for the Index to be used to help accomplish the goal of reducing contemporary slavery.
Slavery target populations are often difficult to collect from, due to their hidden nature and often lack of cooperation with data collectors.One of the most significant challenges in slavery research is the quantification of victims who are often latent either because traffickers hide them from view or because rescued victims seek anonymity to prevent further victimization.The reasons for not cooperating vary (e.g.stigma, fear), but the results are often the same, i.e. unreliable data.Perhaps the most important aspects of collecting data from these target populations is ensuring that the methodology and methods are explicit, as well as making clear the extent to which the sample is and is not representative.The Index is inadequate in these and other aspects.

Methodology
The Index states multiple objectives including: being a tool to understand all forms of slavery in size and scope, recognition of country strengths and weaknesses, development of country policy, evaluation of the progress of policies and programs, and identification of how individuals enter into slavery (Walk Free Foundation, 2013, pp. 10-11).Accomplishing these objectives requires proper methodology and while methodological divisions are profound, proper methodology has suitable definitions; encourages validity, reliability, and accurate measurements and findings; discourages guesses; organizes the study; and allows for replicability and validating.
Proper methodology is called into question from the beginning pages of the Index.The term "main prevalence factor" is defined in multiple ways.The Executive Summary defines the main prevalence factor as the "estimated prevalence of modern slavery by population" (Walk Free Foundation, 2013).The Index later defines the main prevalence factor as "a composite estimate of the number of people in slavery in each country" (Walk Free Foundation, 2013, p. 11).The Index later defines the main prevalence factor in the methodology section as the "estimated prevalence of modern slavery in each country" (Walk Free Foundation, 2013, p. 110).
In addition to definitions being inconsistent and used indiscriminately throughout the Index, methodological issues are compounded by the difference between "composite estimate" and "estimate" and the fact that they are two different concepts.A composite estimate is a statistical estimation using multiple data sources, while an estimate is a judgment or approximation-an estimate has more potential for bias and or inaccuracies.The Index suggests that research teams and experts are used in certain countries.This evokes an image of an estimate being made by the research team or expert.The Index is not clear when "estimates" are used and when "composite estimates" are used or how they are constructed.Furthermore, there is no discussion of how the participating countries and the associated data collection address the hidden nature of slavery, participation reluctance, and potential slavery stigmatization in each country.
The Index admits that its estimates have flaws, but base the flaws on everything except its own methods, i.e., the Index suggests the flaws are based in the hidden nature of slavery and flaws already embedded in the other works used (i.e., in the secondary source and survey data) (Walk Free Foundation, 2013, pp. 111-112).While these are important to note as possible causes of error, the Index does not mention or properly discuss that the entire Index is not grounded in scientific method; that only seven countries have survey data that is based on only four sources and is used to extrapolate data for 155 other countries; the secondary sources and which countries the secondary sources are used in; which surveys are used, how they are used, for which countries they are used; how figures are calculated and used; the composition of the research team(s), how research teams perform their analysis or extrapolations, or a general account of the qualifications and quantity of experts used; which countries used which methods; the criteria used in choosing how countries were placed in the various ordinal categories; the variables used; or how countries were ultimately grouped for extrapolation or the criteria used for extrapolation for each country.Additionally, the Index gives the initial impression that it already performed primary data collection itself, when it did not.Its first year of primary data collection is in 2014 not 2013 when the original Index was published.
Also, a lack of attention appears to plague the Index.Correlation scatterplots have mislabeled horizontal axis (compare Human Development Index and Access to Financial Services sub-labels).There is a presence of incomplete or at least confusing numbered horizontal axis (see Access to Financial Services where from left to right the axis numbers read 02, 55, 07, 5, 100); there are negative relationships indicated where it appears positive (see Access to Financial Services) or at the very least is not visually represented well (Walk Free Foundation, 2013, pp. 104-107).Furthermore, the Index performs correlations on four of 33 variables, but does not explain why these four were chosen or what other findings were discovered through performed correlations with the other variables not mentioned (Walk Free Foundation, 2013, pp. 104-107, 114-115).If calculations were performed with the other variables, then it needs discussing or, at a minimum, the Index should include a statement as to why the authors chose not to include the other variables.
Acknowledging difficulties in measurement or flaws in the data does not give license to produce estimates that are not testable by other researchers.Because of these factors and others, the Index cannot be validated or replicated and without providing means to confirm and test research measurements, the creators of the Index are engaged in producing propaganda rather than social science (Steinfatt, 2002).

Defining Target Populations
Explicit and proper identification and expression of a target population lies at the foundation of constructing valid primary data (Tyldum, 2010, pp. 2-3).The Index's target population includes current victims in "all forms of modern slavery" within 162 countries.Additionally, a second part of the Index investigates individuals at-risk of slavery in those countries.The Index (Walk Free Foundation, 2013) uses the following definition of slavery: Slavery is the possession and control of a person in such a way as to significantly deprive that person of his or her individual liberty, with the intent of exploiting that person through their use, management, profit, transfer or disposal.Usually this exercise will be achieved through means such as violence or threats of violence, deception and/or coercion.(p.11) The definition is constructed using several internationally accepted definitions of slavery, human trafficking, forced labor, child labor, and domestic work from the League of Nations, United Nations, International Labour Organization, and the Council of Europe (Walk Free Foundation, 2013, p. 2).
However, the Index does not adequately explain how their newly constructed definition is superior to others already in use.It claims that other definitions are legally complex, have existing debates on how they "overlap or should be reconciled," and are used differently by various countries (Walk Free Foundation, 2013, p. 11).For example, the League of Nations (1926, art. 1) and the United Nations (1956, art.7) define slavery as: (1) Slavery is the status or condition of a person over whom any or all of the powers attaching to the right of ownership are exercised.
(2) The slave trade includes all acts involved in the capture, acquisition or disposal of a person with intent to reduce him to slavery; all acts involved in the acquisition of a slave with a view to selling or exchanging him; all acts of disposal by sale or exchange of a slave acquired with a view to being sold or exchanged, and, in general, every act of trade or transport in slaves.
Comparing this already established definition with the Index's new definition shows no material difference either theoretically or operationally.Both definitions are equally complex, have similar characteristics for debates on how they overlap and should be reconciled with other definitions, and are used differently by various countries.If one is going to construct and enter a new definition into an already congested arena, then it should explicitly and thoroughly explain how the new definition is superior to all others.If not, then it simply packs another definition into an already overcrowded debate.Furthermore, it is not clear where the Index's definition is used in the study.First, the "primary" data appears to come from other organizations' studies that were performed prior to construction of the Index.So, it is unclear how the Index is defining the term primary data as it does not appear that the Index collected the data first-hand.For instance, the data from Haiti was collected by the Pan American Development Foundation and USAID in 2009, not from Walk Free in 2013 (Pan American Development Foundation & USAID, 2009;Walk Free Foundation, 2013, p. 111).The definitions used in the Haiti report are attributed to the US Trafficking Victims Protection Act (TVPA) and UN Conventions (Pan American Development Foundation & USAID, 2009, pp. 12-13;United Nations, 2000a, 2000c;US Congress, 2000).The Index's definition is not used in the collection of "primary" data.
Second, the Index extrapolates data from countries that had primary source data collection performed.Once again, it is not clear how the Index uses its newly formed definition.The organizations that performed primary source data collection used definitions other than the Index's definition.
Third, the Index's secondary data does not use the Index's definition.The Index does not disclose what secondary sources were used so a proper review of the secondary data sources is not possible.However, for India's assessment, the Index gives the example of the US State Department Country Reports on Human Rights Practices as being used as a secondary source (Walk Free Foundation, 2013, p. 110).The State Department Country Report states that human trafficking is used as an umbrella term for slavery and that "the United States government considers trafficking in persons to include all of the criminal conduct involved in forced labor and sex trafficking, essentially the conduct involved in reducing or holding someone in compelled service" (Patterson, 2012 , 2013).This example accounts for only a fragment of one secondary source used in one country and demonstrates how many different definitions are applied to each country in the Index.The one definition that is surely not used in the secondary sources is the definition constructed by the Index.It is then perplexing why the Index constructed its own definition, how its definition is used in its own study, or how it contributes to the understanding of slavery.The problem with using numerous definitions, especially in the case of comparing countries, is that without using the same definition within and crosscountry the study becomes invalid.

Constructing the Global Slavery Index
The Index is constructed using the three weighted factors of estimated prevalence (95%), in-and outmigration (2.5%), and child marriage (2.5%).How these figures are constructed for each country is unclear and there is no discussion on why the Index chooses these percentages.However, the Index does mention that while the prevalence factor represents 95% of the weight in the Index it is also the "weakest, most approximate, measure" (Walk Free Foundation, 2013, p. 11).The first factor of estimated prevalence has four different described methods in which it is constructed in the Index.First, the Index states: The data that underlies the Index was not collected specifically for the preparation of the Index but some of the data (notably the estimates of prevalence by country) had been compiled and estimated by the in-dex's author, Kevin Bales, with Free the Slaves, prior to this index (Walk Free Foundation, 2013, p. 120).
This account contends that Bales estimated the slavery prevalence for each country prior to the Index.The problem with this account is that it appears discredited by the following three methods discussed, and does not discuss how Bales' performed his analysis.
Second, the Index states that a country's slavery prevalence is constructed using survey data previously collected by other researchers and organizations.Seven of the 162 countries in the Index had prior survey data collected on their slavery prevalence: Belarus, Bulgaria, Haiti, Moldova, Niger, Romania, and Ukraine.The data was collected through four studies: ILO study on Moldova; ILO study on Niger; Pan American Development Foundation and USAID study on Haiti; and Pennington et al. on Belarus, Bulgaria, Moldova, Romania, and Ukraine (Walk Free Foundation, 2013, p. 111).From these studies, the slavery prevalence for each of the seven countries was constructed for the Index.For example, the Pennington et al. survey estimated 115,662 individuals were trafficked in Ukraine in 2006 (Pennington, Ball, Hampton, & Soulakova, 2009, p. 130).Using this estimate and the population of 46,710,816, Pennington et al. ( 2009) calculated a slavery prevalence for Ukraine of 0.002476129.Using this slavery prevalence the Slavery Index adjusts the 2006 number from 115,662 down to 112,895 for 2013 due to a drop in Ukraine's population from 46,710,816 to 45,593,300.For these seven countries, this is the method used to estimate the slavery prevalence and number of trafficked victims.
Unfortunately, the Index fails to thoroughly analyze or discuss these four studies.The Index does not discuss the studies' data limitations, what survey data may be inflated or deflated, or the surveys' rational behind the inflation or deflation.These are essential aspects in ensuring validity when using other data to construct new data.The Index's methodology section briefly mentions that it uses survey data but does not state in the main text which surveys are used, how they are used, or for which countries (Walk Free Foundation, 2013, p. 111).Instead, four studies are mentioned in the footnotes of the methodology section leaving the reader to assume these are the four studies used in the Index.
For instance, the Pennington et al. (2009) study is minimally discussed to demonstrate how the Index uses the survey data, but does not go into detail.It shows that the Pennington et al. (2009) study estimated trafficking victims at 115,662.The Index then states that it created a confidence range for the estimate of 110,000 to 120,000.It does not explain why it uses this range, how the range was calculated, that it readjusted the estimate down to 112,895 victims, how it readjusted the estimate down to 112,895 victims, that it calculat-ed a new range of 107,250 to 118,540, or how these numbers are used.One may assume that the Index used a 95% confidence interval to determine the range of 110,000 to 120,000, but it is significant that the Pennington et al. study calculated its own 95% confidence interval for the same figure at between 53,512 to 177,813 (Pennington et al., 2009, p. 130).
Third, the Index states a country's slavery prevalence is extrapolated from the four studies.The Index states: Because [representative sample data] information does not exist for many countries, the Index uses representative sample data to statistically extrapolate the prevalence of modern slavery for select countries that have not yet had random sample surveys.As only a limited number of representative surveys have been conducted, it was necessary to try to group the 162 countries so that the ratio of prevalence that is most relevant could be applied.To do this, a country-by-country assessment was made based on commonalities between the survey countries and the remaining countries on such factors as GDP, levels of in-and out-migration, levels of conflict and stability.The aim was to use both statistical comparisons where available and expert knowledge to group countries along the hypothesized range of prevalence proportions (Walk Free Foundation, 2013, p. 111).This contends that survey data collected in "a limited number" of countries is used to extrapolate the slavery prevalence of other countries.The Index never explicitly states the number of countries where survey data collection was performed and extrapolated.The seven countries mentioned previously are simply thought to be the only ones used in the Index due to the footnotes.The Index also gives the impression that Walk Free performed primary data collection.It is not clear that Walk Free performed no primary data collection for the Index, unless one really gets into the methodology section-including the footnotes of the section.This suggests that up to 155 countries' slavery prevalence were extrapolated from four data sources used to describe seven countries.
These 155 countries were matched with one of the seven countries using statistical comparisons such as GDP, levels of in-and out-migration, levels of conflict, and stability-as well as expert knowledge (Walk Free Foundation, 2013, p. 111).Probing these examples raises several questions on the origins of the variable data.It is not clear where the "level of conflict" variable data originates.There is no variable in the codebook labeled "level of conflict" or "conflict."There is a "Peace Index" variable that may have been used, but this is simply a guess as it is not explained in the report (Institute for Economics & Peace, 2013).It is not clear where the "stability" data originates as there is no var-iable in the codebook labeled "stability."There is a "Political instability" variable and a "State Stability" variable, either or both of which could have been used, but this is simply a guess as it is not explained in the report (Walk Free Foundation, 2013, p. 111).
Fourth, the Index states that secondary sources are used to construct a country's slavery prevalence when survey data and extrapolation are not possible (Walk Free Foundation, 2013, p. 113).The Index states: From the information gathered from secondary sources and analysis by the research team, an estimate was made of the prevalence of slavery in each of the 162 countries.These estimates were then presented to experts with personal knowledge of a country, a region, or an industry, often with a promise of anonymity.They compared the information that had been collected with their own knowledge and suggested which points might be exaggerations, which might be under-or overestimates, and which might be indicative of social reality (Walk Free Foundation, 2013, p. 110).
This contends that secondary source data was collected by a research team and used to create the slavery prevalence for all 162 countries in the Index.There is no mention of the composition of the research team or how the team performed its analysis.Estimates are then presented to experts to adjust the slavery prevalence.The experts are given anonymity, which is a completely acceptable practice, but the Index does not give a general account of how individuals are determined to be experts (e.g.academic, government employee, NGO worker) or the number of experts consulted.Just because an individual worked on the issue of slavery for a lengthy period, does not necessarily make them an expert on the prevalence or magnitude of slavery (Steinfatt, Baker, & Beesey, 2002, p. 3).Without a scientific study, experts are simply giving guesses and the "average of a set of wild guesses is simply an average wild guess" (Steinfatt et al., 2002, p. 2).
Additionally, it is difficult to discover which countries were used in which method.In some places in the Index it suggests that survey data was used for seven countries, extrapolation was used for 155 countries, and secondary data was used for 162 countries (Walk Free Foundation, 2013, pp. 110-111).This is contradictory unless another method was used that incorporated primary collected data, secondary data, and extrapolation.Other places in the Index suggest that survey data was used for seven countries, extrapolation for all others possible countries, and secondary data only for countries that cannot utilize survey data or extrapolation (Walk Free Foundation, 2013, p. 113).There is no list explaining which country used which method.
The other two weighted factors of the Index are Inand Out-Migration and Child Marriage.The in-and out-migration data comes from the US Department of State Trafficking in Persons Report 2013 where the data was "extracted by the research team and coded on an ordinal scale of 1−4 (where 1 reflects low levels of cross border human trafficking and 4 reflects high levels of cross border human trafficking)" (Walk Free Foundation, 2013, p. 113 and Codebook).The child marriage data comes from UNICEF and similarly extracted and coded into a 1-5 ordinal scale where 1 reflects low levels of child marriage and 5 high levels.However, the Index does not discuss the makeup of the research teams or the criteria used for converting the State Department's and UNICEF's data into the ordinal scales.In other words, there is no description of how a country is defined as a 1 instead of a 2, 3, 4, or 5 and therefore the measurements have no meaning and are not replicable.

Ranking of Risk
Ranking of Risk in the Index is a meta-analysis of 33 variables that are compiled into five separate dimensions.There is very little discussion of how the variables and dimensions are chosen other than to state the 33 variables are comprised of "sources external to the Global Slavery Index and were selected by the research team, on the basis of statistical testing of their relationship to the prevalence of modern slavery" (Walk Free Foundation, 2013, p. 114).The five dimensions are slavery policy (7 variables), human rights (7), development (9), state stability (7), and women's rights and discrimination (3) (Walk Free Foundation, 2013, pp. 114-115).Each variable is normalized and averaged to calculate each individual dimension score.All five dimension scores are then averaged to calculate the overall risk score (Walk Free Foundation, 2013, p. 115).
What the Index fails to realize or mention is that this method either intentionally or unintentionally weights some variables heavier than others.If all 33 variables were averaged together, then all would be equally weighted.However, for example, the development dimension consists of nine variables while the women's rights dimension consists of three variables.One the variables for the development dimension is Access to Financial services.This variable is first averaged with eight other variables to calculate the development dimension.In other words, the Access to Financial services makes up 1/9 of the development dimension.The development dimension is then averaged with the other four dimensions and therefore the development dimension makes up 1/5 of the overall risk score.So, the Access to Financial services makes up 1/9 of 1/5 of the overall risk score or 1/45 of the overall risk score.
Conversely, one the variables for the women's rights dimension is Gender Gini coefficient.This variable is first averaged with two other variables to calculate the women's rights dimension.In other words, the Gender Gini coefficient makes up 1/3 of the women's rights dimension.The women's rights dimension is then averaged with the other four dimensions and therefore the women's rights development dimension makes up 1/5 of the overall risk score.So, the Gender Gini coefficient makes up 1/3 of 1/5 of the overall risk score or 1/15 of the overall risk score.Each of the three variables in the women's rights dimension is then weighted three times heavier than each of the nine variables in the development dimension.This may or may not be an issue, but the Index should address why some variables are weighted heavier than others.

Conclusions
The formation and implementation of sound policy is not possible without sound data.The methodology and methods used in the Index are currently inadequate and therefore the Index cannot be validated or replicated.Furthermore, the publicity given to the Index is leading to the use of this poor data not only by popular culture and reputable magazines and news organizations (e.g.The New Yorker; New York Times; BBC), but also by academic journals and high level policy makers (e.g.former US Secretary of State Hilary Clinton; ex-British Prime Minister Tony Blair; Muhammad Yunus; Bill Gates) which can lead to inaccurate policy formulation and a compounding of harm (BBC, 2013;Datta & Bales, 2014, pp. 287-288;Nossiter, 2013;Okeowo, 2014;Walk Free Foundation, n.d.).It is a good objective of the Index to seek to perform primary data at the country level and then aggregate or disaggregate that data from the country level to regional and world levels, but the difficulty in estimating the size of a global problem is well known and is often exacerbated by pressures on researchers to produce numbers (even if it is beyond their means) in order to allocate funds and implement policy.Such figures generate swift action, but the potentially inaccurate figures may also generate equally inappropriate responses (Dottridge, 2003, p. 82).It is critical that researchers are honest with policymakers and funders about realistic expectations in research.Similarly, policymakers must respect those limitations with the understanding that focused, accurate, and limited data may better serve long-term goals.An important role for researchers and advocates is to make policymakers and funders realize that smaller numbers do not necessarily decrease the seriousness of the issue and that without insistence on validity and reliability, damage is often done to the overall cause if researchers lose credibility (Dottridge, 2003, p. 47).
Researchers can prevent their work from contributing to ineffective policy by strictly defining what they are studying.Since much funding is distributed based on the size of a problem, there is a temptation to include all facets of slavery in one index.Broad statements and large numbers appear helpful in gaining resources and attention for the issue; however, there are risks associated with overly broad definitions.Broad definitions may overwhelm donors with the size of the problem or impose donor fatigue in addressing a widespread and varied problem (Tyldum, 2010, p. 8).Stigmatization of a country that is striving to address the problem, but has high rates of modern slavery, is another real danger with such definitions.
Effective creation and monitoring of antislavery policies requires more focused and disaggregated numbers.Worldwide estimates do little more than suggest that the issue is worth addressing, but the enormity of resources used to attempt such aggregates is more efficiently spent developing and monitoring targeted projects and policies.Focused and replicable studies that use similar definitions and methodologies more accurately examine patterns and trends and regional or national studies that are sector or theme based lead to more effective policies (Tyldum, 2010, p. 8).
While passions surrounding issues like slavery drive the desire to move quickly and attack the problem, insisting on rigorous data collection and analysis helps ensure that gains made through quick action are not lost in the long term to misunderstanding and ineffectual policies (Dottridge, 2003, p. 47).The Index may prove more valuable in the future if proper methodologies and methods are implemented in the coming year or years, but the longer improper methodologies and methods are used the more damage is done to the public policy debate on slavery by advancing data that is not based on sound methodology.To implement proper methodologies and methods, a committee of sophisticated methodologists should be established (similar to what Transparency International did) that develops measurement tools and constantly analyzes and refines these methodologies and methods over the years as data is collected.
Unfortunately, the Index's first attempt in constructing a global quantitative analysis may do more harm to its reputation than good.Instead of being completely transparent on what the Index performed and did not perform, much was left to assumption and interpretation.The difficulty to generate reliable, replicable, and scientifically sound primary data produces difficulty for monitoring change and policy effectiveness that is so often required to secure and maintain attention and funding.In the end, both policymakers who demand the unattainable and the activists who give into those demands have a role to play in improving policymaking to address slavery.Advocacy and raising awareness are crucial to garnering support and funding, but the work of advocates should not replace proper scientific analysis.
; US State Department, 2014).It then attributes its official definitions to the TVPA and states that these are consistent with the definitions found in the United Nations Protocol to Prevent, Suppress and Punish Trafficking in Persons, Especially Women and Children (Palermo Protocol) (United Nations, 2000b; US Congress, 2000; US State Department, 2014).This only establishes the definition used by the US State Department in the Index's Country Reports.However, within each country report, numerous other organizations are referenced that use other definitions of slavery, for example the India report references the Asian Centre for Human Rights, India's Ministry of Labor and Employment, and others (Asian Centre for Human Rights, 2013; UNICEF, 2013; United Nations, 1948; US State Department