Limitations
The data team would like to highlight the following caveats and limitations that should be kept in mind when evaluating the data inputs, process, and outputs.
This information is pulled from the appendix of the data process guide
The data team would like to highlight the following caveats and limitations that should be kept in mind when evaluating the data inputs, process, and outputs.
-
Some people and Brigades provided input through multiple channels, and we have limited ability to identify this in the data.
-
Terms used in survey questions, especially related to pathways, were not clearly defined or consistently understood by respondents. Many survey questions did not specifically focus on a future vision for the Brigades.
-
The open, volunteer-engaged tagging process meant there was little opportunity to promote inter-rater reliability in tagging.
-
Tag counts only represent the number of responses associated with a given tag and should not be used as a proxy for support or importance.
-
We used Pol.is at several points during the process. The first use, open from October 2020 through February 2021, helped shape the early theory of change pathways and involved quantitative analysis of 97 participants’ agreement with statements. The second case involved the forum team using pol.is as an interaction tool during the public-facing forums; these forum Pol.is outputs were not used for data analysis.
Some groups were overrepresented in the survey data (NAC Election and Townhall Typeform Surveys). The Pacific region had more responses than other regions. Areas with high engagement in the NAC elections had more responses (some Pacific, but also BTV and Code 4 Puerto Rico).
The survey questions asked on the NAC Election (ranking) and Townhall Typeform (score) surveys were not in the same format, so they can’t be directly combined or compared.
Values and Narrative questions in the Hattaway survey were not presented in random order, so more complete responses were provided for the first value (Humility) and narrative.
The code counts in the Codebook are not automatically updated. Code counts should only be used as a guide to how many responses are available for each tag. In some cases tags were automatically assigned to survey data based on the question that generated the response: Values, Narratives. See tagging instructions for specifics. Code counts should not be used as a proxy for support or importance. Pro-path and Anti-path tags might be a minor exception to this limitation.