This section will guide you on how to ensure your research findings are representative and reliable.
In this section:
Collect your data
1. Before launch
- First determine the best way to reach your survey population.
For example, if you want to focus on indigenous women’s rights organizers, do you know who the key networks are? Do you have contacts there, people who can introduce you to these organizations or ways of reaching them?
- Determine if your key population can be easily reached with an online survey, if you need to focus on paper survey distribution and collection or a mix of both. This decision is very important to ensure accessibility and inclusiveness.
- Be prepared! Prior to advertizing, create a list of online spaces where you can promote your survey.
If you are distributing paper versions, create a list of events, spaces and methods for distributing and collecting results.
- Plan your timeline in advance, so you can avoid launching your survey during major holidays or long vacation periods.
- Make it easy for your advisors and partners to advertize the survey – offer them pre-written Twitter, Facebook and email messages that they can copy and paste.
2. Launch
- Send the link to the survey via email through your organization’s email databases.
- Advertize on your organization’s social media. Similar to your newsletter, you can regularly advertize the survey while it is open.
- If your organization is hosting events that reach members of your survey population, this is a good space to advertize the survey and distribute paper versions as needed.
- Invite your advisors to promote the survey with their email lists and ask them to copy you so you are aware of their promotional messages. Remember to send them follow-up reminders if they’ve agreed to disseminate.
- Approach funders to share your survey with their grantees. It is in their interest that their constituencies respond to a survey that will improve their own work in the field.
3. During launch
- Keep the survey open for a minimum of four weeks to ensure everyone has time to take it and you have time to widely advertize it.
- Send reminders through your email databases and your partners databases asking people to participate in the survey. To avoid irritating recipients with too many emails, we recommend sending two additional reminder emails: one at midway point while your survey is open and another a week before your survey closes.
- As part of your outreach, remember to state that you are only collecting one response per organization. This will make cleaning your data much easier when you are preparing it for analysis.
- Save an extra week! Halfway through the open window for survey taking, check your data set. How have you done so far? Run initial numbers to see how many groups have responded, from which locations, etc. If you see gaps, reach out to those specific populations. Also, consider extending your deadline by a week – if you do so, include this extension deadline in one of your reminder emails, informing people know there is more time to complete the survey. Many answers tend to come in during the last week of the survey or after the extended deadline.
If you also plan to collect data from applications sent to grant-making institutions, this is a good time to reach out them.
When collecting this data, consider what type of applications you would like to review. Your research framing will guide you in determining this.
Also, it may be unnecessary to see every application sent to the organization – instead, it will be more useful and efficient to review only eligible applications (regardless of whether they were funded).
You can also ask grant-making institutions to share their data with you.
See a sample letter to send to grant-making institutions
Back to top
Prepare your data for analysis
Your survey has closed and now you have all this information! Now you need to ensure your data is as accurate as possible.
Depending on your sample size and amount of completed surveys, this step can be lengthy. Tapping into a strong pool of detail-oriented staff will speed up the process and ensure greater accuracy at this stage.
Also, along with your surveys, you may have collected data from applications sent to grant-making institutions. Use these same steps to sort that data as well. Do not get discouraged if you cannot compare the two data sets! Funders collect different information from what you collected in the surveys. In your final research report and products, you can analyze and present the datasets (survey versus grant-making institution data) separately.
1. Clean your data
- Resolve and remove duplications: If there is more than one completed survey for one organization, reach out to the organization and determine which one is the most accurate.
- Remove ineligible responses: Go through each completed survey and remove any responses that did not properly answer the question. Replace it with “null”, thus keeping it out of your analysis.
- Consistently format numerical data: For example, you may remove commas, decimals and dollar signs from numerical responses. Financial figures provided in different currencies may need to be converted.
2. Code open-ended responses
There are two styles of open-ended responses that require coding.
Questions with open-ended responses
For these questions, you will need to code responses in order to track trends.
Some challenges you will face with this is:
- People will not use the exact same words to describe similar responses
- Surveys with multiple language options will require translation and then coding
- Staff capacity to review and code each open-ended response.
If using more than one staff member to review and code, you will need to ensure consistency of coding. Thus, this is why we recommend limiting your open-ended questions and as specific as possible for open-ended questions you do ask.
For example, if you had the open-ended question “What specific challenges did you face in fundraising this year?” and some common responses cite “lack of staff,” or “economic recession,” you will need to code each of those responses so you can analyze how many participants are responding in a similar way.
For closed-end questions
If you provided the participant with the option of elaborating on their response, you will also need to “up-code” these responses.
For several questions in the survey, you may have offered the option of selecting the category “Other” With “Other” options, it is common to offer a field in which the participant can elaborate.
You will need to “up-code” such responses by either:
- Converting open-ended responses to the correct existing categories (this is known as “up-coding”). As a simple example, consider your survey asks participants “what is your favorite color?” and you offer the options “blue,” “green,” and “other.” There may be some participants that choose “other” and in their explanation they write “the color of the sky is my favorite color.” You would then “up-code” answers like these to the correct category, in this case, the category “blue.”
- Creating a new category if there are several “others” that have a common theme. (This is similar to coding the first type of open-ended responses). Consider the previous example question of favorite color. Perhaps many participants chose “other” and then wrote “red” is their favorite. In this case, you would create a new category of “red” to track all responses that answered “red.”
- Removing “others” that do not fit any existing or newly created categories.
3. Remove unecessary data
Analyze the frequency of the results
For each quantitative question, you can decide whether you should remove the top or bottom 5% or 1% to prevent outliers* from skewing your results. You can also address the skewing effect of outliers by using median average rather than the mean average. Calculate the median by sorting responses in order, and selecting the number in the middle. However, keep in mind that you may still find outlier data useful. It will give you an idea of the range and diversity of your survey participants and you may want to do case studies on the outliers.
* An outlier is a data point that is much bigger or much smaller than the majority of data points. For example, imagine you live in a middle-class neighborhood with one billionaire. You decide that you want to learn what the range of income is for middle-class families in your neighborhood. In order to do so, you must remove the billionaire income from your dataset, as it is an outlier. Otherwise, your mean middle-class income will seem much higher than it really is.
Remove the entire survey for participants who do not fit your target population. Generally you can recognize this by the organizations’ names or through their responses to qualitative questions.
4. Make it safe
To ensure confidentiality of the information shared by respondents, at this stage you can replace organization names with a new set of ID numbers and save the coding, matching names with IDs in a separate file.
With your team, determine how the coding file and data should be stored and protected.
For example, will all data be stored on a password-protected computer or server that only the research team can access?
Back to top
Create your topline report
A topline report will list every question that was asked in your survey, with the response percentages listed under each question. This presents the collective results of all individual responses.
Tips:
- Consistency is important: the same rules should be applied to every outlier when determining if it should stay or be removed from the dataset.
- For all open (“other”) responses that are up-coded, ensure the coding matches. Appoint a dedicated point person to randomly check codes for consistency and reliability and recode if necessary.
- If possible, try to ensure that you can work at least in a team of two, so that there is always someone to check your work.
Back to top
Analyze your data
Now that your data is clean and sorted, what does it all mean? This is the fun part where you begin to analyze for trends.
Are there prominent types of funders (government versus corporate)? Are there regions that receive more funding? Your data will reveal some interesting information.
1. Statistical programs
-
Smaller samples (under 150 responses) may be done in-house using an Excel spreadsheet.
-
Larger samples (above 150 responses) may be done in-house using Excel if your analysis will be limited to tallying overall responses, simple averages or other simple analysis.
-
If you plan to do more advanced analysis, such as multivariate analysis, then we recommend using statistical software such as SPSS, Stata or R.
NOTE: SPSS and Stata are expensive whereas R is free.
All three types of software require staff knowledge and are not easy to learn quickly.
Try searching for interns or temporary staff from local universities. Many students must learn statistical analysis as part of their coursework and may have free access to SPSS or Stata software through their university. They may also be knowledgeable in R, which is free to download and use.
2. Suggested points for analysis
- Analysis of collective budget sizes
- Analysis of budget sizes by region or type of organization
- Most common funders
- Total amount of all funding reported
- Total percentages of type of funding (corporate, government, etc)
- Most funded issues/populations
- Changes over time in any of these results.
Back to top
Previous step
3. Design your survey
Next step
5. Conduct interviews
