Evaluation of Boulder County Questionnaire

Report Date: June 30, 1999
Report Author: Kevin David Vryan
Indiana University
Department of Sociology
Ballantine Hall 744
Bloomington, IN 47405-6628
(812) 855-2833
kdv@indiana.edu

Please direct inquiries regarding this report to the author via the contact information provided above.

I. INTRODUCTION

I understand that there are community and political issues in Boulder, Colorado, related to the recent use of this survey instrument as commissioned by Boulder County. I do not support any particular political position and I have formed no opinion regarding these matters. My purpose here is solely to evaluate the survey research instrument discussed herein and comment briefly on the implications of this evaluation.

Item 1.

1. very much in the right direction
2. somewhat in the right direction
3. somewhat in the wrong direction, or
4. very much in the wrong direction"

Item 2.

1. This is a well-formed question, as long as the interviewer is trained and monitored to avoid directing the respondent while probing for a response.

Item 3.

1. This is also a well-formed question, given that interviewers do not direct responses during probing.

Item 4.

1. This question is generally a good question, although additional response categories would yield more detailed data. Thus, "much more," "a little more," about the same," "a little less," and "much less" would offer respondents more choices and would likely yield a better indication of public opinion. The use of response options such as "much more" and "a little more" (or "somewhat more") is fairly standard in survey research.

Item 5.

1. As with Item 4, this question is generally a good question, although additional response categories would yield more detailed data. Thus, "much more," "a little more," about the same," "a little less," and "much less" would offer respondents more choices and would likely yield a better indication of public opinion.

Items 6 through 11.

1. The introductory paragraph is well-formed.
2. The possible objectives asked about are well-constructed, with the exception of item 11. The word "scarring" in item 11 is value-laden (in a clearly negative way) and therefore introduces bias to the question. It is unlikely that most people would agree with any statement that asked them to support "scarring," but the definition of a particular impact as a "scar" is purely subjective and variable. If two people were looking at the same view, one might define it as a scar, and the other might not. As examples of more value-neutral wordings for this item, it could have been worded: "To limit the impact caused by constructing roads to new houses," or "To limit the visual impact caused by constructing roads to new houses."
3. Regarding item 11, there is no clarification that the respondent is being asked to comment on temporary effects to the environment versus longer-lasting effects. Some respondents may answer the question with a construction zone in mind that they would expect would be temporary, while other respondents may answer the question thinking only of permanent or long-lasting effects. This ambiguity leads to threats to both reliability and validity.
4. The intention behind rotating questions 6 through 11 was likely to minimize bias in the form of "contamination" of an item by a previous one, at least in any consistent manner. However, due to the value-laden term "scarring" used in item 11, bias may have been introduced to any items that followed it. Given the existing wording of items 6 through 11, leaving item 11 as the last item would have minimized contamination of the items that followed.
5. Adding the word "generally" before the options of "agree" or "disagree" in the introductory paragraph would have helped respondents choose between only two possible answers. Additional response options would have made it even easier for respondents to answer this series of questions, and is routinely done in survey research. Thus, "strongly agree," "somewhat agree," "somewhat disagree," and "strongly disagree" would have yielded data more informative of the population’s views.

Item 12.

1. A less biased form of this question would be: "Generally, do you think Boulder County should spend less or more money to purchase open space even if it means a tax increase?" (with response options being "more," or "less," don’t know, or refuse to answer). As it is worded now, there is an increased likelihood that respondents may answer in the affirmative compared to the alternative wording I proposed.
2. The word "even" is unnecessary and leads the respondent to disapprove of what follows it. In survey research, the respondent should never be directed toward any particular position or response.

Item 13.

a. This is a well-formed question. Additional response categories would offer more flexibility in response and richer data ("strongly support," "somewhat support," "somewhat oppose," "strongly oppose").

Item 14.

a. This is a well-formed question. Additional response categories would offer more flexibility in response and richer data ("strongly support," "somewhat support," "somewhat oppose," "strongly oppose").

Items 15 through 20.

1. The introductory paragraph asks respondents to answer questions 15 through 20 "…from the viewpoint of landowners’ rights…" This is a clear introduction of bias in all of the following questions. The respondent could have been asked to give their (own) opinions, but were rather prompted by the interviewer to answer the questions from a particular point of view, which may or may not have been their own. Therefore, the data from these items about land use objectives (items 15 through 20) are not valid if the intention is to determine the respondents’ opinions. It cannot be known if respondents gave answers that they thought landowners would give, or their own.
2. The introductory paragraph also introduces bias in the use of the word "even" in the last sentence of the introduction, which the interviewer was instructed to repeat to the respondent as necessary throughout this series of questions. This word is not at all necessary to communicate the meaning of the sentence, and it implies that the following statements are all "bad" things. This would have the effect of directing the respondent to disapprove of whatever follows.
3. Items 15 and 18 use the highly subjective and seemingly unnecessary term "significant," thus introducing (or enhancing) bias in the questions. When such a term is used, it must be defined for the respondent so that, in effect, the same question is being asked of all respondents regardless of their variable interpretations.
4. Item 18 uses the highly value-laden term "scar." As with item 11, this is likely to lead respondents to a disapproving attitude, even if they might not define a specific effect as a scar without prompting.

Item 21.

1. This is generally a well-formed question, although it might have been appropriate to add an "other" response option in the event that other alternatives were available in the opinions of respondents.

Item 22 (asked only in mountain precincts).

1. The use of the word "we" is unclear. Some respondents may interpret that word as referring to them as mountain residents or citizens, while others may interpret that word as referring to "county government" (which could also be more clearly defined). Thus, this questions suffers from a lack of validity. Further, if a respondent asks for the question to be reread by the interviewer, he or she may interpret the question differently the second time it is heard (after interpreting the "we" from the first sentence in the context of the second sentence), thus threatening reliability.
2. The phrase "not county government’s place" is unnecessarily biased. I see no reason that a more balanced choice shouldn’t be offered to respondents. An example of more appropriate wording would be (assuming that the "we" is supposed to refer to county government): "Some mountain residents believe that county government should do more to limit the number of new houses built in the mountains, while others believe that county government should do less to limit the number of new houses built in the mountains. Others believe that county government should continue doing what they are now" (with response options such as "more," "less," "the same," don’t know, and refuse to answer).

Item 23 (asked only in mountain precincts).

1. I expect that item 23 should only be asked of respondents who feel that the number of new houses built in the mountains should be reduced. If respondents do not feel that the number of new houses built in the mountains should be reduced, this question seems irrelevant to them, as they would not be likely to support a tax to achieve an end they disapprove of.
2. This question asks for an opinion on a possible ballot issue after attempting to describe how a proposed mountain local improvement district might work, and what impact it would have. However, this issue does not seem to me to be presented in a complete and impartial way. The question seems to be intended to elicit a favorable response, rather than a value-neutral probe of respondents’ assessments of legislation that I imagine would have various implications and impacts not presented. If the issue is not familiar enough to a respondent that they have an opinion on the matter without a detailed explanation of the issue, the answer should be "don’t know" after a brief question as to whether the respondent would vote for or against the ballot item. Alternatively, the issue could be presented in a more balanced way, indicating what supporters say (with text approved by supporters) and what opponents say (with text approved by opponents).
3. It is unclear to me why this question is asked only of respondents in mountain precincts, although I imagine that the questionnaire authors had some purpose in mind. As the introductory paragraph explains, Boulder County tax funds (apparently collected from taxpayers not living in the mountains as well as from those living in mountain precincts) would be provided for the proposed improvement district. As this question is not asked of all taxpayers or voters, I assume that its intention is not to assess public opinion of all affected parties on the proposed legislation. If this question is to affect policymaking, I would expect that all relevant parties would be polled.

Item 24 (asked only in mountain precincts).

1. As with item 23, it seems that item 24 should only be asked of respondents who feel that the number of new houses built in the mountains should be reduced.
2. As with item 23, the issue does not seem to be presented in a complete way. Either the respondent already knows about this option and has an opinion, or he or she is being offered a limited presentation of an issue that is in actuality likely to be very complicated and with differing points of view, and then is asked to offer an opinion based on this limited presentation.
3. I am not privy at this time to the intentions of those who constructed or commissioned these questions (items 24 through 26), and it is unclear to me what the intention of these follow-up questions are. I note that they are asked only of those who indicated that they supported the proposition as presented in item 23. It seems to me that questions 24, 25, and 26 are designed to assess either how much of a property tax increase supporters will tolerate before withdrawing support, or how much undeveloped land they desire the proposed local improvement district to acquire. Either way, it could not be known from the data what is actually being measured, and thus the data from these questions suffer from threats to validity.

Item 25 (asked only in mountain precincts).

1. See comments regarding item 24.

Item 26 (asked only in mountain precincts).

1. See comments regarding item 24.
2. I note that this series of questions (items 23 through 26) seems to attempt to assess mountain precinct residents’ positions regarding a single potential ballot issue. This may have been the sole intention of those who designed and commissioned this study, and this singular inquiry may provide all the information desired by them. It is not unheard of to conduct very brief surveys that only ask about one aspect of an issue, but as this survey was commissioned by a government entity and not a special interest group, I would have expected that various options to dealing with this issue would have been presented and asked about in order to determine the public’s policy preferences. At the very least, I would think that policymakers would be interested in an open-ended question asking respondents about what they would like to occur in relation to this issue.

Item 27.

1. This is a fairly standard question and is fine as it is. I assume that interviewers are trained as to how to deal with responses that indicate discontinuous residence (such as someone who lived in Boulder for 3 years in the 1980s, and then moved back into Boulder last year).

Item 28.

1. Age is also a standard question, although asking date of birth is usually the preferred method as it allows for calculation of the exact age of respondents at any point in time, not just at the time of the survey. However, if this is a one-time poll and specific ages and averages are not important, these response categories would work.

Item 29.

1. The response options in this item are poorly constructed. They are not exhaustive, in other words, they do not account for all possible responses (for example, living with family at no cost).
2. The use of the word "home" may imply to some respondents a house, as opposed to an apartment. The word "residence" or the phrase "where you live now" (in a slightly modified question form) would minimize this implication. Further, the use of the word "your" may imply to some respondents that they are expected to be the primary occupant (this may be an issue if the respondent is living in what they consider to be primarily someone else’s residence). To avoid these potential problems, wording could be offered such as: "Where you live now [or, alternatively, "Where you have lived for more than six months in the past year), do you rent or own?" (with the interviewer having an option for "other" responses that the respondent may volunteer).
3. Note that if this poll was conducted during the Summer in a college town, the answers might be very different than during the school year. It is unclear as to how a student who normally lives elsewhere might answer this question in the absence of a response category or clarification to account for such possibilities. Further, there is no question confirming that the respondent lives the majority of the year at the residence in Boulder County (or in the mountains). Thus, if the questionnaire was administered during the Summer, there is no way to distinguish Summer residents from year-round residents. These matters threaten the reliability of the measure.

Item 30.

1. It is unclear to me why these response categories were used, but I assume that educational attainment is not an important variable in this project. Normally, more complete response options are used. For example, I note that less than high school or equivalent is not specifically coded for (for example, "elementary school graduate," "some high school (no degree)"), and that there is no distinction made between those who have attained a Ph.D. and those who have taken one graduate class for no credit without being admitted to a graduate program. There are standard and widely used response options for educational attainment that would normally be used, such as those used by the U.S. Census.

NOTE TO INTERVIEWER:

[(Thank respondent and then complete questions 31 through 35 from your calling sheet information. Double check to make certain that you got all of them.)]

Item 31.

1. This is a pretty standard item, although at the Center for Survey Research, we have a note to the interviewer to ask this if necessary, as is occasionally, albeit rarely, the case. In this study, the interviewer has already hung up the phone before coding this item.

Item 32.

1. Since the interviewer has already hung up the phone, I assume that this information is taken from official voting registration records. While this is informative of the affiliation a respondent claimed the last time they registered to vote, it most definitely does not represent the way they see themselves at the time of the interview. It may not even represent how they saw themselves at the time they registered. For example, many people register an affiliation with a political party just so that they can vote in a primary election. While I would not eliminate this question, if party identification was important at all (which I assume it is, since this question was included), I would certainly add another question that was asked of the respondent while he or she was still on the phone about current party identification.

Item 33.

1. A respondent may be living in a precinct other than the one on record, or may live more than half the year in another precinct. For example, a student may live elsewhere, but be answering this survey while staying at their parents’ house for the Summer, or this may be a Summer or vacation residence. Also, a respondent may have recently moved. There is no item in this instrument to confirm current or primary address (and therefore precinct of primary residence).

Item 34.

1. See comment regarding item 33.
2. Note that I have omitted the response categories for this item in my copy of the text. They consist of a series of numbers only.

Item 35.

1. This is apparently a coding not relevant to my analysis.

III. SUMMARY AND CONCLUSIONS

While survey research done properly is a very valuable tool, when it is done poorly it can be a dangerous thing, leading us to draw conclusions based on fiction veiled as "scientific" fact. Science can be used for the common good, and it can be misused to achieve political or ideological ends. The misuse of polling and "statistics" (as poll results are often called) has led to public skepticism regarding the value of survey research. The possible misuses of the data collected via the Boulder County Questionnaire are far too numerous for me to detail here. While I am a professional survey researcher myself and believe in the promise of polling, I would have to count myself as among those who are appropriately skeptical of any data resulting from the Boulder County Questionnaire in its current form.

For more information contact the Land Use Coalition at info@landusecoalition.org or call 303-666-7903.

home | top

Last updated January 24, 2002.
URL: http://landusecoalition.org/evaluation_of_BC_questionnaire.htm
Page created  by Patrick and Patty Baker
http://home.earthlink.net/~patrickmbaker
Hosted by The Sugarloaf Internet Cooperative.