Evaluation of Boulder County Questionnaire

Report Date: June 30, 1999
Report Author: Kevin David Vryan
Indiana University
Department of Sociology
Ballantine Hall 744
Bloomington, IN 47405-6628
(812) 855-2833
kdv@indiana.edu

Please direct inquiries regarding this report to the author via the contact information provided above.

I. INTRODUCTION

I have produced this report as a sociologist and independent consultant on behalf of the Land Use Coalition of Boulder, Colorado. I am trained and experienced in the conduct of survey research, including polling regarding community and political issues. I am currently employed as a survey researcher in the full-time position as Project Manager for Survey Research at the Center for Survey Research (CSR) at Indiana University in Bloomington, Indiana. I am submitting this report as an independent consultant, and not on behalf of the Center for Survey Research or Indiana University, or in my capacity as a CSR Project Manager.

This report provides an analysis of the survey instrument entitled "Boulder County Questionnaire" as provided to me by the Land Use Coalition.

In section II.: "Item-by-Item Analysis," each item of the questionnaire is provided in its entirety, followed by an analysis of that item. I have omitted the brief introductory statement read by the interviewer prior to the substantive portion of the interview, and the statement at the end of the interview that the interviewer signs certifying that they completely and accurately coded the responses. The item numbers used in this report are those used in the original survey instrument. Within the "TEXT" of questions, any comments within brackets and italicized are instructions to the interviewer and were not to be read to the respondent by the interviewer. The analysis that follows each item is numbered such that individual points are identified by lower-case letter ("a," "b," etc.).

In section III.: "Summary and Conclusions," general comments regarding survey research and the reliability and validity of the survey instrument evaluated in this report are provided.

I understand that there are community and political issues in Boulder, Colorado, related to the recent use of this survey instrument as commissioned by Boulder County. I do not support any particular political position and I have formed no opinion regarding these matters. My purpose here is solely to evaluate the survey research instrument discussed herein and comment briefly on the implications of this evaluation.

II. ITEM-BY-ITEM ANALYSIS

Item 1.

TEXT: First of all, would you say things in Boulder County are generally going in the right direction or do you feel things have pretty seriously gotten off track in the past few years?

    1. Right direction

    2. Off track

    3. No response

ANALYSIS:

  1. This question is poorly formed in that it asks the respondent to choose between a somewhat mild positive statement (as indicated in the use of the word "generally") and a relatively more severe statement (as indicated in the use of the word "seriously").

  2. The first option (right direction) is asked in the context of what a respondent would "say," while the second option (off track) is presented in the context of an emotional response ("…do you feel…"). This indicates a bias in the sense that the respondent is asked to compare a relatively benign thought against a feeling, and a feeling is likely to elicit a stronger response, thus introducing bias to this question. To eliminate this problem and offer a more balanced presentation of the choices, wording such as the following would be used: "In general, would you say (or do you feel) that things in Boulder County are going in the right direction or are off track in the past few years?"

  3. To make the choices that the respondent is given comparable, the same wording should be used in both responses. In particular, the phrase "off track" is a loaded phrase with clearly negative connotations, particularly in comparison to things going "generally" in the right direction. To make the choices comparable and unbiased, the choice "right direction" should be compared to the choice "wrong direction." A better wording would thus be: "In general, would you say (or do you feel) that things in Boulder County are going in the right direction or the wrong direction in the past few years?"

  4. There are only two response categories to a question that should provide more options to the respondent. The respondent is forced to choose between things going generally well, and things being "seriously" off track. More appropriate response categories to this question would be: "very much in the right direction," "somewhat in the right direction," "somewhat in the wrong direction," and "very much in the wrong direction" (with response categories for "don’t know" and "refuse to answer" also available to the interviewer). The use of "very" and "somewhat" options is fairly standard in academic research.

  5. Thus, a more sound wording of this question would be:
    "In general, would you say (or do you feel) that things in Boulder County are going in the right direction or the wrong direction in the past few years?
    Would you say:

    1. very much in the right direction

    2. somewhat in the right direction

    3. somewhat in the wrong direction, or

    4. very much in the wrong direction"

Item 2.

TEXT: In general, what do you consider to be the one most important issue facing Boulder County today? [(Open end. Probe.)]

ANALYSIS:

  1. This is a well-formed question, as long as the interviewer is trained and monitored to avoid directing the respondent while probing for a response.

Item 3.

TEXT: And what would be the next most important issue facing Boulder County today? [(Open end. Probe.)]

ANALYSIS:

  1. This is also a well-formed question, given that interviewers do not direct responses during probing.

Item 4.

TEXT: Generally speaking, do you think that Boulder County government officials should be doing more to manage growth, or do you think that they should be doing less to manage growth, or do you think their present level of activity is about right?

    1. More

    2. Less

    3. About right

    4. No response

ANALYSIS:

  1. This question is generally a good question, although additional response categories would yield more detailed data. Thus, "much more," "a little more," about the same," "a little less," and "much less" would offer respondents more choices and would likely yield a better indication of public opinion. The use of response options such as "much more" and "a little more" (or "somewhat more") is fairly standard in survey research.

Item 5.

TEXT: Now I will ask you several other questions having to do specifically with the mountainous areas of Boulder County.

Do you think that Boulder County government officials should be doing more to manage growth in the mountains, or do you think that they should be doing less to manage growth in the mountains, or do you think their present level of activity is about right?

    1. More

    2. Less

    3. About right

    4. No response

ANALYSIS:

  1. As with Item 4, this question is generally a good question, although additional response categories would yield more detailed data. Thus, "much more," "a little more," about the same," "a little less," and "much less" would offer respondents more choices and would likely yield a better indication of public opinion.

Items 6 through 11.

TEXT: People sometimes agree or disagree on what the objectives of Boulder County government growth policies in the mountains should be. I will read you a list of some of those possible objectives, and ask you to tell me how you feel about each of them. Do you agree or disagree with using the following growth management objectives? [(Rotate questions 6 through 11. Repeat the Introductory statement as necessary.)]

6. To reduce impacts on wildlife.

    1. Agree

    2. Disagree

    3. No Response

7. To limit the visual impact on ridge lines or mountain backdrops.

    1. Agree

    2. Disagree

    3. No Response

8. To create diversity in the size of new houses.

    1. Agree

    2. Disagree

    3. No Response

9. To keep development away from wetlands, flood plains and other areas near streams.

    1. Agree

    2. Disagree

    3. No Response

10. To limit development on steep slopes.

    1. Agree

    2. Disagree

    3. No Response

11. To limit mountain scarring caused by constructing roads to new houses.

    1. Agree

    2. Disagree

    3. No Response

ANALYSIS:

  1. The introductory paragraph is well-formed.

  2. The possible objectives asked about are well-constructed, with the exception of item 11. The word "scarring" in item 11 is value-laden (in a clearly negative way) and therefore introduces bias to the question. It is unlikely that most people would agree with any statement that asked them to support "scarring," but the definition of a particular impact as a "scar" is purely subjective and variable. If two people were looking at the same view, one might define it as a scar, and the other might not. As examples of more value-neutral wordings for this item, it could have been worded: "To limit the impact caused by constructing roads to new houses," or "To limit the visual impact caused by constructing roads to new houses."

  3. Regarding item 11, there is no clarification that the respondent is being asked to comment on temporary effects to the environment versus longer-lasting effects. Some respondents may answer the question with a construction zone in mind that they would expect would be temporary, while other respondents may answer the question thinking only of permanent or long-lasting effects. This ambiguity leads to threats to both reliability and validity.

  4. The intention behind rotating questions 6 through 11 was likely to minimize bias in the form of "contamination" of an item by a previous one, at least in any consistent manner. However, due to the value-laden term "scarring" used in item 11, bias may have been introduced to any items that followed it. Given the existing wording of items 6 through 11, leaving item 11 as the last item would have minimized contamination of the items that followed.

  5. Adding the word "generally" before the options of "agree" or "disagree" in the introductory paragraph would have helped respondents choose between only two possible answers. Additional response options would have made it even easier for respondents to answer this series of questions, and is routinely done in survey research. Thus, "strongly agree," "somewhat agree," "somewhat disagree," and "strongly disagree" would have yielded data more informative of the population’s views.

Item 12.

TEXT: Do you think Boulder County should spend additional money to purchase open space even if it means a tax increase?

    1. Yes

    2. No

    3. Don’t know/NR

ANALYSIS:

  1. A less biased form of this question would be: "Generally, do you think Boulder County should spend less or more money to purchase open space even if it means a tax increase?" (with response options being "more," or "less," don’t know, or refuse to answer). As it is worded now, there is an increased likelihood that respondents may answer in the affirmative compared to the alternative wording I proposed.

  2. The word "even" is unnecessary and leads the respondent to disapprove of what follows it. In survey research, the respondent should never be directed toward any particular position or response.

Item 13.

TEXT: Boulder County’s current one-forth of a cent open space sales tax expires in December 2008. If you knew that Boulder County could continue buying more open space if the tax were extended at its current rate for 10 years, would you generally support or oppose a 10 year extension of the current tax?

    1. Support

    2. Oppose

    3. Don’t know/NR

ANALYSIS:

a. This is a well-formed question. Additional response categories would offer more flexibility in response and richer data ("strongly support," "somewhat support," "somewhat oppose," "strongly oppose").

Item 14.

TEXT: What about extending the current open space sales tax, which would expire in 2008, for another 10 years, and issuing bonds to allow open space purchases much sooner than otherwise would be the case? Would you generally support or oppose such a proposal?

    1. Support

    2. Oppose

    3. Don’t know/NR

ANALYSIS:

a. This is a well-formed question. Additional response categories would offer more flexibility in response and richer data ("strongly support," "somewhat support," "somewhat oppose," "strongly oppose").

Items 15 through 20.

TEXT: A few minutes ago I asked a short series of questions about some things that Boulder County government could do to achieve particular land use objectives. Now I will ask you some similar questions, except they will be asked from the viewpoint of landowners’ rights as you see them. Please tell me if you feel that a landowner should or should not be able to build a house where they want even if the site selected resulted in the following outcomes:

15. The house would be located in significant wildlife habitat.

1. Should be able
2. Should not be able
3. Other
4. No response

16. The house would be located on a ridge line or would impact a mountain backdrop.

1. Should be able
2. Should not be able
3. Other
4. No response

17. People in the neighborhood believe that the size or appearance of the house would not be compatible with nearby houses.

1. Should be able
2. Should not be able
3. Other
4. No response

18. The access road to the house would create a significant new scar across a mountainside.

1. Should be able
2. Should not be able
3. Other
4. No response

19. The house location creates a new wildfire hazard.

1. Should be able
2. Should not be able
3. Other
4. No response

20. People in the neighborhood are concerned about the impact of the new house on their own property or their own view.

1. Should be able
2. Should not be able
3. Other
4. No response

ANALYSIS (items 15 through 20):

  1. The introductory paragraph asks respondents to answer questions 15 through 20 "…from the viewpoint of landowners’ rights…" This is a clear introduction of bias in all of the following questions. The respondent could have been asked to give their (own) opinions, but were rather prompted by the interviewer to answer the questions from a particular point of view, which may or may not have been their own. Therefore, the data from these items about land use objectives (items 15 through 20) are not valid if the intention is to determine the respondents’ opinions. It cannot be known if respondents gave answers that they thought landowners would give, or their own.

  2. The introductory paragraph also introduces bias in the use of the word "even" in the last sentence of the introduction, which the interviewer was instructed to repeat to the respondent as necessary throughout this series of questions. This word is not at all necessary to communicate the meaning of the sentence, and it implies that the following statements are all "bad" things. This would have the effect of directing the respondent to disapprove of whatever follows.

  3. Items 15 and 18 use the highly subjective and seemingly unnecessary term "significant," thus introducing (or enhancing) bias in the questions. When such a term is used, it must be defined for the respondent so that, in effect, the same question is being asked of all respondents regardless of their variable interpretations.

  4. Item 18 uses the highly value-laden term "scar." As with item 11, this is likely to lead respondents to a disapproving attitude, even if they might not define a specific effect as a scar without prompting.

Item 21.

TEXT: The US Bureau of Land Management, sometimes called BLM, currently is proposing to dispose of roughly 2,000 acres of land in the mountains between Boulder and the Peak-to-Peak Highway. Some people would like to see BLM make the land available for new development purposes. Others would like Boulder County to acquire the land from BLM so that new development does not occur.

21. Would you prefer that the BLM make the 2,000 acres available for development, or would you prefer that Boulder County acquire it so that development does not occur?

    1. Open to development

    2. Boulder County acquire

    3. No response

ANALYSIS:

  1. This is generally a well-formed question, although it might have been appropriate to add an "other" response option in the event that other alternatives were available in the opinions of respondents.

Item 22 (asked only in mountain precincts).

TEXT: Some mountain residents believe that we should do more to limit the number of new houses built in the mountains. Others believe that it is not county government’s place to set such limits. Which most closely represents your views? [(Repeat question if necessary)]

    1. Do more to limit

    2. Not county’s place

    3. Other

    4. No response

ANALYSIS:

  1. The use of the word "we" is unclear. Some respondents may interpret that word as referring to them as mountain residents or citizens, while others may interpret that word as referring to "county government" (which could also be more clearly defined). Thus, this questions suffers from a lack of validity. Further, if a respondent asks for the question to be reread by the interviewer, he or she may interpret the question differently the second time it is heard (after interpreting the "we" from the first sentence in the context of the second sentence), thus threatening reliability.

  2. The phrase "not county government’s place" is unnecessarily biased. I see no reason that a more balanced choice shouldn’t be offered to respondents. An example of more appropriate wording would be (assuming that the "we" is supposed to refer to county government): "Some mountain residents believe that county government should do more to limit the number of new houses built in the mountains, while others believe that county government should do less to limit the number of new houses built in the mountains. Others believe that county government should continue doing what they are now" (with response options such as "more," "less," "the same," don’t know, and refuse to answer).

Item 23 (asked only in mountain precincts).

TEXT: One idea to reduce the number of new houses built in the mountains is creation of a local improvement district in the mountains to purchase some of the potential building sites and leave them undeveloped. A special property tax within the district would provide much of the necessary funding. Using a $200,000 home as an example of how it could work, about 200 undeveloped lots could be purchased if $50 in special property taxes were collected by the special district each year for 15 years, with Boulder County supplying some funds to supplement the improvement district’s contribution.

23. If there was a question on next November’s ballot to create a mountain local improvement district to levy property taxes to purchase undeveloped lots as just described, do you feel that you would support or oppose such a measure if the tax on a $200,000 house would be $50 per year for 15 years?

    1. Support

    2. Oppose

    3. No response

[(If response to q. 23 was other than "support," SKIP to q. 27)]

ANALYSIS:

  1. I expect that item 23 should only be asked of respondents who feel that the number of new houses built in the mountains should be reduced. If respondents do not feel that the number of new houses built in the mountains should be reduced, this question seems irrelevant to them, as they would not be likely to support a tax to achieve an end they disapprove of.

  2. This question asks for an opinion on a possible ballot issue after attempting to describe how a proposed mountain local improvement district might work, and what impact it would have. However, this issue does not seem to me to be presented in a complete and impartial way. The question seems to be intended to elicit a favorable response, rather than a value-neutral probe of respondents’ assessments of legislation that I imagine would have various implications and impacts not presented. If the issue is not familiar enough to a respondent that they have an opinion on the matter without a detailed explanation of the issue, the answer should be "don’t know" after a brief question as to whether the respondent would vote for or against the ballot item. Alternatively, the issue could be presented in a more balanced way, indicating what supporters say (with text approved by supporters) and what opponents say (with text approved by opponents).

  3. It is unclear to me why this question is asked only of respondents in mountain precincts, although I imagine that the questionnaire authors had some purpose in mind. As the introductory paragraph explains, Boulder County tax funds (apparently collected from taxpayers not living in the mountains as well as from those living in mountain precincts) would be provided for the proposed improvement district. As this question is not asked of all taxpayers or voters, I assume that its intention is not to assess public opinion of all affected parties on the proposed legislation. If this question is to affect policymaking, I would expect that all relevant parties would be polled.

Item 24 (asked only in mountain precincts).

TEXT: Still using a $200,000 house as an example, do you feel that you would support or oppose such a measure if the special property tax would be $100 per year and about 400 undeveloped lots would purchased [sic]?

    1. Support

    2. Oppose

    3. No response

[(If response to q. 24 was other than "support," SKIP to q. 27)]

ANALYSIS:

  1. As with item 23, it seems that item 24 should only be asked of respondents who feel that the number of new houses built in the mountains should be reduced.

  2. As with item 23, the issue does not seem to be presented in a complete way. Either the respondent already knows about this option and has an opinion, or he or she is being offered a limited presentation of an issue that is in actuality likely to be very complicated and with differing points of view, and then is asked to offer an opinion based on this limited presentation.

  3. I am not privy at this time to the intentions of those who constructed or commissioned these questions (items 24 through 26), and it is unclear to me what the intention of these follow-up questions are. I note that they are asked only of those who indicated that they supported the proposition as presented in item 23. It seems to me that questions 24, 25, and 26 are designed to assess either how much of a property tax increase supporters will tolerate before withdrawing support, or how much undeveloped land they desire the proposed local improvement district to acquire. Either way, it could not be known from the data what is actually being measured, and thus the data from these questions suffer from threats to validity.

Item 25 (asked only in mountain precincts).

TEXT: What if the special property tax would be $150 per year and about 600 undeveloped lots would be purchased?

    1. Support

    2. Oppose

    3. No response

[(If response to q. 25 was other than "support," SKIP to q. 27)]

ANALYSIS:

  1. See comments regarding item 24.

Item 26 (asked only in mountain precincts).

TEXT: What if the special property tax would be $200 per year and about 800 undeveloped lots would be purchased?

    1. Support

    2. Oppose

    3. No response

ANALYSIS:

  1. See comments regarding item 24.

  2. I note that this series of questions (items 23 through 26) seems to attempt to assess mountain precinct residents’ positions regarding a single potential ballot issue. This may have been the sole intention of those who designed and commissioned this study, and this singular inquiry may provide all the information desired by them. It is not unheard of to conduct very brief surveys that only ask about one aspect of an issue, but as this survey was commissioned by a government entity and not a special interest group, I would have expected that various options to dealing with this issue would have been presented and asked about in order to determine the public’s policy preferences. At the very least, I would think that policymakers would be interested in an open-ended question asking respondents about what they would like to occur in relation to this issue.

Item 27.

TEXT: How long have you lived in Boulder County?

[(Read brackets if necessary)]

    1. Less than 1 year

    2. 1 to 4 years

    3. 5 to 9 years

    4. 10 to 19 years

    5. 20 years or more

    6. No response

ANALYSIS:

  1. This is a fairly standard question and is fine as it is. I assume that interviewers are trained as to how to deal with responses that indicate discontinuous residence (such as someone who lived in Boulder for 3 years in the 1980s, and then moved back into Boulder last year).

Item 28.

TEXT: How old are you?

[(Read brackets if necessary)]

    1. 18 to 24

    2. 25 to 34

    3. 35 to 44

    4. 45 to 54

    5. 55 to 64

    6. 65 or older

    7. Refused/no response

ANALYSIS:

  1. Age is also a standard question, although asking date of birth is usually the preferred method as it allows for calculation of the exact age of respondents at any point in time, not just at the time of the survey. However, if this is a one-time poll and specific ages and averages are not important, these response categories would work.

Item 29.

TEXT: Do you own your home or do you rent it?

    1. Own

    2. Rent

    3. No response

ANALYSIS:

  1. The response options in this item are poorly constructed. They are not exhaustive, in other words, they do not account for all possible responses (for example, living with family at no cost).

  2. The use of the word "home" may imply to some respondents a house, as opposed to an apartment. The word "residence" or the phrase "where you live now" (in a slightly modified question form) would minimize this implication. Further, the use of the word "your" may imply to some respondents that they are expected to be the primary occupant (this may be an issue if the respondent is living in what they consider to be primarily someone else’s residence). To avoid these potential problems, wording could be offered such as: "Where you live now [or, alternatively, "Where you have lived for more than six months in the past year), do you rent or own?" (with the interviewer having an option for "other" responses that the respondent may volunteer).

  3. Note that if this poll was conducted during the Summer in a college town, the answers might be very different than during the school year. It is unclear as to how a student who normally lives elsewhere might answer this question in the absence of a response category or clarification to account for such possibilities. Further, there is no question confirming that the respondent lives the majority of the year at the residence in Boulder County (or in the mountains). Thus, if the questionnaire was administered during the Summer, there is no way to distinguish Summer residents from year-round residents. These matters threaten the reliability of the measure.

Item 30.

TEXT: How much formal education have you completed?

    1. High school or equivalent

    2. College degree

    3. Graduate studies

    4. Other___________

    5. No response

ANALYSIS:

  1. It is unclear to me why these response categories were used, but I assume that educational attainment is not an important variable in this project. Normally, more complete response options are used. For example, I note that less than high school or equivalent is not specifically coded for (for example, "elementary school graduate," "some high school (no degree)"), and that there is no distinction made between those who have attained a Ph.D. and those who have taken one graduate class for no credit without being admitted to a graduate program. There are standard and widely used response options for educational attainment that would normally be used, such as those used by the U.S. Census.

 

NOTE TO INTERVIEWER:

[(Thank respondent and then complete questions 31 through 35 from your calling sheet information. Double check to make certain that you got all of them.)]

Item 31.

TEXT: Sex:

    1. Male

    2. Female

ANALYSIS:

  1. This is a pretty standard item, although at the Center for Survey Research, we have a note to the interviewer to ask this if necessary, as is occasionally, albeit rarely, the case. In this study, the interviewer has already hung up the phone before coding this item.

Item 32.

TEXT: Party affiliation:

    1. Republican

    2. Democrat

    3. Independent/Unaffiliated

ANALYSIS:

  1. Since the interviewer has already hung up the phone, I assume that this information is taken from official voting registration records. While this is informative of the affiliation a respondent claimed the last time they registered to vote, it most definitely does not represent the way they see themselves at the time of the interview. It may not even represent how they saw themselves at the time they registered. For example, many people register an affiliation with a political party just so that they can vote in a primary election. While I would not eliminate this question, if party identification was important at all (which I assume it is, since this question was included), I would certainly add another question that was asked of the respondent while he or she was still on the phone about current party identification.

Item 33.

TEXT: Precinct Number: ___________

ANALYSIS:

  1. A respondent may be living in a precinct other than the one on record, or may live more than half the year in another precinct. For example, a student may live elsewhere, but be answering this survey while staying at their parents’ house for the Summer, or this may be a Summer or vacation residence. Also, a respondent may have recently moved. There is no item in this instrument to confirm current or primary address (and therefore precinct of primary residence).

Item 34.

TEXT: City Code (Circle "00" if there is no code)

ANALYSIS;

  1. See comment regarding item 33.

  2. Note that I have omitted the response categories for this item in my copy of the text. They consist of a series of numbers only.

Item 35.

TEXT: Calling sheet number: ______________

ANALYSIS:

  1. This is apparently a coding not relevant to my analysis.

III. SUMMARY AND CONCLUSIONS

Survey research (or polling) is susceptible to error and bias, as is any research method. Great care must be taken in the construction of questions and responses presented to respondents, and in the way that the questions are asked by interviewers, in order that data accurately represent the population being studied and yield the information desired. Survey researchers have developed many techniques to ensure these goals, and are continually improving upon these techniques as well as developing new ones. The promise that quality survey research holds for informing us of the state of our social world is great, and the information gathered can lead to significant improvements in the ability of citizens and public policymakers to determine needed change (or constancy) and develop more informed courses of action. Unfortunately, survey research can also be done poorly, leading to misinformed courses of action. While I have no interest in affecting any particular political course of action in relation to the issues that the Boulder County Questionnaire is intended to inform, as a professional researcher, I do have an interest in ensuring that survey research is done properly and used appropriately. My evaluation of the Boulder County Questionnaire leads me to conclude that the instrument as I understand that it was recently administered to Boulder County residents suffers from some serious threats to reliability and validity, and possesses unnecessary biases that would taint any findings. It concerns me that data yielded from this poll may be used to determine public policy or affect public opinion.

Reliability is the term that survey researchers use to refer to our confidence that the same information would be collected in repeated observations of the same event—for example, in the repeated administration of the same question. There are always threats to reliability in survey research, such as those introduced by the effects of context, including the salience of community issues at the time of the interview, and the time of year. This needs to be taken into account whenever interpreting survey data. Further, I have noted some specific threats to reliability that are embodied in the Boulder County Questionnaire. It is also the case that the questions that the interviewer repeats, as well as the words stressed by the interviewer (which are unknown to me as I did not supervise or monitor administration of this poll) may seriously affect the reliability of the data gathered. While I am able to note some inherent threats to reliability in this questionnaire, only the entity that conducted the poll could have ensured the elimination of other possible threats that may be introduced by interviewers. Having no familiarity whatsoever with the organization contracted by Boulder County to administer this questionnaire, or with the manner in which they train and monitor their interviewers, I have no specific reason to believe that the poll was conducted in any way that would introduce additional threats to reliability. However, given that the design of the questionnaire and question construction is not of the highest standards possible (as evidenced in comments in Section II of this report), I would hope that those who commissioned this poll, and those who would seek to interpret its findings, would take steps to ensure that reliability was ensured during administration of this questionnaire.

Validity is the term that survey researchers use to refer to the degree that a measure (such as a survey question) accurately reflects the concept that it is intended to measure. In my professional opinion, the Boulder County Questionnaire suffers from some serious threats to validity, as noted throughout my item-by-item analysis. I imagine that those who commissioned this study wanted an accurate picture of the views of Boulder County residents on important community issues, but the data may end up reflecting what some biased wording led respondents to say as much or more than what people actually believe and feel. There is no way to eliminate these threats to validity from the data already collected. The only way to ensure valid data would be to modify the questionnaire and conduct another poll using the highest of professional standards.

While survey research done properly is a very valuable tool, when it is done poorly it can be a dangerous thing, leading us to draw conclusions based on fiction veiled as "scientific" fact. Science can be used for the common good, and it can be misused to achieve political or ideological ends. The misuse of polling and "statistics" (as poll results are often called) has led to public skepticism regarding the value of survey research. The possible misuses of the data collected via the Boulder County Questionnaire are far too numerous for me to detail here. While I am a professional survey researcher myself and believe in the promise of polling, I would have to count myself as among those who are appropriately skeptical of any data resulting from the Boulder County Questionnaire in its current form.

Home | Newsletters | Know Your Rights | About Us | Links/Referrals | Membership | Contact
© 2008 Land Use Coalition - www.landusecoalition.org - E-mail: contactus@landusecoalition.org
PO Box 20265 - Boulder, CO 80304 - Voice Mail: 303.666.7903