I. Introduction

A. We must identify the population we want to study, then figure out how to select a representative sample.

B. Usually we use a random sample, but sometimes that is not desirable.

1. We may be interested in different segments of the population and want to guarantee that we have a sufficient sample size from each segment.

2. We may want to study certain segments more intensively than others, requiring a larger sample from those segments.

C. The major issue is not the overall sample size. It is the sample size within each cell.

1. Example: my wine study (n=809)

2. Hand out and discuss

D. Ideally we would like to survey the entire population, thus eliminating many of the statistical problems.

1. This tends to be costly: survey administration, data processing, and so on.

2. Occasionally you may be able to actually use an entire population in which case your work will be much easier.

3. It's tempting to define the population narrowly to avoid this problem, but usually this costs you external validity. As you narrow the definition of your population, you are less able to generalize your results to the rest of the world.

E. As Black notes (p. 111) "A relatively small sample carefully selected may provide a more valid result than a large sample poorly chosen.

II. Identifying the population(s)

A. We call it a population because we're usually studying people.

B. As noted earlier, the definition of the population is an important factor in determining external validity.

C. Many research questions ask about the effects of various aspects of life on people's behavior and attitudes. Life is the experimenter, the life experience is the treatment and the researcher is only an observer.

D. The population definition and research question are closely intertwined.

1. If your research will require a lengthy questionnaire, you may find you can't persuade very many people to answer the survey, in which case your population will be limited.

2. Narrowing the research question to produce a shorter questionnaire will allow you to expand the population.

E. Always remember that theory defines causation while data can only tell us about correlation.

1. While it's rare in economics, you should at least think about using a control group for comparison.

2. Usually, however, we can't use that sort of experiment.

III. Sampling

A. Two important consequences of using samples from a population.

1. Since we want external validity to our study, we must do our best to make sure the sample represents the population.

2. Do your best to obtain a sample that won't introduce new potentially confounding variables.

B. What does the "random" in the phrase "random sampling" mean?

1. Might use a random number table starting at a randomly chosen location.

2. Might use a RAND() function in a computer, preferably seeded with the date and time (discuss the idea of the Julian date and how it might be used).

3. But once a series of numbers has been generated, can it truly be said to be random?

C. Random sampling pulls a portion of the population such that all possible samples of size n have the same probability of being selected.

IV. Common Sampling Techniques

A. Simple random sample

1. Highly representative if all subjects participate; ideal.

2. Not possible without complete list of population members.

B. Stratified random sample

1. Random sample from identifiable groups (strata), subgroups, etc.

2. Can ensure that specific groups are represented, even proportionally, in the sample.

3. Strata must be carefully defined.

C. Cluster sample

1. Random samples of successive clusters of subjects (e.g., by institution) until small groups are chosen as units.

2. May be useful when no population list exists, but local lists do exist.

3. Clusters must be equivalent (somehow) and some natural clusters are not grouped by essential characteristics. For example, a geographic cluster may yield equal sample sizes, but unemployment rates in different areas will most likely not be the same.

D. Stage sample

1. Combines cluster and random/stratified sampling.

2. Can make up a probability sample by randomizing in stages and within groups. Possible to select nearly random sample when population lists are very localized.

3. Very complex, combines limitations of cluster and stratified sampling (if random not used).

E. Purposive sample (focus groups)

1. Hand selected subjects on the basis of some specified characteristic.

2. Ensures balance of group sizes.

3. Samples not easily defended; external validity likely to be compromised.

F. Quota sample

1. Select individuals as they arrive to fill a quota by characteristics proportional to populations.

2. Ensures adequate sample sizes within each group.

3. Not possible to prove the sample is representative of the population.

G. Snowball sample

1. Subjects with desired characteristics give names of other appropriate subjects.

2. Possible to include members of groups where no lists even exist (e.g., drug abusers, criminals).

3. No way of knowing whether the sample is representative of the population.

H. Volunteer, accidental, convenience sample

1. Major advantage is low cost.

2. Can be highly unrepresentative.

3. Example: wine survey.

V. Sampling Errors

A. Sampling errors occur simply because data are being collected on a sample instead of the entire population.

B. First four sampling techniques were devised to increase the likelihood that any sample will be representative of the entire population.

C. Statistical accuracy when the population mean and standard deviation are known.

1. The natural variability in a set of sample means (from successive samples) is measured as the standard error of the mean. It is estimated from the population standard deviation and the sample size as .

2. From this we can derive our first equation that gives us an inkling about what size sample we need: .

3. Clearly as n gets very large the standard error of the mean will approach the actual mean so will approach zero. Looking at this the other way, the closer we want to be to the actual mean, the larger n must be.

4. We often calculate the z-score

5. See fig. 5.3. Discuss meaning of 95% confidence interval and one vs. two tailed tests.

D. Statistical accuracy when population mean and/or standard deviation are not known.

1. We can estimate the standard error of the mean using sample statistics where s_A is the standard deviation of the sample group A and n_A is the sample size of sample group A.

2. Strictly speaking this only applies to purely random sampling.

3. Using this equation we can establish a confidence interval. For example, a 95% confidence interval would be . Explain why it's 1.96 with reference to figs 5.3 and 5.4.

4. See example, p. 129

E. The chi-square test

1. Used to test how likely it is that a sample represents the population. Only useful when you know the population frequencies for each category.

2.

3. See example p. 131 and note that the degrees of freedom are usually m-1 not m.

F. Sampling errors are unavoidable, but can be reduced by sound sampling procedures. Non-sampling errors can be caused by incorrect sampling frame (population is voters, sample drawn from phone book), poor measuring instruments, incorrect data processing and non-response by subjects in the sample.

VI. Avoiding Subject Loss

A. Is your sample a true random sample or simply volunteers?

1. Who responded and who failed to respond? How do you measure statistics from the group of non-respondents? (Example: wine)

2. Even the time you query subjects may cause a bias, e.g., calling during dinner.

3. Must convince subjects that it is worthwhile to participate. (wine: offered them copy of the finished paper) Must assure them of confidentiality, then keep your word. Must be careful in your analysis.

B. Contending with non-responses

1. Must discuss who responded and who didn't.

2. Try to provide evidence that any non-response has nothing to do with the research instrument.

3. Think about ways in which your study may discourage participation before you start sending out surveys.

4. Question: how do you follow up to non-responders and still maintain confidentiality? How do you know who answered and who didn't? What does "confidentiality" mean?

5. Can send follow-up questionnaire.

6. Structure questionnaires so that optionally answered questions don't mean data loss. Figure out which questions are more important and which are less, then (perhaps) label the less important questions optional.

VII. Sampling Techniques and Sample Size

A. Fraction of population sampled is less important than n.

B. We want to test whether characteristics of our sample are close to those of the population. We can use parametric tests (t-tests, analysis of variance), chi-square tests and so on.

C. Remember, the prime concern is to have a reasonable n in each cell, not just a large overall n. See p. 136.

D. Nan: n=50. Black: n=30.

E. Non-responses become very important when cell sizes are small. You may need to sample with replacement.

F. Random selection and assignment are both important.

VIII. Ethics

A. Be cautious. When in doubt ask. Talk to other researchers.

B. DO NOT VIOLATE PROMISES OF CONFIDENTIALITY.

IX. Technical notes on sample distributions and statistical tests (Black, ch. 13)

A. At what point do ordinal data become interval discrete data?

1. Need between 10 and 20 intervals, implying a sample size of 30 if the shape of the probability distribution is to be decipherable.

2. Do not assume the data are normally distributed. Test for skew and kurtosis (at the minimum).

3. Is IQ normally distributed or are IQ test results only normally distributed? Don't confuse the instrument with the "thing" being measured.

B. Collect this data: height of each person in class. Plot on a histogram. Are they normally distributed?

C. Recall the z-score: where x_i is observation I, is the mean of the distribution of observations and S is the standard deviation and we use n-1 because we lost one df calculating the mean. Alternatively, you can use the STDEV( ) function in Excel.

D. Why the t-test is so popular among economists

1. We rarely know the population standard deviation. Therefore, we must calculate the sample standard error of the mean using sample statistics. See V.D.1 above for details.

2. However, we can no longer use the z-score but instead use . The usual rule of thumb is that t scores greater than 2.0 are good at the 95% level. However, cowards can look it up in the table.

E. Testing for normality.

1. . where M is the median. A normal distribution has a skewness of 0.0.

2. Alternatively, we can use the Glass-Hopkins measure

X. Technical note on sample size (Black, ch. 14)

A. Significance test denoted bya. Table 14.1 shows significance levels and the corresponding z-score. This works pretty well for t-scores too.

B. Note that the level of significance should be chosen in advance.

C. One criterion is that the statistical test should have a high statistical power -- a high probability of correctly rejecting the null hypothesis.

D. We can estimate the sample size necessary to achieve a desired level of power: where z(b) is the z-score for a Type II error (incorrectly classifying the group as belonging to the original population when it did not) and z(a) is the z-score for probability level a from table 14.1.

XI. What is "power?"

A. a is the probability of a Type I error (rejecting Ho when it's true).

B. b is the probability of a Type II error (accepting Ho when it's false).

C. The power of a test is the probability of rejecting Ho when it is false = 1-b.

1. Suppose the sample actually belongs to a different population altogether. That implies the sample mean really belongs to a different distribution of sampling means for a population whose mean we can only estimate by assuming that it is equal to the sample mean . To determine a value for b necessitates finding the overlap of these two sampling distributions.

2. The sampling distribution of means is normally distributed around . Therefore, it's often true that b=0.5 and the power is also 0.5.

3. Do the example in Black, pp. 381-382.

4. We can calculate where m₀ is the mean for the original known population and s_x0 is the standard error of the mean for the original population.

5. With x(a) the process is simply reversed for the non-central distribution where m1 is the mean of the alternative population (equals the sample mean and s_x1 is the standard error of the mean for the alternative population (the standard error of the mean for the sample).

6. Solving for

7. If we accept that the two standard errors of the mean are equal and that the best estimate of the mean for the non-central distribution is the sample mean () then we can simplify the above equation to

XII. Estimation (Gujarati, ch. 7 plus OLS without inverting a matrix)