Read Homogeneous Samples in Cross-National Research text version


EBMS Working Paper EBMS/2000/11 (ISSN: 1470-2398)

Homogeneous Samples in Cross- National Research

Dr Nina Reynolds Professor Antonis C. Simintiras and Adamantios D. Diamantopoulos

European Business Management School Singleton Park Swansea SA2 8PP UK Tel: 01792 295601 (International: +44 1792 295601) Fax: 01792 295626 (International: +44 1792 295626) e-mail: [email protected]

Homogeneous Samples in Cross-National Research Abstract Homogeneous samples are often used in cross-national research, yet somehow, the impact of these samples on the validity of cross-national research projects has not generally been evaluated. This paper examines sampling and its relationship to the validity of the different `types' of research. Next, it discusses cross-national sampling issues, in particular, an often inherent trade-off - representativeness versus comparability. Then, by combining crossnational sampling issues with the different research `types,' this paper assesses the extent to which the validity of cross-national research may be affected by the use of homogeneous samples. Keywords: Sampling, Cross-national research methodology, Validity 1: Introduction Validity is linked with sampling (Calder et al 1981; Lynch 1982; McGrath & Brinberg 1983). A convincing sampling strategy, and the implementation of that strategy, is, therefore, crucial to ensuring valid research results. Indeed, "one's ability to make inference[s] from subset to larger group depends on the method by which the sample of elements was chosen" (Churchill 1999, p.496). The soundest sampling strategy is generally believed to be the collection of a probability1 sample from some well defined population (Sudman & Blair 1999). In contrast, non-probability2 samples are often considered dubious and believed to imperil the validity of the research results. A non-probability sampling, for instance, was blamed for Gallup's incorrect prediction that Thomas Dewey would beat Harry Truman in the 1948 US presidential elections. One of the problems with non-probability samples is that one cannot judge whether they represent the population of interest. Non-probability samples may favour the selection of a particular `type' of sampling unit and/or lead to unintentionally homogeneous samples. In contrast, probability samples are expected to reflect all the differences that are present in the population from which they are drawn; that is, probability samples tend to be as heterogeneous as the population they represent. However, "a significant development in recent decades has been reduced interest in sampling the general population of consumer and increased interest in sampling specific groups" (Sudman & Blair 1999, p.272). These groups tend to be more homogeneous than the general population and, as such, probability sampling may result in fairly homogeneous samples. This trend restricts the type of generalization that can be made from probability samples as "almost all ... consumer research can concern itself only with `generalizing across subpopulations.' [These] subpopulations are defined in terms of independent (background) variables that might affect the behavior of interest" (Lynch 1982, p.229). While probability sampling may, overall, give the appearance of greater externally validity3 than non-probability samples (Malhotra 1993), this impression is inaccurate. External validity can, in actuality, only be established after "the results found [have been generalized] to other populations, settings, and so forth" (Winer 1999, p.349). Consequently, the external validity of any research project can only be established when other research projects have confirmed the results (Calder et al. 1982; Lynch 1982). External validity is not directly related to whether a probability or nonprobability sample is used. While the external validity of a research project is important as it allows the research results to be generalised, it is not the only type of validity that needs to be established. Before external validity can be addressed, the internal validity of the research needs to be determined; that is, the researcher needs to consider "the extent to which competing

explanations for the results are avoided" (Aaker et al. 1998, p.360). If internal validity has not been established, then the value of considering external validity is questionable. Consequently, enhancing external validity by using methods that had an adverse impact on internal validity would be methodologically dubious (Lynch 1982). Hence, when considering the impact of sampling on validity, researchers need to consider both internal and external validity. Overall, the relative weight placed on achieving each is dependent on the objectives of the study. For instance, if the research objective was to determine the decision making process for low involvement products and the researcher established the decision making process for purchasing a car, then while the research could be internally valid (consumers purchasing a car go through steps A to G), it could not be externally valid as the decision making process for low involvement goods (as a car is not generally considered a low involvement good). In this case, even though internal validity was achieved, it did not contribute to solving the research problem. When considering the impact of sampling on both internal and external validity, it is useful to understand the differences between the generic research types and how the cross-national environment has an impact on sampling methodology. In the following two sections sampling issues concerned with the `type' of research being conducted and with cross-national research are considered separately. The final section of the paper will integrate these two areas. 2: Research Classification Schemes - Sampling Issues Research classifications consider how different research projects can be grouped into generic `types' of research. Two main classification schemes, one advocated by Calder et al. (1981), the other by McGrath and Brinberg (1983), are considered. The former scheme classifies research by considering the overall research objectives and divides research into effects application research and theory application research. The latter classification scheme looks more closely at the building blocks of the research (conceptual, methodological and substantive domains), and divides research into three generic `paths' - experimental, theoretical and empirical. Figure 1 defines the terms used by Calder et al. (1981) and McGrath and Brinberg (1983) in their classification schemes. Figure 1: Terminology Used By The Research Classification Schemes

Calder et al.'s `Research Types' · Effects application is designed to obtain findings that can be generalized directly to a real-world situation of interest. · Theory application is research designed to obtain scientific theory that can be generalized through the design of theory-based interventions that are viable in the real world. McGrath & Bringberg's `Research Domains & Research Paths' · The conceptual domain contains elements that are concepts, and relations between elements that are conceptual models about patterns of concepts · The methodological domain contains elements that are methods (or instruments or techniques) for making observations or manipulating variables, and relations that are structures or comparison models for comparing (i.e., for exploring covariation and differences in) sets of observations. · The substantive domain contains elements that are events (behaviours in temporal/ situational/ spatial contexts) and relations that are phenomena (patters of relations among events). · The experimental path (i) combines the conceptual and methodological domains to give a `design' (ii) involves implementing that design using data from the substantive domain. · The theoretical path (i) combines the conceptual and substantive domains to give a `set of hypotheses' (ii) involves testing the theory (set of hypotheses) using the techniques from the methodological domain. · The empirical path (i) combines the methodological and substantive domains to give a `set of observations' (ii) involves explaining these observations by using theories from the conceptual domain.

According to Calder et al. (1981, p.197) when the objective of the research is to test theory (i.e., theory application) and the researcher only intends to "use ... scientific theory to explain events beyond the research setting," then homogeneous samples are acceptable if not desirable as they reduce the likelihood of extraneous variables having an impact on the research results (i.e., high internal validity). Theory application does not require a representative sample "because statistical generalization of the findings is not the goal. It is the theory that is applied beyond the research setting ... [Indeed] homogeneous samples are preferred because they typically provide a stronger test of the theory" (Calder et al. 1981, p.200). As such, a homogeneous sample (even a convenience sample such as students) could, for example, be used to determine whether variable C intervened in the relationship between variables A and B. In contrast, when research aims to map "observed data directly into events beyond the research setting" (Calder et al. 1981, p.197 - effects application), then samples should be drawn that are representative of the population of concern4. To determine the number of consumers who will buy product X at price Y, for example, would require a representative sample. Unlike theory application, with effects application "correspondence procedures require that research participants match individuals in the real world setting of interest" (Calder et al. 1981, p.199). In short, Calder et al. (1981; 1982; 1983) maintain that homogeneous samples are suitable when conducting theory application research. In contrast to Calder et al.'s (1981; 1982; 1983) view, Lynch (1982; 1983) holds that researchers will not be able to foresee all the background variables that could have an impact when investigation theory. So although Lynch (1983, p.111) agrees that "one can never guarantee external validity" he holds that "theoretical researchers should concern themselves with variables external to their theories" and use sampling methods, such as deliberately sampling for heterogeneity, that allow extraneous variables to be considered when assessing the theory itself. That is, Lynch (1982; 1983) considers homogeneous samples unsuitable for both effects and theory application research. In an attempt to reconcile these two views, McGrath and Brinberg (1983) use a framework where all research has three stages. The first stage involves "development, clarification, and selection of elements and relations" (McGrath & Brinberg 1983, p.118) from three research `domains' (conceptual, methodological and substantive). In the second stage, individual research projects then prioritise two of the three research domains leading to one of three research `paths'5 (experimental, theoretical or empirical). (Figure 1 provides definitions of the research domains and paths). Each of these paths gives empirical findings that then, in stage three, are assessed for external validity. The further research that assesses for external validity will consider the domain neglected on the original research path. With the theoretical path, for example, external validity would be established by repeating the research using different methods. This classification scheme removes the differences between Calder et al. (1981; 1982; 1983) and Lynch (1982; 1983) concerning the use of homogeneous (even convenience) samples by separating research addressing the conceptual domain into two distinct paths (experimental and theoretical). When Calder et al. (1981) talk about research in the conceptual domain (i.e., theory application), they are considering the experimental path. Lynch (1982; 1983), however, is concerned with the theoretical path. With the former, the substantive domain is neglected and issues of sample representativeness are not critical. With the latter, the substantive domain is critical, thus sample representativeness is a consideration. 3: Cross-National Research - Sampling Issues The different research classifications can be applied to both domestic and international research. In cross-national research, however, additional sampling issues arise. A well-known

methodological issue of non-domestic marketing research is the emic-etic dilemma (Berry 1969). An emic approach "attempts to describe items of behaviour occurring in a particular culture utilising only concepts employed in that culture" (Davidson et al. 1976, p.1). In contrast, an etic approach "is primarily concerned with identifying and assessing universal attitudinal and behavioural concepts and developing pan-cultural or `culture free' measures" (Craig & Douglas 2000, p.154). The emic-etic dilemma is inherent in cross-cultural (and cross-national) research as "to be `cultural' requires the emic viewpoint, and `cross' requires the etic perspective" (Malhotra et al. 1996, p.12). At the sampling stage of the research process, the emic-etic dilemma is reflected in the trade-off between the within country representativeness of each national sample and the between-country comparability of the samples (Craig & Douglas 2000). Comparability in cross-national research is often achieved by selecting matched, homogeneous samples. While respondents in these samples are not representative of their national populations, they can be used to represent the characteristics of their nations (Netemeyer et al. 1991). Matching of cross-national samples helps to ensure that any observed differences between nationalities are due to national differences rather than more basic demographic differences that may exist between the groups; that is, it helps to rule out alternate hypotheses (e.g., Hofstede 1991; van de Vijver & Leung 1997). However, the specific criteria on which samples are matched needs to be theoretically defensible. The cross-national researcher cannot just chose the nearest convenient sample, since "without a defensible sampling strategy, the results of the study may be ambiguous or misleading" (Lonner & Berry 1986, p.85). If matched samples, whether homogeneous or heterogeneous, are used then the reasoning behind the matching variables needs to be explicit, as does the relationship between the matching variables and the research problem being investigated. In this way, even the commonly scorned student samples (e.g., Ferber 1977; Soley & Reid 1983) can be valid if they are theoretically justifiable, for instance, if the problem being studied is applicable to students (Latour et al. 1990). 4: Types Of Research And Sampling In The Cross-National Environment The arguments concerning the use of homogeneous samples with different generic research classifications lead to the conclusion that homogeneous (matched) samples are useful in cross-national research when: i. The type of research being conducted has culture/nationality as a variable of interest; ii. The construct(s) of interest is (are) relevant to the specific homogeneous samples chosen; and, iii. any matching that takes place is done using variables that are theoretically justifiable given (i) and (ii). Looking more closely at the first condition, if the research objective is, for instance, to determine the general level of demand for a particular product in several different countries (empirical path or effects application research), then samples will need to be as representative as possible of the national populations. In contrast, if the researcher is concerned with the cross-national comparability of different data collection methods, then matched homogeneous samples would be suitable as they isolate the variable of interest (nationality) (experimental path). If, on the other hand, the research aims to determine the impact of culture/nationality on a particular theoretical model, or is concerned with exploring the applicability of a model in another cultural/national6 context, it would be reasonable for extraneous (i.e., non-model specific) antecedent factors to be held stable; that is, use extraneous antecedent factors to match the samples. Technically, this would be the theoretical path and require a sample representative of the substantive domain. However, as the substantive area of interest is

culture or nationality, using homogeneous matched samples within each culture/nationality isolates the substantive domain of interest to the researcher. In short, as in domestic research, in cross-national research the type of sample desired is heavily dependent on the questions being asked by the researcher. If the research objective allows for homogeneous sampling (i.e., the experimental path or in cross-national research the theoretical path), then, it becomes important that the construct of interest is relevant to the homogeneous sample chosen (ii above). The researcher needs to select samples for which the construct is relevant. One would not, for example, select schoolchildren if one was concerned with the buying process for high-involvement goods such as cars. Similarly, low-income groups are unlikely to be a suitable for, say, an investigation into the reasons for employing full-time child care. As such, before specific matching variables are selected, the researcher must reflect on the applicability of the research constructs and methods to the sample if they are to be establish internal validity. In essence, a homogeneous sample should not be selected on the basis of its availability/convenience. Once the relevance of the research to a particular homogeneous group has been determined, the specific matching variables that will be used to match the samples need to be decided (iii above). At this point, "one must take a theoretical approach to deciding which background factors warrant attention" (Calder & Tybout 1999, p.360). The reasoning used at this point must look beyond surface-level factors, such as whether respondents are students, if validity is to be established (internal and external). Researchers "should think on the level of constructs when [they] consider what biases might be engendered by relying on a [homogeneous] sample... The critic should specify (a) the construct on which the sample is atypical and (b) the case for why this construct might interact with the experimental treatment manipulations" (Lynch 1999, p.370). In cross-national research criticism of homogeneous samples should also specify how the criteria used to match the samples have a disparate impact on the construct of interest in the particular countries being studied. In conclusion, if the central issue of concern to a cross-national researcher is the differences between cultures, then matching samples can be desirable as this "makes the effect of national differences ... stand out" (high internal validity) (Hofstede 1991, p.13). However, if the aim of a research project is to generalise the results across national populations, then samples that are as representative as possible of the nationalities of concern are desirable (high external validity). Endnotes


Probability sampling: procedures in which each element of the population has an equal and fixed chance (greater than zero) of being selected for the sample (Malhotra 1999; Aaker et al. 1998). 2 Non-probability sampling: procedures where the chance of each element of the population being selected for the sample is unknown (Malhotra 1999; Aaker et al. 1998). 3 External validity deals with the issue of generalizability of the results found to other populations, settings and so forth (Campbell & Stanley 1963). 4 If the population of interest is homogenous for a given research project then a homogenous sample is, again, the most appropriate. 5 Under this classification scheme, each research path initially neglects one of the research domains. So, the experimental path neglects the substantive domain leading to accusations that the material is trivial, the empirical path neglects the conceptual domain and researchers following this path may be accused of `dust bowl empiricism,' and, finally, those taking the theoretical path may appear to be casual about the methods used in their investigations (McGrath & Brinberg 1983). 6 This type of objective is concerned with establishing the external validity of an existing theoretical model; that is, testing the model's applicability in a different substantive domain.

References are available on request

HOMOGENEOUS SAMPLES IN CROSS-NATIONAL RESEARCH by Nina L Reynolds1, Antonis C Simintiras2 & Adamantios D Diamantopoulos3 TRACK: Marketing Models and Marketing Research (1)


Address: Lecturer in Marketing, Department of Management Studies, University of Glasgow, 55-59 Southpark Avenue, Glasgow, G12 8LF, UK. Telephone: +44 (0)141 330 4064. Fax: +44 (0)141 330 5669. E-mail: [email protected] 2 Affiliation: Professor of Marketing, European Business Management School, University of Wales Swansea 3 Affiliation: Chair of Marketing and Business Research, Business School, Loughborough University

REFERENCE LIST FOR HOMOGENEOUS SAMPLES IN CROSS-NATIONAL RESEARCH Aaker, David A, V Kumar & George S Day (1998) Marketing Research, Sixth edition. John Wiley & Sons: New York. Berry, John (1969) `On cross-cultural comparability' International Journal of Psychology 4(2), 119-128. Calder, Bobby J and Alice M Tybout (1999) "A Vision of Theory, Research, and the Future of Business Schools" Journal of the Academy of Marketing Science 27(3), 359-366. Calder, Bobby J, Lynn W Phillips & Alice M Tybout (1981) `Designing Research for Application' Journal of Consumer Research 8(Sept), 197-207. Calder, Bobby J, Lynn W Phillips & Alice M Tybout (1982) `The Concept of External Validity' Journal of Consumer Research 9(Dec), 240-244. Calder, Bobby J, Lynn W Phillips & Alice M Tybout (1983) `Beyond External Validity' Journal of Consumer Research 10(June), 112-114. Churchill, Gilbert A Jr. (1999) Marketing Research: Methodological Foundations, Seventh edition. Forth Worth, TX: The Dryden Press. Craig, C Samuel & Susan P Douglas (2000) International Marketing Research, Second edition. New York: John Wiley & Sons. Davidson, AR, JJ Jaccard, HC Triandis, ML Morales & R Diaz-Guerrero (1976) `Crosscultural model testing toward a solution of etic-emic dilemma' International Journal of Psychology 11, 1-13. Ferber, R (1977) `Research by convenience' Editorial, Journal of Consumer Research, 4 (June), 57-58. Hofstede, G (1991) Cultures and Organizations. Harper Collins: London. LaTour, Michael, Paul J Champagne, G Steven Rhiel & Robert Behling (1990) `Are students a viable source of data for conducting survey research on organizations and their environments?' Review of Business and Economic Research 26(1), 68-82. Lonner, Walter & John Berry (1986) `Sampling and surveying' In Lonner, Walter & John Berry (eds.) Field Methods in Cross-Cultural Research. Sage Publications: Beverly Hills, CA. Lynch, John G Jr. (1982) `On the External Validity of Experiments in Consumer Research' Journal of Consumer Research 9(Dec), 225-239. Lynch, John G Jr. (1983) `The Role of External Validity in Theoretical Research' Journal of Consumer Research 10(June), 109-111. Lynch, John G Jr. (1999) `Theory and External Validity' Journal of the Academy of Marketing Science 27(3), 367-376. Malhotra, NK (1993) Marketing Research: An Applied Orientation. Prentice-Hall: Englewood Cliffs, NJ. Malhotra, Naresh K, James Agarwal & Mark Peterson (1996) `Methodological issues in cross-cultural marketing research - a state-of-the-art review' International Marketing Review 13(5), 7-43. McGrath, Joseph E and David Brinberg (1983) `External Validity and the Research Process: A comment on the Calder/Lynch dialogue' Journal of Consumer Research 10 (June), 115124. Netemeyer, Richard G, Srinivas Durvasula & Donald R Lichtenstein (1991) `A cross-national assessment of the reliability and validity of the CETSCALE' Journal of Marketing Research 28, 320-327. Soley, Lawrence C & Leonard N Reid (1983) `On the validity of students as subjects in advertising experiments' Journal of Advertising Research 23(4), 57-59.

Sudman, Seymour & Edward Blair (1999) "Sampling in the Twenty-First Century" Journal of the Academy of Marketing Science 27(3), 269-277. Van de Vijver, Fons & Kwok Leung (1997) Methods and Data Analysis for Cross-Cultural Research. Sage Publications: London. Winer, Russell S (1999) `Experimentation in the 21st Century: The Importance of External Validity" Journal of the Academy of Marketing Science 27(3), 349-358.


Homogeneous Samples in Cross-National Research

10 pages

Report File (DMCA)

Our content is added by our users. We aim to remove reported files within 1 working day. Please use this link to notify us:

Report this file as copyright or inappropriate


You might also be interested in

Leadership styles and outcome patterns for the nursing workforce and work environment: A systematic review
lena book 1_Raj.indd
Microsoft Word - Content_analysis.doc
Microsoft Word - Final Final Corrected Version PhD NKOANE 2006.doc