Exploring Difficult-to-Research Technical Markets: The Delphi Investigative Paradigm

By Charles H. Ptacek, CMC

Marketing research is the primary tool for both bridging the communication gap with customers and enabling managers to stay in touch with their markets. It involves the specification, gathering, analyzing, and interpretation of information to help management understand the environment, identify problems and opportunities, and develop and evaluate alternative courses of marketing action. The information generated in a market research study is dependent upon the technique employed; and, the technique employed is dependent upon management’s information objectives, derived from the problem definition, as well as the nature of the market.

It is frequently possible and often preferable to obtain market information through specifically focused qualitative research methods as opposed to the more costly and time-consuming quantitative survey methods that dominate marketing research practice. In technical and industrial marketing research, often aimed at hard-to-reach “subject matter experts” and professionals, surveys based on random samples can produce non-representative information because high demand concentration spells a high degree of information concentration among a very limited number of sources. It is not uncommon to find that less than one percent of the ‘knowledgeable’ persons associated with an industry possess virtually all of the relevant information about the market. This concentration of relevant information requires that careful attention be given to the sample selection process, using purposive judgment samples rather than random sampling methods[1]. One interview with a knowledgeable person can have greater information value than one hundred inter-views with uninformed respondents. A more logical and workable approach is to select a truly representative judgment sample of knowledgeable persons upon which results can be pooled to make valid decisions.

Delphi Investigative Paradigm

The Delphi method is a systematic, interactive research methodology based on structural surveys and makes use of information from the experience and knowledge of the participants, who are mainly “subject matter experts” (SMEs). It therefore yields both qualitative and quantitative results and draws on exploratory, predictive, even normative elements. The Delphi method was developed over a period of years by the Rand Corporation at the beginning of the cold war to forecast the impact of technology on warfare[2]. Since its development in the 1950s and 60s, this procedure has become a popular tool in long-term sales forecasting, technological forecasting, business strategy analysis, and new business development. The Delphi research paradigm is one of the most important exploratory techniques for investigating future manifestations in finite technical and industrial universes. For example, the iterative aspect of the method, as well as the in-depth normative nature of the investigative paradigm, lends itself to a wide range of new product development situations[3]. In instances where the market for a new product is not well-defined and/or the product concept is emerging, the technology is still fluid, and history seems inappropriate for forecasting future events, the Delphi technique can produce reliable and valid market intelligence.

The Delphi investigative paradigm can be accomplished by implementing a two-stage research program. The first stage is exploratory in nature and begins with a thorough review of the organization’s tech-nology goals and objectives, as well as an extensive review of secondary information relating to those goals and objectives. Stage II-Delphi represents an in-depth research effort with a broader, more representative panel of knowledgeable persons from target market businesses[4].

Stage I-Exploratory

With an emphasis on the boundaries that are used to define or limit the market in question, a preliminary view of the current situation is the key objective of the Stage I exploratory effort. The basic definition of a business, resource allocation decisions, identification and assessment of specific opportunities, and competitor threats are only a few of the areas which are strongly influenced by the way a market is structured. The Stage I-Exploratory effort can best be characterized as a preliminary market assessment that is used to focus the Stage II-Delphi research effort.

Exhibit A summarizes the Stage I approach and presents several objectives as well as two basic methodologies and four different information sources that are often employed in developing a preliminary view of the current situation. There are two interrelated efforts associated with the Stage I effort: (1) secondary research and (2) “expert” interviews. It is difficult to separate the two efforts since they overlap and are often performed in parallel to enhance investigative efficiencies. Virtually every question can be illuminated, at least in part, by secondary sources of information.

Secondary data can be internal or external to the organization and may exist in published or unpublished form. Secondary data are often supplemented with a number of expert interviews with key trade personalities. Expert interviews are often required to (1) identify key information sources, (2) close loose ends, and (3) obtain informed subjective evaluations. Frequently, the experts who are contacted during the Stage I exploratory effort are re-contacted several times during the course of the study to follow up on a particular topic or to test preliminary hypotheses Stage II-Delphi – Stage II utilizes the Delphi methodology to achieve a convergence of subject matter expert (SME) opinion, and the “key informant technique” to identify knowledgeable persons. The two keys to the successful use of SMEs are (1) identifying who they are, and (2) achieving a convergence of expert opinion. The Delphi method relies on repeated measurement and controlled feedback to obtain the consensus opinion of a group of SMEs[5] The selection of the panel of SMEs is critical to the success of the study. The pool of “experts” is often identified via the “key informant technique”[6] and recruited during the process of collecting relevant secondary data from trade publications and associations. The “key informant technique” represents a form of networking knowledgeable persons in an industry, thereby identifying which persons are considered most knowledgeable; these are the persons who are recruited as SMEs. Generally, 30-50 experts is an acceptable range. Less than 30 experts limits the analysis and may not provide confidence in the derived consensus. Interviews with more than 50 experts make administration of the Delphi method overly complex.

Exhibit B presents a six-step approach for implementing a Delphi investigation. Steps 1 and 2 describe the participant selection process. This process allows for the exclusion of marginal (less knowledgeable contributors) populations which, if sampled, provide less informative data. This two-step approach involves the initial contacting and screening of knowledgeable persons and the performing of in-depth telephone interviews with the recruited participants identified in Step 1. Based on previous experience, initial in-depth interviews are typically 45 minutes to one hour in length, depending on whether the participant is providing critical baseline information or supportive data. The convergence process, Steps 3 and 4, utilizes the Delphi method. With this approach, opinions of experts are converted into an informed consensus through a structured multi-step recruiting and polling process. The initial evaluation process, depicted in Step 3, includes the analysis of information needed to provide preliminary evaluations. This information is analyzed with respect to the participant’s area of expertise as well as how this might bias certain estimates, the completeness of the information obtained, and the exogenous and endogenous variables affecting this information. Participant evaluations that deviate from the general consensus are analyzed, and substantiated reasons for variance are cited. When unsubstantiated evaluations are identified, participants are recontacted to clarify their assessments. These participants are provided additional information obtained from other contributors and given the opportunity to revise their assessments. One of the most important benefits derived from this approach is that results are improved by having a trained moderator work with participants during the interviewing process using one or more methods to explore the basis of participant information, evaluations and assessments.

In addition to the original in-depth Delphi interviews, recontact and supplemental interviews are usually performed. Recontact interviews are usually performed to achieve a convergence of opinion on critical issues or to obtain additional information. Supple-mental interviews are usually required when multiple sources exist and these particular experts are crucial to the convergence process. Past experience indicates that supplemental and recontact interviews provide a level of validity that must be achieved in small sample Delphi studies. The analysis, depicted in Step 5, represents an integration and interpretation of in-depth qualitative information[7]. All expert interviews are taped and transcribed for this analysis. The in-depth qualitative analysis can best be described as a psychological content analysis. Following the in-depth content analysis, findings from this analysis and the Stage I-Exploratory secondary research are integrated to produce the final report and/or presentation.

Delphi Application – An Example: “Books in the 21st Century”

Breakthroughs in digital technologies have impacted printing and publishing processes and altered the roles and relationships of major industry players. Additionally, emerging electronic technologies have created new media for delivering intellectual information that has the potential to complement or compete with traditionally printed books. The overall purpose of this study was to provide printers and publishers in America with insight into forces influencing future book demand, book publishing and printing processes, and relationships between major players in eleven different book industry segments. This assignment was intended to explore and define how book publishing and printing is likely to evolve. The effort included forecasting the growth of key technologies and assessing the impact on demand for printed books.

As illustrated in Exhibit C, this Delphi investigation was conducted through a programmatic research effort. Stage I consisted of an exploratory investigation incorporating secondary information and in-depth interviews with industry experts. The general objective developed for Stage I was to investigate future book publishing trends from the perspective of major industry “experts” and to provide a historical timeline documenting major technological and societal milestones which have impacted the book publishing industry. We were interested in understanding what had happened to the market for books and, based on that understanding, developing an informed judgment as to what could happen in the years ahead.

In addition, Stage I provided a baseline forecast of book sales for the eleven book industry segments, along with estimates of how new technologies and other factors including demographic changes are likely to affect the baseline forecast.

Stage II provided an in-depth Delphi investigation with leading book publishers, printers, suppliers, and other technology experts. Knowledgeable individuals within these companies were asked to evaluate the baseline forecast derived in Stage I and provide their expert opinion on factors influencing book demand. These opinions were used to adjust and refine the Stage I baseline forecast to project the demand for books and related supplies. The major conclusion of “Books in the 21st Century” was that successful publishers, printers, and suppliers will be those who recognize and pursue the opportunities of electronic media. Those who are able to develop the capability to supply both traditional print and electronic on-demand publishing will be well positioned to continue growing into the 21st century. On the other hand, industry players who remain complacent will find the market moving away from them. Explicit details were provided for each of the book industry segments.


The ultimate objective of research is to generate meaningful and actionable information that is capable of reducing the risks in management decision-making. The purpose of the Delphi methodology is to exploit the specialized knowledge and judgment of experts to ascertain critical information that can be used by management to make informed decisions. The use of judgmental data (expert evaluations) in decision making is hardly avoidable today, and is rapidly becoming the foundation of many of the more advanced and sophisticated decision models in marketing.

The Delphi paradigm meets five key guideposts of sound inquiry: objectivity, reliability, validity, intensive analysis, and marketing applicability. We have used the Delphi technique to identify and assess B2B new product/market growth opportunities, investigate market structure changes, emerging and converging technologies, and evaluate strategic strengths/weaknesses in competitive environments.


Charles Ptacek heads up Charles & Charles Associates, Inc. and has over 25 years of consumer, industrial, and B-to-B quantitative and qualitative marketing research experience specializing in the identification and assessment of new product/markets growth opportunities, investigating market structure changes/emerging & converging technologies, and evaluating strategic strengths & weaknesses in competitive environments.



[1] Ptacek, C. H., “Nonprobability sampling assures representation and validity with B2B universes,” Quirk’s Marketing Research Review, March, 2008

[2] Lindstone, H., and Turoff, M. (ed.), The Delphi Method, Addison Wesley Publishing Co., 1975

[3] (Boufford, D., “Finding a clear path to new ideas,” Quirks.com, March, 2006

[4] Ptacek, C.H., “Using the Delphi investigative paradigm for market forecasting,” Quirk’s Marketing Review, December, 2007).

[5] Fink, A., Kosecoff, J., Chassin, M., & Brook, R. (1991) Consensus Methods: Characteristics and Guidelines for Use., Santa Monica, CA: RAND

[6] Marshall MN, “The key informant techniques, Family Practice, 1996, 13, 92-97

[7] Ptacek, C.H., “Beyond initial impressions,” Quirk’s Marketing Research Review, March, 2009

Leave a Reply

Your email address will not be published. Required fields are marked *