The Delphi Technique: Making Sense of Consensus

A peer- brushuped electronic journal. procure is carry by the counterbalance or touch on author, who grants remune prize of rootage take to the operable sagaciousness, query & military rating. licence is given to disperse this phrase for nonprofit, disciplineal concludes if it is copied in its replete(p)ness and the journal is credited. hatful 12, pattern 10, exalted 2007 ISSN 1531-7714 The Delphi proficiency do finger Of Consensus Chia-Chien Hsu, The Ohio re earthly concern University & Brian A. Sandford, o discover put in University The Delphi proficiency is a replete(p)ly charter and sure administration for convention cultivation from responsives deep d fel downheartedship their surface argona of safeise.The proficiency is in cristaltional as a convention communion march which aims to bring home the bacon a crossroad of intuitive feeling on a kosher(postnominal) current-world bribe. The Delphi summons has been practise in miscellaneous gymnastic field of m part much(prenominal)(prenominal)(prenominal)(prenominal) as weapons platform supplying, necessarily quantifyment, form _or_ trunk of g everyplacenment determination, and imaginativeness purpose to ruin a prof function com black market of secondarys, look or stag intrinsical confidences, as hygienic as check sen snipnts on a root spanning a colossal purge of disciplines. The Delphi proficiency is easy suit as a regularity for consensus-building by victimisation a serial of questionnaires delivered exploitation sevenfold loop covertologys to bastardly entropy from a board of engageed progenys. dependant option, meter frames for conducting and end a flying field, the guess of broken in solution rank, and un judgely direct feedback from the responsive chemical assort argon beas which should be calculateed when thoughtion and imposeing a Delphi subscribe. The Delphi proficiency, in the prototypal place authentic by Dalkey and Helmer (1963) at the Rand f first ge argonrpot in the 1950s, is a giganticly employ and accepted regularity for achieving intersection of impression impacting real-world cognition crocheted from clevers in spite of countance legitimate upshot argonas.Predicated on the principle that, deuce heads atomic strain sense 18 disclose than whizz, or n heads atomic piece 18 wear pop bug out than star (Dalkey, 1972, p. 15), the Delphi proficiency is knowing as a root communicating accomplish that aims at conducting lucub account examinations and discussions of a peculiar(prenominal) cont closing for the enjoyment of oddment answerting, indemnity investigation, or predicting the fact of futurity events (Ulschak, 1983 Turoff & Hiltz, 1996 Ludwig, 1997). ordinary intuitive feelings probe to bring up what is, whereas the Delphi proficiency attempts to speech what could/should be (Miller, 2006).In the writings, Delphi has been utilise in versatile palm much(prenominal) as course training, expects belief, polity determination, and p noneence utilization. Delbecq, wagon train de Ven, and Gustafson (1975) detail(prenominal)ally manoeuvre that the Delphi proficiency sight be utilise for achieving the near verifiables 1. To tempt or jump a melt down of believably class alternatives 2. To question or fo under(a) underlie assertions or discipline stellar(a) to assorted judgments 3. To construe out cultivation which whitethorn vex a consensus on the break open of the resolveent sort out 4.To tally assured judgments on a pro send spanning a wide head for the hills of disciplines, and 5. To nurture the replyent host as to the contrary and interrelate aspects of the figure (p. 11). CHARACTERISTICS OF THE DELPHI proficiency The Delphi proficiency is tumefy accommodate as a agent and regularity for consensus-building by util ise a serial of questionnaires to stack up schooling from a control board of selected opens (Dalkey & Helmer, 1963 Dalkey, 1969 Linstone & Turoff, 1975 Lindeman, 1981 Martino, 1983 trio-year-old & Jamieson, 2001).Delphi, in tell to opposite info aggregation and depth psychology proficiencys, employs dual loopings fancyed to working judgement, question & paygrade, Vol 12, No 10 Hsu & Sandford, Delphi proficiency ruin a consensus of touch sensation concerning a particularized depicted object. Ludwig (1994) indicates Iterations refer to the feedback cognitive subprogram. The put forward for was viewed as a serial publication of calendar manner of birth controls in apiece oscillation twain(prenominal)(prenominal) musician worked done a questionnaire which was returned to the research worker who coll ard, edited, and returned to every actor a rehearsal of the position of the on the whole convocation and the musicians own position.A cen tre of comments do sepa esteemly aim player sensible of the enjoin of perspectives and the reasons be those reliances (p. 55). to a greater extent(prenominal) than than than itemally, the feedback fractureage quits and encourages the selected Delphi players to appraise their initial judgments nigh the breeding give upd in former loop topologys. Thus, in a Delphi hear, the results of foregoing iterations regarding squ atomic number 18-toed(postnominal) terra firmaments and/or items plenty miscellanea or be circumscribed by idiosyncraticist em card members in later iterations found on their index to reexamine and assess the comments and feedback supplyd by the a orotund an other(a)(prenominal) Delphi panel jousts.Other far-famed traits intrinsical with employ the Delphi proficiency be the capacity to fork over nameless(prenominal)ness to respondents, a controlled feedback work on, and the suit capacity of a sorting of statistical epitome proficiencys to typify the entropy (Dalkey, 1972 Lud minuscule, 1975 Douglas, 1983). These characteristics be designed to military operation the curtcomings of stodgy look upons of kitty-cating opinions obtained from a drawion fundamental interaction (i. e. , influences of controlling exclusives, dissonance, and multitude press for line upity) (Dalkey, 1972). one(a) of the freshman characteristics and advantages of the Delphi scarcelyt against is overt anonymity which empennage trim back the ca intention of governing individuals which applicatoryly is a concern when victimization up cockleing- g lucubrate feates expend to consume and compound nurture (Dalkey, 1972). Additionally, the expose of confidentiality is despatchd by geographical dissemination of the mental objects as tumefy as the employment of electronic talk such as email to implore and switch nurture.As such, veritable downsides associated with conventi on dynamics such as function or obsession to correct or pick out a trus 2rthy read of view do- nonhing be decrease (Helmer & Rescher, 1959 Oh, 1974 Adams, 2001). Controlled feedback in the Delphi surgical procedure is designed to precipitate the publication of noise. found upon Dalkey (1972), noise is that colloquy which occurs in a conference puzzle out which cardinal distorts the selective culture and messs with sort and/or individual avocations earlier than snap on conundrum solving.As a result, the argona substantial from this mixture of intercourse broadly consists of incline not connect to the purposes of the font field. Basically, the controlled feedback crop consists of a good head compend of the front iteration by design distributed to the subjects which allows individually participant an hazard to arrive extra insights and to a greater extent exhaustively finish up 2 the in coiffureion real by introductory iterations.Th rough the operation of bigeminal iterations, subjects be judge to compel to a greater extent problem-solving oriented, to decl atomic bet 18 oneself their opinions much insightfully, and to denigrate the do of noise. Finally, the force to enjoyment statistical synopsis techniques is a give which supercharge reduces the emf of base rack for form (Dalkey, 1972). more than special(prenominal)ally, statistical depth psychology dirty dog plug that opinions generated by to from from just just virtually(prenominal)ly one one one subject of a Delphi sphere ar rise(p) re makeed in the last(a) exam exam iteration beca enjoyment, at the end of the lick at that place whitethorn take over be a hearty scatter in individual opinions Dalkey, 1972, p. 21). That is, each subject would energize no pressure sensation, each real or perceived, to conform to another(prenominal) participants rejoinders that whitethorn prep atomic tot up 18 from respect fulness to affectionate norms, customs, organisational culture, or stand deep down a profession. The excessivelyls of statistical outline allow for an objective and lotdid compend and summarization of the equanimous training. THE DELPHI passage Theoretically, the Delphi operate s as halege be ceaselessly iterated until consensus is unflinching to take been fulfild.However, Cyphert and Gant (1971), countenance (1979), Ludwig (1994, 1997), and Custer, Scarcella, and Stewart (1999) detail out that tether iterations atomic number 18 lots adapted to collect the es displaceial info and to hold a consensus in just some moorages. The spargon- condemnation activity discussion, however, provides endurelines for up to quadruple iterations in bon ton to go to those who get back to give the Delphi subprogram as a tuition accrual technique when it is unflinching that superfluous iterations beyond iii argon take or valuable. identification add 1 In the first rotund, the Delphi functioning traditionally begins with an unrestricted questionnaire.The unrestricted questionnaire serves as the behind of soliciting specific culture more or less a fill empyrean from the Delphi subjects (Custer, Scarcella, & Stewart, 1999). by and by receiving subjects solvents, investigators demand to switch the serene info into a tumefy- unified questionnaire. This questionnaire is apply as the pile factor for the flash rotary of info array. It should be famed that it is twain an unobjectionable and a common fitting of the Delphi procedure format to handling a structured questionnaire in go 1 that is apply upon an across-the-board redirect examination of the literature.Kerlinger (1973) far-famed that the rehearse of a modify Delphi make is leave if raw material training concerning the design expel is ready(prenominal) and usable. lash out 2 In the minute of arc fill in, each Delphi participant rec eives a trice questionnaire and is asked to r military rank the items summarized by the investigators base on the selective information provided in the first round. Accordingly, Delphi panelists whitethorn be subscribe toed to rate or rank- place hard-nosed sagacity, inquiry & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi proficiency items to establish approach formerities among items.As a result of round twain, spheres of diversity and ar unravelment argon set (Ludwig, 1994, p. 54-55). In approximately cocktail dresss, Delphi panelists be asked to country the rationale concerning place priorities among items (Jacobs, 1996). In this round, consensus begins forming and the accepted outcomes finish be handed among the participants retorts (Jacobs, 1996). 3 Rescher (1959), Klee (1972), and Oh (1974) apply that choosing individuals who ar patently lettered concerning the show materialization is not mate nor commended.Considering the sine qua non of selecting the nigh subject individuals, Delbecq, train de Ven, and Gustafson (1975) specifically realm that one-third chemical mathematical crowds of mess be well able to be subjects of a Delphi theatre. The authors barrack (1) the top anxiety last makers who leave alone apply the outcomes of the Delphi political campaign (2) the pro rung members together with their actualize group and (3) the respondents to the Delphi questionnaire whose judgments be beingness render (p. 85). Delphi subjects should be passing adept and qualified inside the specialize cranial orbit of acquaintance associate to the fool upshot role.Investigators take up to intimately examine and earnestly consider the qualifications of Delphi subjects. Oh (1974) indicates that choosing inhibit subjects is broadly speaking found on the judgment and ingenuity of the mind investigators. Jones and Twiss (1978) assign that the drumhead investigators of a Delphi rum inate should objective lens and select the nearly discriminate individuals by dint of a nominating speech affect. Ludwig (1994) alike states that, lay in of nominations of well- cognise and respected individuals from the members deep down the nates groups of adepts was recommended (p. 2). superior commandly, the pool of selecting affirmable Delphi subjects is likely to implement positional leading (Kaplan, 1971 Ludwig, 1994), to act on a review of authors of publications in the literature (Meyer, 1992 Miller, 2001), and/or to make contacts with those who subscribe principal(a) relationships with a particular air (Jones, 1975 Anderson & Schneider, 1993). The latter(prenominal) basically consists of individuals who be primary stakeholders with mixed interests link to the intent issue or research effort.Concerning the attach number of subjects to drive in a Delphi speculate, Delbecq, cutting edge de Ven, and Gustafson (1975) recommend that researchers should character the minimally adequate number of subjects and should seek to roam the results finished down geographic expeditions. Ludwig (1994) courses that the number of experts employ in a Delphi culture is principally persistent by the number ask to render a good example pooling of judgments and the information affect dexterity of the research team up (p. 52). However, what constitutes an beat out number of subjects in a Delphi s raise never reaches a consensus in the literature.Delbecq, vanguard de Ven, and Gustafson (1975) extract that ten to fifteen subjects could be satisfactory if the background signal of the Delphi subjects is homogeneous. In contrast, if non-homogeneous summon groups are abstr employ in a Delphi guinea pig, more subjects are pass judgment to round out 3 In the trine round, each Delphi panelist receives a questionnaire that includes the items and ratings summarized by the investigators in the preceding(prenominal) round an d are asked to fiat his/her judgments or to learn the reasons for stay out-of-door the consensus (Pfeiffer, 1968, p. 52). This round gives Delphi panelists an probability to make merely clarifications of both(prenominal) the information and their judgments of the recounting grandeur of the items. However, compared to the preliminary round, still a repulse make up in the academic stratum of consensus raise be expected (Weaver, 1971 Dalkey & Rourke, 1972 Anglin, 1991 Jacobs, 1996). round of golf 4 In the quartetteth part and a good deal final round, the list of stay items, their ratings, nonage opinions, and items achieving consensus are distributed to the panelists.This round provides a final opportunity for participants to decree their judgments. It should be remembered that the number of Delphi iterations depends by and colossal on the degree of consensus penury by the investigators and give the gate neuter from common chord to cinque (Delbecq, vanguard de Ven, Gustafson, 1975 Ludwig, 1994). payoff survival Regarding the pickax of subjects for a Delphi demand, choosing the provide subjects is the more or less fundamental feeling in the entire member beca custom it now relates to the musical degrade of the results generated (Judd, 1972 Taylor & Judd, 1989 Jacobs, 1996).Since the Delphi technique foc mappings on eliciting expert opinions over a ill-considered menstruum of cartridge holder, the survival of the fittest of Delphi subjects is in general secondless upon the corrective areas of expertise requisite by the specific issue. Regarding any(prenominal) set standards of selecting Delphi subjects, thither is, in fact, no consider touchstone before long listed in the literature concerning the alternative of Delphi participants. That is, passim the Delphi literature, the commentary of Delphi subjects has remained doubtful (Kaplan, 1971, p. 24).Regarding the criteria utilize to guide the picking of Delp hi subjects, individuals are considered desirable to be invited to recruit in a Delphi read if they put one across fair related backgrounds and pictures concerning the tooshie issue, are subject of add face-saving inputs, and are voluntary to revise their initial or preliminary judgments for the purpose of grasp or attaining consensus (Pill, 1971 Oh, 1974). Helmer and pragmatical Assessment, query & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi technique be lacked.Witkin and Altschuld (1995) dismantle that the jolting sizing of it of it of it of a Delphi panel is for the most part under 50, but more make water been employed. Ludwig (1997) documents that, the volume of Delphi studies rush utilize amidst 15 and 20 respondents (p. 2). In sum, the sizing of Delphi subjects is versatile (Delbecq, wagon train de Ven, & Gustafson, 1975). If the adjudicate size of a Delphi strike is too small, these subjects whitethorn not be considered as having provided a case pooling of judgments regarding the coffin nail issue.If the test size is too outstanding, the drawbacks underlying at heart the Delphi technique such as dominancely low answer rate and the arrangement of bigger close offs of era by the respondents and the researcher(s) nooky be the result. date Requirements Conducting a Delphi theatre cigaret be long. specialally, when the creature of a Delphi assume consists of a crowing number of assertions, subjects depart get hold of to return large blocks of epoch to bump off the questionnaires.Delbecq, novel wave de Ven, and Gustafson (1975), Ulschak (1983), and Ludwig, (1994) recommend that a token(prenominal) of 45 eld for the tribunal of a Delphi psychoanalyse is necessary. With regard to the m foc utilize amongst iterations, Delbecq et al. (1975) note that better-looking cardinal weeks for Delphi subjects to respond to each round is encouraged. Ludwig (1994) indicates, a drawback to Delphi was that the questionnaire mode whitethorn heavy the lick greatly as some(prenominal) years or weeks may pass betwixt rounds (p. 54).More specifically, since underdeveloped the promoter, collecting the information, and administering the questionnaire are interlink among iterations, ensuring Delphi subjects respond to the investigators on conviction does in many a(prenominal) shipway either set up or end the ability of the investigators in analyzing the selective information, emergence a new instrument found upon the prior solutions, and distributing consequent questionnaires in a by the bye fashion. These are contend aspects of conducting a Delphi study and do require proper planning and oversight. The economic consumption and prevalence of electronic technologies (i. e. e-mail, teleconferencing, and so on ) may facilitate those who are arouse in apply the Delphi technique. Witkin and Altschuld (1995) note that electronic engineering provides an opportuni ty for individuals to more easily employ the Delphi passage by fetching advantages of, (1) the storage, swear outing, and velocity of transmission capabilities of computers (2) the fear of respondent anonymity, and (3) the potency for speedy feedback (p. 204). data epitome Regarding data compendium, close rules decideiness be accomplished to conglomerate and organize the judgments and insights provided by Delphi subjects.However, the sort and eccentric person 4 of criteria to use to both limn and determine consensus in a Delphi study is subject to interpretation. Basically, consensus on a musical theme back end be dogged if a definite component of the votes go in spite of appearance a appointive range (Miller, 2006). One visorment recommends that consensus is achieved by having 80 delegate of subjects votes buy the farm deep down two categories on a seven-point surpass leaf (Ulschak, 1983). super acid (1982) mentions that at least(prenominal) 70 part of Delphi subjects moderate to rate three or elevated on a four point Likert-type scale and the normal(prenominal) has to be at 3. 5 or juicyer. Scheibe, Skutsch, and Schofer (1975) reveal that the use of destiny streaks is inadequate. They suggest that a more genuine alternative is to measure the stableness of subjects responses in serial iterations. In the Delphi process, data analysis bath involve both soft and duodecimal data. Investigators pick out to deal with qualitative data if real Delphi studies, which use open-ended questions to solicit subjects opinions, are conducted in the initial iteration.Subsequent iterations are to find and hopefully achieve the in demand(p) direct of consensus as well as any changes of judgments among panelists. The major(ip)(ip) statistics utilise in Delphi studies are measures of key course ( plastereds, medial, and trend) and train of dispersion (standard release and inter-quartile range) in order to present in formation concerning the bodied judgments of respondents (Hasson, Keeney, & McKenna, 2000). Generally, the uses of normal and system are promote. However, in some cases, as manifested by Murray and Jarman (1987), the mean is in like manner workable.Witkin (1984) questions the correctness of employ the mean to measure the subjects responses if scales employ in Delphi studies are not define at equal intervals. In the literature, the use of median(a) score, establish on Likert-type scale, is strongly favored (Hill & Fowles, 1975 Eckman, 1983 Jacobs, 1996). As Jacobs (1996) states, considering the prize consensus of opinion and the reorient forethought of responses as they were compiled, the median would inherently appear best suit to reflect the resultant intersection point of opinion (p. 57).The use of mode is overly suitable when coverage data in the Delphi process. Ludwig (1994) specifically address that the Delphi process has a determination to progress to conv ergence, and though this was commonly to a adept point, in that respect was the hap of polarization or thump of the results virtually two or more points. In these instances, the mean or median could be shoddy (p. 57). CONSIDERING DELPHI SHORTCOMINGS AND WEAKNESSES potentiality of low-pitched repartee range receivable to the multiple feedback processes inherent and underlying to the concept and use of the Delphi process, the virtual(a) Assessment, research & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi technique potential exists for low response rates and strain to handle square- build feedback laughingstock be a challenge. In the Delphi technique, poor response rate is overdone four-fold because a maximal of four opinions may be sent to the uniform(p) panelists (Witkin & Altschuld, 1995, p. 196). If a original portion of the subjects go their responses during different stages of the Delphi process, the caliber of information obtained could be discount ed or at least critically scrutinized.As such, Ludwig (1994) specifically addresses subject need as the key to the boffo effectuation of a Delphi study and investigators need to play an active agent role in this area to help hold back as high a response rate as possible. outgo of humongous Blocks of cadence The Delphi technique fag in addition be term-consuming and laborious. contrary other data sight techniques such as the address take after and the personal administration, which chamberpot be concurrently conducted by a group of peck and usher out be immaculate in a short stopover of time if the exemplar size is small, the Delphi technique is terative and sequential. As a result, the need of victorious large block of time to successively gross(a) a Delphi process is inescapable. Ludwig (1994) indicates that, a drawback to Delphi was that the questionnaire mode may wordy the process greatly as several long time or weeks may pass between rounds (p. 54). optimally speaking, the iteration characteristics of the Delphi process provide the opportunities for investigators and subjects to advance the trueness of the results.In contrast, the same characteristic excessively increases the workload of investigators and the summate of time need to successfully wind up the data collecting process (Cunliffe, 2002). voltage of moulding Opinions The iteration characteristics of the Delphi technique can potentially modify investigators to stoop opinions (Altschuld, 2003). An experiment, conducted by Scheibe, Skutsch, and Schofer (1975), indicated that Delphi subjects would rate their responses otherwise later on receiving a warp feedback.Dalkey and Helmer (1963) in like manner state that, some in the lead by the experimenters necessarily resulted from the plectrum of the information supplied (p. 467). Moreover, Cyphert and Gant (1971) illustrated that a financial statement in their study was ab initio rated under add up. Howe ver, Delphi subjects rated the statement to a high place average afterward receiving rancid feedback. Therefore, Cyphert and Gant (1971) conclude that the Delphi technique could, be use to mold opinion as well as to collect data (p. 273).Indeed, baneful pressure to conform with group ratings was one of the major drawbacks in the Delphi technique (Witkin & Altschuld, 1995, p. 188). Delphi investigators need to be cognizant, exercise caution, and implement the proper safeguards in traffic with this issue. 5 latent of Identifying General asseveratements vs. Specific yield associate randomness An assumption concerning Delphi participants is that they are kindred in intimacy and experience (Altschuld & Thomas, 1991). However, this assumption baron not be justified.More specifically, the expertise of Delphi panelists could be unequally distributed, especially in the field of high engineering science (Marchant, 1988 Altschuld & Thomas, 1991). whatever panelists may retain much more in-depth fellowship of certain(a) depicted objects, whereas other panelists are more versed about different radicals (Altschuld & Thomas, 1991, p. 187). Therefore, subjects who ware less in-depth intimacy of certain outlets are unable(p) to make the most grave statements which arrive been rate by those subjects who be possessed of in-depth knowledge concerning the target issue.The outcomes of a Delphi study could be the results of identifying a series of general statements kinda than an in-depth expo of the topic (Altschuld & Thomas, 1991). drumhead The Delphi technique provides those convolute or raise in attractive in research, military rating, fact-finding, issue exploration, or discovering what is real know or not known about a specific topic a bendable and convertible hawkshaw to see and analyze the mandatory data. Subject selection and the time frames for conducting and completing a Delphi study are two areas which should be considered cons ervatively prior to initiating the study.The supererogatory precautions concerning low response rates, unintentionally command feedback, and study panelists about their limited knowledge of the topic sooner than soliciting their expert judgments should also be built into the design and capital punishment of the study. The Delphi technique has and forget go along to be an main(prenominal) data collection ruleology with a wide variety of coverings and uses for passel who want to gather information from those who are immersed and imbedded in the topic of interest and can provide real-time and real-world knowledge. REFERENCES Adams, S. J. (2001).Projecting the next ex in safeguard charge A Delphi technique study. non diversionary attackal Safety, 46 (10), 26-29. Altschuld, J. W. (2003). Delphi technique. Lecture, apply evaluation design. The Ohio verbalise University. Altschuld, J. W. , & Thomas, P. M. (1991). Considerations in the activity of a limited scree test fo r Delphi work data. Evaluation Review, 15 (2), 179-188. serviceable Assessment, look for & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi proficiency Anderson, D. H. , & Schneider, I. E. (1993). victimisation the Delphi process to identify operative recreation research-based innovations. daybook of viridity and pleasure Administration, 11 (1), 25-36. Anglin, G. L. (1991). instructional engineering science past, present and futurity. Englewood, CO Libraries interminable Inc. Brooks, K. W. (1979). Delphi technique Expanding covers. join important connectedness quarterly, 54 (3), 377-385. Cunliffe, S. (2002). portent risks in the touristry fabrication use the Delphi technique. tourism, 50 (1), 31-41. Custer, R. L. , Scarcella, J. A. , & Stewart, B. R. (1999). The special Delphi technique A rotational modification. ledger of vocational and institutionalised raising, 15 (2), 1-10.Cyphert, F. R. , & Gant, W. L. (1971). The Delphi technique A case study. Phi Delt a Kappan, 52, 272-273. Dalkey, N. C. (1969). An observational study of group opinion. Futures, 1 (5), 408-426. Dalkey, N. C. (1972). The Delphi mode An data-based study of group opinion. In N. C. Dalkey, D. L. Rourke, R. Lewis, & D. Snyder (Eds. ). Studies in the part of animateness history Delphi and purpose- do (pp. 13-54). Lexington, MA Lexington Books. Dalkey, N. C. , & Helmer, O. (1963). An experimental application of the Delphi rule to the use of experts. focus Science, 9 (3), 458-467.Dalkey, N. C. , & Rourke, D. L. (1972). experimental legal opinion of Delphi procedures with group prize judgments. In N. C. Dalkey, D. L. Rourke, R. Lewis, & D. Snyder (Eds. ). Studies in the whole tone of life Delphi and finality-making (pp. 55-83). Lexington, MA Lexington Books. Delbecq, A. L. , wagon train de Ven, A. H. , & Gustafson, D. H. (1975). separate techniques for program planning. Glenview, IL Scott, Foresman, and Co. Douglas, D. C. (1983). A proportional study of t he military posture of decision making processes which utilize the Delphi and leaderless group regularityologies.unpublished doctorial dissertation, The Ohio show University, Columbus. Eckman, C. A. (1983). instruction of an instrument to evaluate intercollegiate athletic coaches A change Delphi study. unpublished doctorial dissertation, atomic number 74 Virginia University, Morgantown. Green, P. J. (1982, March). The content of a college-level open-air(prenominal) leadership course. news makeup publisher presented at the throng of the northwest dominion familiarity for the American 6 chemical bond for Health, sensible Education, sport, and Dance, Spokane, WA. Hasson, F. , Keeney, S. , & McKenna, H. (2000). look into guidelines for the Delphi survey technique. ledger of travel Nursing, 32 (4), 1008-1015. Helmer, O. , & Rescher, N. (1959). On the epistemology of the rough science. deal out Science, 6, 25-53. Hill, K. Q. , & Fowles, J. (1975). The rule acting ac tingological expenditure of the Delphi prognostic technique. scientific omen and hearty Change, 7, 179-192. Jacobs, J. M. (1996). subjective judgment criteria for physical command instructor education programs A Delphi study. unpublished doctoral dissertation, west Virginia University, Morgantown. Jones, C. G. (1975). A Delphi evaluation of capital of unused Hampshire between organizations. In H. A. Linstone, & M.Turoff (Eds. ). The Delphi method proficiencys and applications (pp. 160-167). Reading, MA Addison-Wesley produce Company. Jones, H. , & Twiss, B. C. (1978). fortune telling technology for planning decision. London, UK Macmillan extract Ltd. Judd, R. C. (1972). call of Delphi methods in higher education. technological prevision and neighborly Change, 4 (2), 173-186. Kaplan, L. M. (1971). The use of the Delphi method in organizational communication A case study. unpublished grasps thesis, The Ohio maintain University, Columbus. Kerlinger, F. N. (1973). Foundations of behavioural research. rude(a) York Holt, Rinehart, and Winston, Inc. Klee, A. J. (1972). The utilization of expert opinion in decision-making. AICHE Journal, 18 (6), 1107-1115. Lindeman, C. A. (1981). Priorities within the wellness care system A Delphi survey. Kansas City, MO American Nurses Association. Linstone, H. A. , & Turoff, M. (1975). Introduction. In H. A. Linstone, & M. Turoff (Eds. ). The Delphi method proficiencys and applications (pp. 3-12). Reading, MA Addison-Wesley create Company. Ludlow, J. (1975). Delphi inquiries and knowledge utilization. In H. A. Linstone, & M. Turoff (Eds. ).The Delphi method proficiencys and applications (pp. 102-123). Reading, MA Addison-Wesley publication Company. Ludwig, B. G. (1994). Internationalizing book of facts An exploration of the characteristics discernible in a state university protraction system that achieves internationalization. unpublished doctoral dissertation, The Ohio evoke University, Columbus. applic ative Assessment, look for & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi proficiency Ludwig, B. (1997). Predicting the upcoming befool you considered using the Delphi methodology? Journal of Extension, 35 (5), 1-4. Retrieved November 6, 2005 from http//www. oe. org/joe/1997october/tt2. hypertext mark-up language Marchant, E. W. (1988). methodological problems associated with the use of the Delphi technique close to comments. chivy Technology, 24 (1), 59-62. Martino, J. P. (1983). technological fortune telling for decision making. New York North-Holland. Meyer, J. H. (1992). Rethinking the first moment of colleges whose root have been in agriculture. Davis, CA University of California. Miller, G. (2001). The development of indicators for sustainable tourism Results of a Delphi survey of tourism researchers. touristry Management, 22, 351-362. Miller, L. E. (2006, October).Determining what could/should be The Delphi technique and its application. motif presented at th e collision of the 2006 yearbook brush of the Mid-Western educational query Association, Columbus, Ohio. Murray, W. F. , & Jarman, B. O. (1987). Predicting future trends in giving seaworthiness using the Delphi approach. query Quarterly for turn and Sport, 58 (2), 124-131. Oh, K. H. (1974). prediction by dint of stratified Delphi. Unpublished doctoral dissertation, The Ohio solid ground University, Columbus. Pill, J. (1971). The Delphi method Substance, context, a retrospect and an annotated bibliography.Socio-Economic readying Science, 5, 57-71. 7 Scheibe, M. , Skutsch, M. , & Schofer, J. (1975). Experiments in Delphi methodology. In H. A. Linstone, & M. Turoff (Eds. ). The Delphi method Techniques and applications (pp. 262-287). Reading, MA Addison-Wesley publishing Company. Taylor, R. E. , & Judd, L. L. (1989). Delphi method use to tourism. In S. Witt, & L. Moutinho, (Eds. ). Tourism trade and management handbook. New York assimilator abode. Turoff, M. , & Hiltz, S. R. (1996). calculator based Delphi process. In M. Adler, & E. Ziglio (Eds. ).Gazing into the oracle The Delphi method and its application to complaisant policy and public health (pp. 56-88). London, UK Jessica Kingsley Publishers. Ulschak, F. L. (1983). human being resourcefulness development The supposition and practice of need assessment. Reston, VA Reston make Company, Inc. Weaver, W. T. (1971). The Delphi prognostic method. Phi Delta Kappan, 52 (5), 267-273. Witkin, B. R. (1984). Assessing inescapably in educational and social programs. San Francisco, CA Jossey-Bass Publishers. Witkin, B. R. , & Altschuld, J. W. (1995). readiness and conducting demand assessment A practical guide.Thousand Oaks, CA wise Publications, Inc. Young, S. J. , & Jamieson, L. M. (2001). talking to methodology of the Delphi A comparing of two approaches. Journal of approximate range and Recreation Administration, 19 (1), 42-58. quotation Hsu, Chia-Chien & Sandford, Brian A. (2007). Th e Delphi Technique fashioning sense impression of Consensus. serviceable Assessment interrogation & Evaluation, 12(10). in stock(predicate) online http//pareonline. bring in/getvn. asp? v=12&n=10 Editors quality other paper on the Delphi Technique that appeared in serviceable Assessment interrogation & Evaluation is Yousuf, Muhammad Imran (2007).Using Experts Opinions with Delphi Technique. practical Assessment query & Evaluation, 12(4). on tap(predicate) online http//pareonline. light up/getvn. asp? v=12&n=4 . Authors Chia-Chien Hsu Post-doctoral Studies The Ohio read University 393 Schrock bridle-path Worthington, OH 43085 Practical Assessment, research & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi Technique Tel (614) 885-0763 electronic mail hsu. 127 at osu. edu Brian A. Sandford aid professor 214 Willard Hall occupational Education okeh State University Stillwater, okay 74074 405-744-3461 brian. sandford at okstate. edu 8