Do psychology researchers tell it like it is? A microgenetic analysis of research strategies and self-report accuracy along a continuum of expertise
- David F. Feldon
- … show all 1 hide
Purchase on Springer.com
$39.95 / €34.95 / £29.95 *
Rent the article at a discount
Rent now* Final gross prices may vary according to local VAT.
Abstract
Acquiring research skills is considered to be a highly challenging aspect of developing expertise in the social sciences. Because instruction and mentoring in these skills are typically grounded in the self-report of researchers, difficulties in learning the material may be due to the content and accuracy of these explanations. Using a mixed-method, microgenetic design, this study examines the explanations of problem-solving processes by researchers along a continuum of expertise during simulated experiment design and subsequent data analysis. Findings indicate that participants’ self-explanations are largely inaccurate. Further, frequency of inaccurate statements is positively associated with the frequency of abstract cognitive processes, such as mental modeling and situation assessment. Implications of these findings for instruction and future research directions are discussed.
Inside
Within this Article
- Characteristics of expertise in scientific problem solving
- Accuracy of self-report
- Research questions and hypotheses
- Methods
- Results
- Discussion
- References
- References
Related Content
Supplementary Material (0)
References (73)
- Adams, G. B., & White, J. D. (1994). Dissertation research in public administration and cognate fields: An assessment of methods and quality. Public Administration Review, 54(6), 565–576. CrossRef
- Allen, R. M., & Casbergue, R. M. (1997). Evolution of novice through expert teachers’ recall: Implications for effective reflection on practice. Teaching and Teacher Education, 13(7), 741–755. doi:10.1016/S0742-051X(97)00018-8. CrossRef
- Anderson, J. R. (1993). Problem solving and learning. The American Psychologist, 48(1), 35–44. doi:10.1037/0003-066X.48.1.35. CrossRef
- Anderson, J. R. (1995). ACT: A simple theory of complex cognition. The American Psychologist, 51, 355–365. doi:10.1037/0003-066X.51.4.355. CrossRef
- Anzai, Y., & Yokoyama, T. (1984). Internal models in physics problem solving. Cognition and Instruction, 1, 397–450. doi:10.1207/s1532690xci0104_2. CrossRef
- Bainbridge, L. (1999). Verbal reports as evidence of the process operator’s knowledge. International Journal of Human-Computer Studies, 51, 213–238. doi:10.1006/ijhc.1979.0307. CrossRef
- Baird, R. R. (2001). Experts sometimes show more false recall than novices: A cost of knowing too much. Learning & Individual Differences, 13(4), 349–355. doi:10.1016/S1041-6080(03)00018-9. CrossRef
- Barrett, L. F., Tugade, M. M., & Engle, R. W. (2004). Individual differences in working memory capacity and dual-process theories of mind. Psychological Bulletin, 130(4), 553–573. doi:10.1037/0033-2909.130.4.553. CrossRef
- Blessing, S. B., & Anderson, J. R. (1996). How people learn to skip steps. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22(3), 576–598. doi:10.1037/0278-7393.22.3.576. CrossRef
- Chao, C., & Salvendy, G. (1994). Percentage of procedural knowledge acquired as a function of the number of experts from whom knowledge is acquired for diagnosis, debugging, and interpretation tasks. International Journal of Human-Computer Interaction, 6(3), 221–233. CrossRef
- Chen, Z., & Siegler, R. S. (2000). Across the great divide: Bridging the gap between understanding of toddlers’ and older children’s thinking. Monographs of the Society for Research in Child Development, Serial No. 261.
- Chi, M. T. H., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In R. J. Sternberg (Ed.), Advances in psychology of human intelligence (Vol. 1, pp. 7–75). Hillsdale, NJ: Erlbaum.
- Chung, G. K. W. K., de Vries, F. L., Cheak, A. M., Stevens, R. H., & Bewley, W. L. (2002). Cognitive process validation of an online problem solving assessment. Computers in Human Behavior, 18, 669–684. CrossRef
- Clark, R. E., Feldon, D., van Merriënboer, J. J. G., Yates, K., & Early, S. (2008). Cognitive task analysis. In J. M. Spector, M. D. Merrill, J. J. G. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). New York: Routledge.
- Clement, J. (1988). Observed methods for generating analogies in scientific problem solving. Cognitive Science, 12(4), 563–586. CrossRef
- Collinson, J. (1998). Professionally trained researchers? Expectations of competence in social science doctoral research training. Higher Education Review, 31(1), 59–67.
- Cooke, N. J. (1992). Modeling human expertise in expert systems. In R. R. Hoffman (Ed.), The psychology of expertise: Cognitive research and empirical AI (pp. 29–60). Mahwah, NJ: Lawrence Erlbaum Associates.
- Cooke, N. J., & Breedin, S. D. (1994). Constructing naive theories of motion on the-fly. Memory and Cognition, 22, 474–493.
- Delamont, S., & Atkinson, P. (2001). Doctoring uncertainty: Mastering craft knowledge. Social Studies of Science, 31(1), 87–107. CrossRef
- Dufresne, R. J., Gerace, W. J., Hardiman, P. T., & Mestre, J. P. (1992). Constraining novices to perform expertlike problem analyses: Effects on schema acquisition. Journal of the Learning Sciences, 2(3), 307–331. CrossRef
- Dunbar, K. (2000). How scientists think in the real world: Implications for science education. Journal of Applied Developmental Psychology, 21(1), 49–58. CrossRef
- Ericsson, K. A., & Charness, N. (1994). Expert performance: Its structure and acquisition. American Psychologist, 49(8), 725–747. CrossRef
- Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Revised ed.). Cambridge, MA: Bradford.
- Feldon, D. F. (2007). Implications of research on expertise for curriculum and pedagogy. Educational Psychology Review, 19(2), 91–110. CrossRef
- Garcia-Mila, M., & Andersen, C. (2007). Developmental change in notetaking during scientific inquiry. International Journal of Science Education, 29(8), 1035–1058. CrossRef
- Hassebrock, F., Johnson, P. E., Bullemer, P., Fox, P. W., & Moller, J. H. (1993). When less is more: Representation and selective memory in expert problem solving. American Journal of Psychology, 106(2), 155–189. CrossRef
- Hoffman, R. R., Crandall, B., & Shadbolt, N. (1998). A case study in cognitive task analysis methodology: The critical decision method for the elicitation of expert knowledge. Human Factors, 40, 254–276. CrossRef
- Jackson, P. W. (1985). Private lessons in public schools: Remarks on the limits of adaptive instruction. In M. C. Wang & H. J. Walberg (Eds.), Adapting instruction to individual differences (pp. 66–81). Berkeley, CA: McCutchan.
- Kirschner, P., Sweller, J., & Clark, R. E. (2006). Why minimally guided learning does not work: An analysis of the failure of discovery learning, problem-based learning, experiential learning and inquiry-based learning. Educational Psychologist, 41(2), 75–86. CrossRef
- Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1–55. CrossRef
- Klein, G. A., & Calderwood, R. (1996). Investigations of naturalistic decision making and the recognition-primed decision model (Research Note 96-43). Yellow Springs, OH: Klein Associates, Inc. Prepared under contract MDA903-85-C-0327 for U. S. Army Research Institute for the Behavioral and Social Sciences, Alexandria, VA.
- Lamberti, D. M., & Newsome, S. L. (1989). Presenting abstract versus concrete information in expert systems: What is the impact on user performance? International Journal of Man-Machine Studies, 31(1), 27–45. CrossRef
- Larkin, J. H., McDermott, J., Simon, D. P., & Simon, H. A. (1980). Models of competence in solving physics problems. Cognitive Science, 4(4), 317–345. CrossRef
- Lohman, D. F. (1986). Predicting mathemathantic effects in the teaching of higher order thinking skills. Educational Psychologist, 21(3), 191–208. CrossRef
- Lovett, M. C., & Anderson, J. R. (1996). History of success and current context in problem solving: Combined influences on operator selection. Cognitive Psychology, 31, 168–217. CrossRef
- Merrill, M. D. (2002). A pebble-in-the-pond model for instructional design. Performance Improvement, 41(7), 39–44. CrossRef
- Moors, A., & De Houwer, J. (2006). Automaticity: A theoretical and conceptual analysis. Psychological Bulletin, 132(2), 297–326. CrossRef
- Morgan, D. L., & Morgan, R. K. (2001). Single-subject research design: Bringing science to managed care. American Psychologist, 56(2), 119–127. CrossRef
- Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs, NJ: Prentice-Hall.
- Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231–259. CrossRef
- Onwuegbuzie, A. J., Slate, J. R., Paterson, F. R. A., Watson, M. H., & Schwartz, R. A. (2000). Factors associated with achievement in educational research courses. Research in the Schools, 7(1), 53–65.
- Pearson, M., & Brew, A. (2002). Research training and supervision development. Studies in Higher Education, 27(2), 135–150.
- Radziszewska, B., & Rogoff, B. (1988). Influence of adult and peer collaborators on children’s planning skills. Developmental Psychology, 24(6), 840–848. CrossRef
- Rikers, R., Schmidt, H. G., & Boshuizen, H. (2000). Knowledge encapsulation and the intermediate effect. Contemporary Educational Psychology, 25(2), 150–166. CrossRef
- Ritter, F. E., Nerb, J., & Lehtinen, E. (2007). Getting things in order: Collecting and analyzing data on learning. In F. E. Ritter, J. Nerb, E. Lehtinen, & T. M. O’Shea (Eds.), In order to learn: How the sequences of topics affect learning (pp. 81–92). New York: Oxford University Press.
- Russo, J. E., Johnson, E. J., & Stephens, D. L. (1989). The validity of verbal protocols. Memory & Cognition, 17(6), 759–769.
- Schaafstal, A., Schraagen, J. M., & van Berlo, M. (2000). Cognitive task analysis and innovation of training: The case of the structured troubleshooting. Human Factors, 42(1), 75–86. CrossRef
- Schmidt, H. G., & Boshuizen, H. P. A. (1993). On the origin of intermediate effects on clinical case recall. Memory and Cognition, 21(3), 338–351.
- Schmidt, H. G., Boshuizen, H. P. A., & Hobus, P. P. M. (1988). Transitory stages in the development of expertise: The “intermediate effect” in the recall of clinical case representation studies. In Proceedings of the Cognitive Science Society Meeting (pp. 139–145). Hillsdale, NJ: Lawrence Erlbaum Associates.
- Schneider, W., & Shiffrin, R. M. (1977). Controlled and automatic human information processing: I. Detection, search, and attention. Psychological Review, 84(1), 1–66. CrossRef
- Schooler, J. W., Ohlsson, S., & Brooks, K. (1993). Thoughts beyond words: When language overshadows insight. Journal of Experimental Psychology: General, 122(2), 166–183. CrossRef
- Schraagen, J. M. C. (1990). How experts solve a novel problem within their domain of expertise (IZF 1990 B-14). Soesterberg, The Netherlands: Netherlands Organization for Applied Scientific Research. Prepared under HDO assignment B89-35 for TNO Institute for Perception, TNO Division of National Defense Research, Soesterberg, The Netherlands.
- Schraagen, J. M. C. (1993). How experts solve a novel problem in experimental design. Cognitive Science, 17(2), 285–309. CrossRef
- Schraagen, J. M. C., Chipman, S. F., & Shute, V. J. (2000). State-of-the-art review of cognitive task analysis techniques. In J. M. C. Schraagen, S. F. Chipman, & V. L. Shalin (Eds.), Cognitive task analysis (pp. 467–487). Mahwah, NJ: Lawrence Erlbaum Associates.
- Schunn, C. D., & Anderson, J. R. (1999). The generality/specificity of expertise in scientific reasoning. Cognitive Science, 23(3), 337–370. CrossRef
- Schunn, C. D., & Anderson, J. R. (2001). Acquiring expertise in science: Explorations of what, when, and how. In K. Crowley, C. D. Schunn, & T. Okada (Eds.), Designing for science: Implications from everyday, classroom, and professional settings (pp. 83–113). Mahwah, NJ: Lawrence Erlbaum Associates.
- Schwartz, D., & Bransford, J. D. (1998). A time for telling. Cognition and Instruction, 16(4), 475–522. CrossRef
- Siegler, R. S. (2006). Microgenetic analyses of learning. In D. Kuhn & R. S. Siegler (Eds.), Handbook of child psychology (6th ed., pp. 464–510). Hoboken, NJ: John Wiley & Sons, Inc.
- Siegler, R. S., & Crowley, K. (1991). The microgenetic method: A direct means for studying cognitive development. American Psychologist, 46(6), 606–620. CrossRef
- Singley, M. K., & Anderson, J. R. (1989). Transfer of cognitive skill. Cambridge, MA: Harvard University Press.
- Sloutsky, V., & Yarlas, A. (2000). Problem representation in experts and novices: Part 2. Underlying processing mechanisms. In Proceedings of the XXII Annual Conference of the Cognitive Science Society (pp. 475–480). Mahwah, NJ: Erlbaum.
- Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257–285. CrossRef
- Sweller, J. (1999). Instruction design in technical areas. Camberwell, Australia: ACER.
- Taylor, K. L., & Dionne, J. P. (2000). Accessing problem solving strategy knowledge: The complementary use of concurrent verbal protocols and retrospective debriefing. Journal of Educational Psychology, 92(3), 413–425. CrossRef
- Thorley, N., & Stofflet, R. (1996). Representation of the conceptual change model in science teacher education. Science Education, 80, 317–339. CrossRef
- van Gog, T., Paas, F., van Merriënboer, J., & Witte, P. (2006). Uncovering the problem-solving process: Cued retrospective reporting versus concurrent and retrospective reporting. Journal of Experimental Psychology: Applied, 11(4), 237–244.
- Veermans, K., van Joolingen, W., & de Jon, T. (2006). Use of heuristics to facilitate scientific discovery learning in a simulation learning environment in a physics domain. International Journal of Science Education, 28(4), 341–361. CrossRef
- Velmahos, G., Toutouzas, K., Sillin, L., Chan, L., Clark, R. E., Theodorou, D., et al. (2004). Cognitive task analysis for teaching technical skills in an animate surgical skills laboratory. The American Journal of Surgery, 187, 114–119. CrossRef
- Wigboldus, D. H. J., Dijksterhuis, A., & van Knippenberg, A. (2003). When stereotypes get in the way: Stereotypes obstruct stereotype-inconsistent trait inferences. Journal of Personality and Social Psychology, 84, 470–484. CrossRef
- Wilson, T., & Nisbett, R. (1978). The accuracy of verbal reports about the effects of stimuli on evaluations and behavior. Social Psychology, 41(2), 118–131. CrossRef
- Yang, S. C. (2003). Reconceptualizing think-aloud methodology: Refining the encoding and categorizing techniques via contextualized perspectives. Computers in Human Behavior, 19, 95–115. CrossRef
- Yarlas, A., & Sloutsky, V. (2000). Problem representation in experts and novices: Part 1. Differences in the content of representation. In Proceedings of the XXII Annual Conference of the Cognitive Science Society (pp. 1006–1011). Mahwah, NJ: Erlbaum.
- Yates, K. A., & Feldon, D. F. (2008). Towards a taxonomy of cognitive task analysis methods for instructional design: Interactions with cognition. Presented at the Annual Meeting of the American Education Research Association, New York, NY, March, 2008.
About this Article
- Title
- Do psychology researchers tell it like it is? A microgenetic analysis of research strategies and self-report accuracy along a continuum of expertise
- Journal
-
Instructional Science
Volume 38, Issue 4 , pp 395-415 - Cover Date
- 2010-07-01
- DOI
- 10.1007/s11251-008-9085-2
- Print ISSN
- 0020-4277
- Online ISSN
- 1573-1952
- Publisher
- Springer Netherlands
- Additional Links
- Topics
- Keywords
-
- Expertise
- Research methodology
- Automaticity
- Self-report
- Scientific problem solving
- Authors
-
- David F. Feldon (1)
- Author Affiliations
-
- 1. Washington State University, Pullman, WA, USA