Scientists who study patterns in survey results might be dealing with data on language rather than what they鈥檙e really after -- attitudes -- according to an international study involving the University of Colorado Boulder.
The study, published in the journal听, found that people naturally responded to surveys by selecting answer options that were similar in language to each other as they navigated from one question to another, even when the similarities were subtle.
For the study, researchers looked specifically at surveys on organizational behavior, such as leadership, motivation and job satisfaction.
鈥淭he findings suggest many survey participants likely fit the first question into their language understanding and, when they get to the next question, move in their language network to figure out how close it is to the previous question in order to respond,鈥� said Kai Larsen, information scientist and associate professor of management and entrepreneurship at CU-Boulder鈥檚听. Larsen is a co-author of the paper.
The findings also raise questions about the way scientists design and analyze surveys, inadvertently focusing attention on the shared language understanding of respondents, said Larsen.
鈥淭he methods used for surveys are making it difficult to get at what鈥檚 unique about an organization rather than what鈥檚 embedded in general language,鈥� he said.
Often when social scientists conduct surveys with human participants, they look at more than just average scoring. In the results they detect -- and measure -- patterns. They quantify, for example, how much a popular answer to one question likely leads to a popular answer on another question to find common relationships.
The measurements help form statistics like, 鈥減eople who highly rate their manager鈥檚 leadership style are more likely to stay longer at their jobs.鈥�
In the case of the current study, researchers measured the degree of similarity in survey language instead of human response patterns. When they compared the measurements to measurements of human response patterns, the two sets of numbers were nearly identical, indicating the measurement of language similarities and people鈥檚 selection of survey answer options were practically the same thing. 听
For the study, the researchers applied two algorithms, or complex computer-operated calculations -- each using radically different approaches -- to measure sentence similarities.
The first algorithm involved about 100,000 newspaper articles to evaluate word similarities used within. The second algorithm relied on an online database created by linguists that shows the relationship between tens of thousands of words.
The surveys used in the study were already published and taken by anonymous respondents in a variety of fields from finance and government to engineering and the military. The respondents also included business students.
One type of survey that was not found to be language-based in the study was personality testing. 听
Other authors of the paper included lead investigator Jan Ketil Arnulf, associate professor of leadership and organizational behavior at BI Norwegian Business School; 脴yvind Lund Martinsen, professor of leadership and organizational behavior at BI; and Chih How Bong, senior lecturer in computer science and informational technology at the University of Malaysia at Sarawak.
The results of the findings may point to ways of improving research methods.
鈥淲ith surveys, we may be able to help researchers focus on respondents who aren鈥檛 answering in a language-based way,鈥� said Larsen. 鈥淏ecause they are revealing actual and unexpected attitudes, they may be the ones you want to pay attention to.鈥�
The study also highlights the growing prowess of data science.
鈥淪emantic algorithms are becoming new tools for the social sciences and are broadening perspectives on survey responses that other longtime theories cannot explain,鈥� said Arnulf. 鈥淭his represents a study of how the relatively young data sciences can address problems not approachable with traditional methods.鈥�
To see the complete study visit听.
Contact:
Kai Larsen, 720-938-2436
kai.larsen@colorado.edu
Elizabeth Lock, CU-Boulder media relations, 303-492-3117
elizabeth.lock@colorado.edu
Credit: iStock