TOPIC 10 CONTENT ANALYSIS OF DOCUMENTS AND OTHER COMMUNICATION MEDIA An important feature of human environments is the messages that

people encode in various forms: 1. Written document ! Written materials include textbooks, students completed homework assignments, tests, computer printouts of school data, newspapers, and memoranda. 2. "i u#$ medi#: hotographs, posters, and drawings are examples of visual materials that ma! be anal!"ed. #. Audio medi#: $he researcher can anal!"e audiotape recordings, laser disc recordings, or radio programs. %. Com%in#tion o& medi#: &t ma!be desirable to anal!"e a variet! of t!pes of media, such as $' programs and ()*+,- discs that combine print, visual images, and sound. $he content of these materials comprises forms of communication from one individual or group to another individual or group. $hese messages are the ob.ect of stud! in some research pro.ects. $extbooks are of particular interest, because the! conve! much about the school curriculum. $extbooks and other written materials usuall! are called documents b! researchers. $he term communication media refers both to documents and to materials whose messages are primaril! visual or auditor!. I& 'ou ($#n to inc$ude communic#tion medi# # # d#t# ource in 'our re e#rc)* 'ou +i$$ need to %e ' tem#tic #%out identi&'in, #nd #n#$'-in, t)em . &n the following sections !oull find appropriate procedures, depending on whether !our research will be conducted from a /uantitative or /ualitative perspective. Content An#$' i in .u#ntit#ti/e Re e#rc) &n /uantitative research, the anal!sis of documents t!picall! involves content anal!sis. Content #n#$' i )# %een de&ined # 0# ,eneric n#me &or # /#riet' o& me#n o& te1tu#$ #n#$' i t)#t in/o$/e com(#rin,* contr# tin,* #nd c#te,ori-in, # cor(u o& d#t# to te t )'(ot)e e . $he raw material for content anal!sis can be an! t!pe of document or other communication medium. -ost content anal!ses in education involve collecting data on various aspects of the messages encoded in the communication product. $hese anal!ses generall! involve fairl! simple classifications or tabulations of specific information. (ontent anal!ses of student compositions for language arts classes, for example, could be used to develop a t!polog! of students grammatical and spelling errors and the fre/uenc! of different t!pes of errors $his information can be used to revise language arts courses or develop remedial programs. Ste( in Content An#$' i &n describing the steps involved in a content anal!sis, we refer to a research stud! of vocabular! instruction in social studies textbooks conducted b! 0anis 1armon, Wand 1edrick, and 2li"abeth 3ox.%4 Although this example involves printed text, the procedure are e/uall! appropriate for the anal!sis of other communication media, such as Web pages film, and audio recordings.

5pecif! research /uestions, h!potheses, or ob.ectives $he researchers stud! grew out of their appreciation of the important role that vocabular! instruction pla!s in students abilit! to understand social*studies concepts presented in the upper* elementar! and midd1e school grades. Also, the! were aware of effective research*based practices for promotin vocabular! development and wondered whether these practices were represented in th teachers editions of social*studies textbooks. $hese interests and concerns led 1armon and her colleagues to frame three researc /uestions to guide their stud!: 1. What is the nature of the words or ke! terms selected b! the social studies textbook publishers6 2. $o what extent and how is vocabular! represented at each grade level and across series of published social studies programs for grades %786 #. What vocabular! instructional supports do publishers provide for teachers6%9$extbooks, especiall! teachers editions, contain a great man! content*related elements. :! framing the research /uestions stated above, these researchers made explicit the scope of their investigation and also focused it sufficientl! so that a manageable stud! could be designed. 5elect a sample of documents to anal!"e. $he researchers selected teachers editions of the social studies textbooks on the 1;;97 1;;8 $exas state adoption list for grades %78. $he teachers editions were selected because the! contain both the text that students read and recommendations to teachers about how to teach this text. &t was necessar! to have documents that included these recommendations in order to address the third research /uestion stated above. )evelop a categor!*coding procedure. $he essence of a content anal!sis is the coding of the documents messages into categories. 2ach categor! should represent a discrete variable that is relevant to !our research ob.ective. $he categories should be mutuall! exclusive, such that an! bit of communication can be coded b! onl! one categor! in the categor! s!stem. &f appropriate, consider emplo!ing a coding s!stem that has been used in previous research. $his option saves the time re/uired to develop !our own s!stem. Also, the use of standard coding categories permits comparison with other studies that have used the same s!stem. (onse/uentl!, the research pro.ect is more likel! to make a contribution to theor! and knowledge related to the topic !ou are researching. &n the vocabular! stud! that we are considering here, the researchers developed their own categories, but the! represent standard features of textbooks, including teachers editions. Among the categories are these: instructional ob.ectives< instructional components =i.e., suggestions for tasks that teachers or students might complete in order to enhance instruction and learning>< and ke! terms =i.e., words or phrases highlighted b! the publishers>, with subcategories for =1> general terms that might be found in an! sub.ect =e.g., malad!, revenge>, =2> technical multiple*meaning terths =e.g., table, legend>, =#> technical, domain* specific words uni/ue to social studies =e.g., pueblo, colonist>, and =%> terms relating to specific persons, places, and events =e.g., :ull +un, )wight 2isenhower>. After initial development of the content classification s!stem, !ou should determine whether several raters can use it with a high degree of consistenc!. ?ou can do this b! calculating a correlation coefficient of inter*rater reliabilit! for different raters classifications. %8 &f the inter* rater reliabilit! is low, !ou will need to identif! points of ambiguit! in the content classification

s!stem and clarif! them. $o increase the reliabilit! with which the classification s!stem can be used, it is helpful to develop an explicit set of scoring rules. A specific reliabilit! check was not reported b! the researchers who conducted the vocabular! stud!. $here are repeated references to @weA in the report, so presumabl! the researchers checked each others coding as the content anal!sis proceeded. Bonetheless, bur confidence in the stud! findings would be strengthened b! a reliabilit! check. (oding of data A t!pical content anal!sis consists of making a fre/uenc! count of the occurrence of each coding categor! in each document in the sample. $he general procedure to create a computer file that reproduces the text of the document. =?ou can t!pe the text or possibl! use a scanning device that reads the text directl! into a computer file.> $hen read the computer file and t!pe a code for each message that fits a particular categor! of the content*anal!sis s!stem. A computer program can be used to count the fre/uenc! of each code, or to list together all the text messages that fit a particular code. (omputer programs of this t!pe are described in (hapter 1%. $he fre/uenc! counts can be presented in the results section of the research report. )escriptive statistics also can be reported, for example, the mean number and standard deviation of the occurrences of each coding categor! across all the documents in the sample. +elationships between the variables represented b! the different categories can be anal!"ed and reported, too. &n the vocabular! stud!, the researchers conducted several of these statistical anal!ses. 3or example, the! calculated the mean percentage of ke! terms that fit each of their categories across the textbooks included in the content anal!sis. $he! found that 98 percent of the ke! terms were in the categor! of domain*specific words, with another 1# percent in the categor! of terms relating to specific persons, places, and events. &n a related anal!sis, the! asked three teachers to identif! what the! considered to be ke! terms in representative samples of text from the textbooks =the samples had been scanned so that the teachers could not identif! which terms the publishers had marked as ke!>. 5urprisingl!, there was onl! %8 percent agreement between the publishers and teachers lists. $able ;.C shows the results of an anal!sis of the instructional components involving ke! terms that were included in the teachers editions of the textbooks. As !ou can see, the researchers classified the instructional components into three categories of tasks: instruction =i.e., teaching activities for the teacher>< application =i.e., activities re/uiring students to use the ke! term>< and review =i.e., activities in the review section of the textbook units, chapters, and lessons>. 3or most of the textbooks, the vast ma.orit! of the instructional components involved application and review. A more fine*grained anal!sis of the components revealed an emphasis on surface*level vocabular! activities, for example, filling in blanks, doing crossword pu""les, and matching words with their definitions. &nterpret the results $he final stage of a content anal!sis is to interpret the meaning of the results. $he interpretive process will depend on the purpose of the stud! and its theoretical or conceptual framework. &n the vocabular! stud!, 1armon, 1edrick, and 3ox used research knowledge about effective vocabular! instruction as a basis for evaluating publishers practices as revealed b! their content anal!ses. $his research knowledge demonstrates the value of such instructional activities as word sorts, semantic mapping, and graphic organi"ers to help students make connections

between their prior knowledge and new terminolog!. 1armon, 1edrick, and 3ox found virtuall! no utili"ation of this research knowledge in the textbooks the! anal!"ed: @,ur findings indicate that although publishers do give consideration to vocabular! in their programs, man! activities are still grounded in vocabular! teaching and learning activities that are not supported b! empirical evidence. $hus, publishers need to take a more aggressive stance to integrate current knowledge about vocabular! into their instructional procedures.A $he researchers also raise an important /uestion for future research, based on their data anal!sis of teachers and publishers identification of ke! terms in social studies textbooks: @)isagreements between teachers and publishers over what should he high*D lighted as conceptuall! loaded terms raises /uestions concerning how publishers select ke! terms.ACE $his is a good example of the principle that the value of a stud! often lies less with its findings and more in the /uestions it raises.

Anal!sis of )ocuments and +ecords in Fualitative ,bservation Fualitative researchers often stud! written communications found in natural situations. $hese written communications are of various t!pes. ?vonna Gincoln and 2gon Huba define documents as written communications that are prepared for personal rather than official reasons.C &n contrast, records are written communications that have an official purpose. 3or example, personal letters, personal diaries, and drafts of articles are documents, whereas legal contracts, commission reports for general circulation, tax statements, and newspaper articles are records.

)ocuments and records rel! primaril! on language to conve! meaning. ,ther communications7for example, mathematics, music, and highwa! signs7rel! on different t!pes of sign s!stems to conve! meaning. 5emiotics is a field of in/uir! that studies the meaning of these signs.C2 3or example, eter -anning and :ets! (ullum*5wan did a semiotic anal!sis of the various meanings conve!ed b! the different sign s!stems at -c)onalds restaurants, among them, the menu board, lighting, arrangement of space, utensils and food wrappings, outdoor pla!grounds, and the use of the @-cA prefix to label various food items.C# =5emiotics is explained further in (hapter 1C.> &n traditional /uantitative research, the meaning of a text is assumed to be invariant across readers and across time. &n other words, the meaning is in the text itself, and the meaning can be represented as discrete content variables and studied b! the methods of content anal!sis. &n contrast, /ualitative researchers believe that the meaning of a text resides in the minds of its writer and its readers. $hus, the meaning of a particular document or record can change from reader to reader and from one historical period to another. 3urthermore, a document or record can have different meanings at different levels of anal!sis. 3or example, the content of a textbook can be anal!"ed to determine what topics it covers. 1owever, it also can be seen as a secondar! source that is derived from primar! sources. = rimar! and secondar! sources are explained in (hapter %.> $he relationship between the textbook and its primar! sources thus can be the focus of a /ualitative researchers stud!. $o full! understand a document or record, therefore, the /ualitative researcher needs to stud! the context in which it was produced7the authors purpose in writing it, the authors working conditions, the authors intended and actual audience, and the audiences purpose for reading it. $he /ualitative researcher also must reali"e that, in reading and anal!"ing the text, she creates her own meanings. )ocuments and records from the distant past pose particular problems for the /ualitative researcher because it is not possible to interview the author or readers from that time period. Also, the researcher cannot observe the situations in which these written communications were used originall! or how the! were made available for different audiences. 1istorians have developed various research methods to deal with these problems. $heir methods are described in (hapter 14. Fualitative researchers follow some of the same steps as /uantitative researchers who use text and other communication media as data sources. $he! t!picall! begin b! identif!ing documents and records that are representative of the phenomenon that the! plan to stud!. ,nce the! have identified these materials, their next step is to determine which materials might be relevant to their research stud!. $hen the! determine how the! can collect these materials for anal!sis within the guidelines for ethical conduct of research. &f the materials cannot be removed from the natural setting, it ma! be possible to make photocopies and photographs of them in their setting for later anal!sis. ,therwise, the researchers will need to devise a method for anal!"ing them on site. 3inall!, the! wifi need to consider the validit! of the materials. (onceptions of validit! that appl! to historical documents and records =i.e., internal and external criticism> are described in (hapter 14. $he use of documents and records as data sources in /ualitative research differs most from /uantitative research in the anal!sis phase. &n /uantitative research, a set of variables is defined and applied uniforml! to all the written communications in the sample. $he variables are measured in such a wa! as to !ield /uantified data that can be anal!"ed b! conventional statistics. &n /ualitative research, the anal!sis procedure is likel! to be emergent. $he same document or record can be anal!"ed at different points in the stud!, with each anal!sis !ielding new constructs,

h!potheses, and insights. $he results of the anal!sis need not be expressed in /uantified form. 3urthermore, the same document or record can be anal!"ed from different perspectives and for different purposes. $he results of the /ualitative researchers anal!sis take the form of interpretations and h!potheses. 1odder proposes that these h!potheses and interpretations be weighed in relation to two different contexts7the context in which the documents and records were developed and the context in which the! are now being interpreted for research purposes. C% $he researcher must take into account variations in meaning as the! are studiedacross space, time, and cultures. 1odder suggests five criteria for confirming interpretations based on data obtained from documents and records: 1. internal coherence, meaning that different parts of the theoretical argument do not contradict one other and the conclusions follow from the premises< 2. external coherence, meaning that the interpretation fits theories accepted in and outside the discipline< #. correspondence between theor! and data< %. the fruitfulness of the theoretical suppositions, that is, how man! new directions, lines of in/uir!< or perspectives are opened up< and C. the trustworthiness, professional credentials, and status of the author and supporters of an interpretation. An example of the use of documents, records, and signs in /ualitative research is a stud! b! H. Henevieve atthe!*(have" concerning her perceptions of the cultural conflict between Gatino students and their mainstream teachers in a Gos Angeles high school.CC atthe!*(have" puts the high school in context b! describing the neighborhood in which it was located. 1er observations, carried out from 1;84 to 1;8;, include various forms of public communications in the neighborhood: @Area businesses now sport huge 5panish*language signs next to modest 2nglish*language ones< video stores advertise their 5panish*language collections< and several 5panish*language newspapers and radio and television stations compete ver! successfull! with 2nglish*language media.A 5he concludes from these observations that 5alvadorean and -exican cultural networks have been firml! and successfull! established in this neighborhood. atthe!*(have"s observations of the schools curriculum focused on the 2nglish as a 5econd Ganguage =25G> program. &n examining the text of 25G curriculum materials in use at the school, she found that the! emphasi"ed 2nglish grammatical structures. $his emphasis, she notes, runs counter to current recommendations for 25G methodolog!, which emphasi"es the development of students oral communicative proficienc!. atthe!*(have" draws upon a local newspaper article about the high school to illustrate what she perceives as its assimilationist mission, serving as @a port of entr! for educational opportunit! and the American wa!, a port of entr! that students, but for a few miscreants, were eagerl! passing through.AC9 &n her research report, atthe!*(have" illustrates this theme b! /uoting the following excerpt from the newspaper article: @5tudent (elia $oche, 19, came from 2l 5alvador with her famil! five !ears ago, knowing no 2nglish. Bow she is an honor student and hopes to go to college. 5he is usuall! so absorbed with ph!sics tests, economic theor! trigonometr! exercises and 2nglish literature that she rarel! thinks an!more of such painful losses and frightening times as when armed guerifias perched on the roof of her school.

5ometimes, however, she is reminded. @-! parents didnt get a chance to get an education,A she sa!s. @&ts a famil! treasure, to get an education here. & reall! value it a lot. 5o when & see people wasting the opportunities the!ve got here, the choices the! have here, thats when & think about 2l 5alvador. 1ere at Gima I1igh 5choolI, we know what we have7and we appreciate it.A atthe!*(have" claims that this students experience represents a biased view of the high schools effectiveness. &n working with other students from 2l 5alvador, she found that some of them perceived 2l 5alvadoran schools to be more rigorous and moral than American schools. $he use of a newspaper article as a data source for this stud! is significant for two reasons. 3irst, it reveals information about one students experience at the high school being studied. 5econd, the newspaper article creates meanings that are different from those created if the researcher had reported a personal interview with the student: An interview is a personal event, whereas a newspaper article provides a widel! circulated public record of a conversation. $he fact that the newspaper reporter chose to include this particular conversation supports atthe!* (have"5 claim that the school is supporting an assimilationist mission7a mission that the newspaper reporter apparentl! supports and wishes the public to know. &n fact, one can speculate that the newspaper reporter is telling the readers what the! want to hear, namel!, that their public schools are striving to assimilate immigrants and succeeding in the task. $he inclusion of the newspaper article as a data source widens the context of the stud! and provides a contrast with atthe!*(have"s other data that /uestion the assimilationist mission of schools. +2(,--2B)A$&,B5 3,+ J5&BH ,:52+'A$&,BAG AB) (,B$2B$*ABAG?5&5 -2$1,)5: 1. )etermine whether mechanical or electronic device can facilitate !our collection of observational data. 2. &n making /ualitative observations, provide ade/uate training for observer to minimi"e observer bias, response sets, contamination, drift, and reliabilit! deca! #. (heck the validit! and reliabilit! of /uantitative observational data that !ou obtain. %. &n designing a form for /uantitative observation, include onl! as man! variables as the observers can code reliabl! and avoid asking them to make overl! precise discriminations. C. &n doing /ualitative research, consider changing the focus of observation as data collection progresses in order to accommodate emerging findings and /uestions. 4. &n making /ualitative observations, consider carefull!7and take data on7how !ou react to informants and others in the field setting, and how the! react to !ou. 9. &n making /ualitative observations, prepare field notes that are rich in detail and in low* inference descriptions. 8. (onsider whether it is possible to measure !our research variables b! collecting observational data on ph!sical ob.ects and @residuesA of human behavior. ;. &n doing a /uantitative content anal!sis, include appropriate checks of validit! and reliabilit! 1E. &n doing a /ualitative anal!sis of communication media, consider the meaning of the text from different perspectives, including those of the author and intended audiences.