The following research report was written in November 2006 and describes a pilot study conducted in the Fall of 2006. The goal of this research was to evaluate the effectiveness of Visual Thesaurus as a translational resource for middle school students with learning disabilities.
EDC’S PLAN
Judith Zorfass, Cathy Morocco, Karen Clay, and Alise Brann
November 2006
INTRODUCTION
As one of the four research teams contributing to the overall efforts of the National Center for the Study of e-Text (NCSeT), EDC is proposing to conduct a series of four design studies. Each study focuses on helping middle grades students improve their reading comprehension of social studies texts through their use of the technology tool Visual Thesaurus. ]Over time, the design and findings of one design study to lay the foundation for the subsequent study .
The design studies are taking place from August 2006-May 2007. Two short design studies have already been completed, one in August and a second one in October. Below, for each study, we identify the research questions, participating students, content, process, data gathering strategies, data analysis strategies, findings/conclusions, and recommendations for the next study.
Based on these two completed studies, we have included a proposal for a third design study to be conducted in November and December. Although we identify our plan to conduct a fourth study somewhere between January and March, we cannot specify the plan for that study at this time (without the results of the third design study).
SETTING THE CONTEXT
This section describes the context for our series of studies by identifying the literacy goals we are focusing on, the research site, the students, and the content.
Literacy Goals
Our studies concentrate on improving reading comprehension. Reading comprehension is defined as “intentional thinking during which meaning is constructed through interactions between the text and the reader” (http://reading.uoregon.edu/comp/comp_what.php). The ultimate goal of comprehension is to “make meaning” of text.
In the social studies classroom students are required to read a variety of materials for different purposes. For example, they are asked to read textbooks, worksheets, texts on websites, etc. Comprehension can occur at two different levels: literal and inferential.
- Literal comprehension refers to being able to identify and understand exactly what is in the text. In math, this would mean that students are able to recall, identify, classify, and sequence directions, details, facts, and stated main ideas.
- Inferential comprehension goes beyond the literal. It draws heavily on prior knowledge and experience. It is about making meaning of the text based on the goals of the reading experience. The following quote from Mosaic of Thought by Keene & Zimmerman (1997) captures the essence of inferential comprehension.
"It can be a conclusion drawn after considering what is read in relation to one's beliefs, knowledge, and experience. Inference can be a critical analysis of a text: a mental or expressed argument with an author, an active skepticism about what is stated in the text, or recognition of propaganda. Inference is, in some situations, synonymous with learning and remembering… Predictions are inferences. We base a prediction on what has been stated in the text, but we add to it an informed guess about what is to come." (p. 153)
The comprehension process does not only occur during that time that the reader is directly engaging in the text. As the chart below shows, the meaning-making process starts before reading, continues during reading, and extends beyond reading the text. For each phase of reading, Figure 1 describes what should be happening to ensure successful comprehension.
Figure 1: Reading Comprehension Across Phases
Phases of the Reading Process What Should Be Happening
Before Reading • Preview the text
- • Make a prediction
- • Build background knowledge
- • Understand key vocabulary
- • Set purposes
During Reading
- Check understanding
- • Make sure the meaning of words are understood
- • Monitor comprehension
- • Integrate new concepts
After Reading
- Summarize
- • Evaluate the ideas
- • Make applications
- • Utilize new vocabulary
As shown above, one aspect of reading comprehension is the development of vocabulary, especially vocabulary that helps to build concepts related to specific social studies content. Vocabulary problems are common characteristic of students with disabilities who struggle in reading, writing, listening, and speaking. Some students may have visual discrimination problems that cause them to confuse one word or term with a similar one. For others, it may be auditory discrimination problems, causing them to confuse one word for another during discussion or students might avoid including in their speech multi-syllabic or hard to pronounce words. There can also be other sources that cause difficulty. For example, students may have small working vocabularies, where they do not know the meaning or cannot retrieve the meanings or definitions of specific words from short or long term memory. In addition they may be unable to use context clues when they see a word embedded in text or apply strategies that will help them analyze a word, such as picking up on semantic features.
Technology Tool; Visual Thesaurus
Visual Thesaurus is designed to build word knowledge by allowing the reader to look up a word directly from a piece of electronic text. The tool provides a visual array of word meanings in web format. The array opens before the viewer’s eyes, adding descriptors around the focal word; if the viewer wishes to make a different word in the display the focal word, she clicks on it and the array moves to reorient items around that word. This mobile, three-dimensional quality is immediately appealing to adults and young people.
A viewer can adjust the display to include a narrow or wide range of kinds of word meanings. For example, the word “sail” can be a noun (part of boat), verb (guide a boat through the water), or adjective (sail cloth, sail boat); the Thesaurus can be constrained to define just the noun or verb. The tool is appealing to both adults and young learners because of the array of meanings and the ways that the learner can manipulate the array (rearrange around one of the synonyms, expand the array of descriptors) and see it move quickly in response, as shown in Figure 2.
Figure 2: Potato
Additional features allow the user to search the Internet for images related to the target word, hear the word read aloud and perform searches in a variety of languages (i.e. Spanish, French, German, Italian, etc.), and access Visual Thesaurus from Microsoft Word (PCs only).
As shown in Figure 3, The Visual Thesaurus embodies four of the categories of resource components that guide the work of NCSeT.
Figure 3: Selected Supported E-Text Resources
Category Specific Features
Translational Provides dictionary definitions and explanations
Illustrative Links to related visual information on the Internet
Enrichment Provides related terms and concepts such as synonyms, antonyms, in the form of a concept map; can be linked to related Internet websites
Collaborative Can be used in a small group context and can be linked to blogs
Overarching Research Questions
There are three categories of research questions;
- What characteristics of Visual Thesaurus text facilitate or impede access to and learning of social studies content?
- Does use of the Visual Thesaurus improve learning of academic content in actual educational settings with typical resources and levels of teacher support?
- What student characteristics (e.g., disability, technology skills, components of the tool) influence the effectiveness of use of Visual Thesaurus.
- What contextual factors (e.g., teacher training, hardware resources, student groupings) influence the effectiveness of supported e-text?
The Research Site
Our research site is Holten Richmond Middle School in Danvers, MA, a suburb of Boston. Holten Richmond serves 893 students in grades 6 through 8; approximately 14% of the students are identified as having special needs and 2% speak English as second language. As a newly renovated school, Holten Richmond is well-equipped with modern technology resources, including SMART Boards, projectors in every classroom, mobile laptop “labs” with wireless Internet technology and one computer for every six students. To utilize all this technology, the school administration had invested resources in several professional development trainings, particularly for the SMART Board.
Participating Students and Teachers
Our target students are those in grades 6-8 with mild to moderate learning disabilities and cognitive delays. We are interested in these students because their disabilities can have a direct impact on their ability to make meaning when reading—the crux of comprehension, and features of electronic learning environments may assist them in this comprehension process.
These students struggle with several aspects of reading: lack of word knowledge (vocabulary), lack of fluency, difficulty in literal comprehension, difficulty in inferential comprehension, poor ability to read strategically, lack of ability to identify text structures, and an inability to apply fix-up strategies. The first aspect—a lack of word knowledge (vocabulary)— characterizes most students with cognitive delay and many students with learning disabilities (LD) and contributes to several of the other aspects. Lacking word knowledge, the students focus on individual words, process slowly (lack of fluency), and fail to grasp the literal meaning (difficulty in literal comprehension). These all contribute to difficulty in making inferences from longer “chunks of text.” With impoverished word knowledge, students have difficulty reading and may bring less background knowledge to their content area learning.
The participating teachers include special education teachers, social studies teachers, and reading specialists.
DESIGN STUDY #1
Over a two-day period of time in mid-August, Catherine Morocco and Alise Brann worked with two students each to explore the use of Visual Thesaurus. Each day’s session lasted approximately 2 hours.A full report of their work can be found at http://ncset.editme.com/.
Research Questions
The following questions guided their work:
• How quickly and easily can students learn to use the tool? What difficulties do they have?
• To what extent does the tool engage the students and motivate them to look up words?
• What tool features are helpful? What features are confusing?
• What student characteristics are important to the success of the tool?
• What activity and text features are important to the success of the tool?
• What are directions for further study?
Participating Students
Cathy and Alise worked with four middle school students, one male and one female student each. Cathy worked with Stephen and Lisa; Alise worked with Margaret and David. The students were participating in a summer program in July and early August for students with identified reading and learning difficulties. All have individual education plans (IEPs). With the exception of Margaret, all students had just completed 6th grade (Margaret had just completed 7th grade). Lisa, David and Stephen are all placed in self-contained classrooms, while Margaret is placed in the general education classroom. As it turned out, these students all had serious decoding problems.
Content
At the suggestion of the summer school teacher, Cathy and Alise selected readings related to four states, New Mexico, Hawaii, Florida and Texas, which students had studied in class during 2005-2006. They identified readings with estimated varying levels of difficulty for each state; however, the texts were not analyzed for vocabulary level, lexile level, or other measure of reading difficulty. Both observers began with the readings on Florida to determine how students with different reading challenges and strengths used VT on a common set of texts.
Process
For each reading (e.g., Florida Manatees, Zebra Longwing Butterflies, Florida Wild cats, and the Seminoles) the researchers identified a series of comprehension questions and pinpointed words in the text that was likely to pose a challenge to the students’ answering the question. The procedure they developed for a work session, included the following.
- 1. Introduce the student to Visual Thesaurus, giving him/her a word to explore.
- a. Look up purple, blue, red. W
- b. Ask the student: what are you getting?
- c. Let the student choose a word, unstructured.
- 2. Show students how to print out a VT screen.
- 3. Introduce the student to the reading topics.
- a. “We are going to talk about Florida and here are some topics kids can read about. What interests you? Big cats, state butterfly, snakes, and panthers? Or are you more interested in native American history?”
- 4. Leaving VT open, go on-line and open the topic/text they have chosen to look at first.
- a. Remind them that they might see unfamiliar words and can use the VT to understand them.
- b. Show them how they can highlight and drag a word from the article to VT and how they can copy and paste a word into VT..
- 5. Introduce the reading and the pre- and post-reading questions, e.g. “Here are some things I thought about.”
- 6. Use the Chart of readings, questions, and words as a guide to working with the student, but be flexible in following the student’s questions and focus.
- 7. Read the selection together. Ask the student if s/he would like to start. If they say no, the observer can start by reading one or more sentences and then ask the student to read.
- 8. Talk about the text and ask them if they see words that are unfamiliar to them or might be hard for other kids. Circle or highlight on the hard copy and then have them look them up. Have the student print out the VT display periodically as we go along.
- 9. Relate words back to the questions. Look up other words related to that question. Keep a focus on that question if possible.
- 10. Continue with other words, linking back to the key questions whenever possible, but also playing with VT, and talking with the student. Encourage them to “think aloud” as they work with VT; get them to verbalize what they are doing and why.
- 11. Go to another topic or text if they are ready to move on. And do the same steps.
- 12. At the end of the session, ask “Do you think this would help you understand your reading? How would you use it? Would it help you write a paper? What part of VT might help you (go back to VT and have it on the monitor at this point.). If a student next year asked you what this is for, what would you tell him/her?
Example Drawn from One Student
After looking up several words, we continued reading and came to a section discussing coral snakes; leading David to look up the word ‘aggressive’. David easily highlighted the word and dragged it into the Visual Thesaurus window.
D: Voila!
A: Voila! Okay, so let’s see…belligerent…that’s a good word. Let’s see what else we’ve got here…predatory…Does that sound like another word we were reading about? Predator? What does a predator do? A snake is predator, a lion is a predator…
D: Hide from their…(unintelligible) enemies…
A: Close…a deer is not a predator, a squirrel is not a predator. But a wolf is a predator, or a bear, or a lion, or a snake. What do those animals all do?
D: Attack
Ultimately, David was able to make the connection between the animals I listed and ‘predator’ and what those animals do. However, this knowledge was not immediately available to him and required a little assistance to bring it out.
Data Gathering
The strategies used for documenting the researchers’ interactions with the students and their work with VT included the following:
• Tape recorded our full interactions with the student and transcribed portions
• Annotated the hard copy text as the student read, by circling words that posed difficulty for the student, noting points where decoding was slow or difficult, writing questions, or noting the students’ affect (high interest).
• Periodically printed out the VT display that the student was discussing.
• Asked the teacher about aspects of the students’ reading or behavior. “Is this typical?”
• Annotated the chart in Appendix A
Data Analysis
We prepared a lengthy report that included a detailed description of each of the four student’s interaction with VT (see http://ncset.editme.com/)
Conclusions
- After conducting this first short-term study with four students, the team concluded the following.
- Visual Thesaurus has the potential to help students who are struggling readers improve their reading comprehension. The presentation of words as concept maps with connections to other words and other ideas can help students gain a deeper understanding of their reading than they would simply reading a straightforward definition.
- However, Visual Thesaurus assumes that students have moderately good foundation reading skills of phonemic awareness, ability to decode text, reading fluency, and word recognition. It also assumes that students have a strong foundation vocabulary of high incidence words.
- •Students need to bring a set of readiness skills to use of the tool as listed in Figure 4 below.
- Figure 4: Readiness for Using Visual Thesaurus
- • Good foundation reading skills, including phonemic awareness, ability to decode unfamiliar words, moderately strong sight vocabulary for high incidence words.
- • Able to learn and manage the procedural demands of the tool, moving cursor, typing in text, clicking on Look it up, ‘drag and drop’, copy and paste words into search field; understands the system for revealing definitions and examples.
- • Able to self-limit the complexity of the array (e.g. recognizes the part of speech of the word and turns off /limits functions).
- • Understands the concept of “synonym”
- • Able to read “chunks” of text and hold the gist in mind while searching for the meaning of a particular word.
- • Able to filter extraneous information when definitions don’t make sense in the context of the reading
- Recommendations for Next Design Study
- We determined that the next design study should make the following modifications:
- • Profile of Participating Students; Rather than work with students who have serious decoding problems, we will select students who can decode, but who have trouble comprehending. These students still fall within the category of “struggling” but their difficulties are not at the foundation skill level. They can decode, but have difficulty making meaning from text. They are able to be included in the general education classroom, rather than a substantially separate setting, for social studies.
- • Grade: Rather than work with students in the 6th or 7th grade, we will work with 8th graders who struggle to comprehend. Even though they may have comprehension problems, they may have more background knowledge that can contribute to taking fuller advantage of Visual Thesaurus.
- • Content: Rather than use content created by the research team, we will use the social studies content from the inclusive setting.
DESIGN STUDY #2
During one day, in mid-October, Judith Zorfass, Karen Clay and Alise Brann worked with six students, two students each for 45 minutes at a time to further explore the use of Visual Thesaurus.
Research Questions
This second study was guided by the same research questions as used in the first design study:
• How quickly and easily can students learn to use the tool? What difficulties do they have?
• To what extent does the tool engage the students and motivate them to look up words?
• What tool features are helpful? What features are confusing?
• What student characteristics are important to the success of the tool?
• What activity and text features are important to the success of the tool?
• What are directions for further study?
Participating Students
The participating students were selected by two eighth grade special education teachers, Janis Clifford and Lauren Sideri. We worked with 6 eighth grade students who had identified learning disabilities and difficulties in the area of reading comprehension, but were fluent decoders.
Content
We selected a reading passage from the grade eight social studies textbook. The text – Journey Across Time: The Early Ages – is published by Glencoe/McGraw Hill and is available both in hard copy and as a PDF on CD-ROM. We accessed the text using the CD-ROM and read with the students the biography of Muhammad.
Process
Judy, Karen, and Alise all followed a common set of steps in working with students. These included the following:
• Showed VT to student and allowed student to play with or explore the tool by looking up words of the students own choosing such as, “blue”, or “basketball.”
• Opened the Mohammed selection.
• Pointed out the VT “button” and how it could be accessed.
• Asked student to begin reading the text aloud from the screen.
• Read one sentence or paragraph at a time.
• Paused at regular intervals and asked student to restate what they just read to measure comprehension
• Researcher or student selected a potentially difficult word that was important for concept building
• Looked up these words
• Read around the VT created web to see which of the “definitional” words the student knew
• Discussed the meaning of the word and accompanying definitions as shown in VT
• Clicked on the read-back function so students could hear the word pronounced correctly
• Read and discussed the definition from the column on the right of the screen as well as when the mouse rolled over the word
• Used antonyms if appropriate to help build concept.
• Reorganized the web by placing other words in the center to check definition.
• Looked up the Google images (especially for map of Syria and merchant and Mecca).
• Discussed the meaning of the word back in the sentence or paragraph.
• Explored relationships between several words from reading (i.e. looked up both Mecca and Mohammed; discussed relationship between two words, where Mecca is, why it is now a holy city, etc.)
• Related words to one another that had similar meanings (e.g., prosperous, merchant).
• Asked overall comprehension question that was included in the text at the end of the selection.
• Asked student to retell three important big ideas after concluding the reading.
• Debriefed with student about using VT.
Example Drawn from One Student
One of the students, Edward, worked with Alise. He became interested in Visual Thesaurus almost immediately and was readily reading the words in the web as we searched. We began looking up the word ‘green’ and exploring the various meanings of the word (unripe, color green, ill/sick, etc.). Thomas followed green to sick which then led to ‘bilious’ (which he read aloud), then clicking on ‘bile’ and ‘gall’ as we learned about the differences between merely feeling ‘sick’ and being ‘bilious’. Edward commented “It’s good that you can take one word and turn into a whole different thing…”
Once we began the reading from the text on Muhammad, Edward asked if he could read first. He read very clearly with few (if any) mistakes. At the end of the first section, I asked Edward about Muhammad’s early life.
A: So, was Muhammad’s early life easy?
E: No, he didn’t have any positive stuff…
A: Right, because it says his grandfather died. Who else close to him died?
E: His dad, his mom…
A: Yeah, so he was pretty much all alone
E: Except his uncle
Edward clearly understood the basic meaning of the paragraph; that Muhammad’s early life had been difficult and that he had experienced a great deal of loss. I asked him if there were any words he wanted to look up that were unfamiliar to him. He indicated that there were no words he wanted to look up, so I asked him if he knew the word nomads. He thought that it might mean ‘commoner’ so we looked it up.
VT: A member of a people who have no permanent home but move about according to the seasons”
A: So what does that mean to you? When Muhammad went and lived with the nomads…
E: He wasn’t in the same spot all the time
A: Right. Nomads are people who go place to place…So these are probably people who stay somewhere until winter and then go someplace a little warmer…
E: Yeah, they go south…Even though it is pretty warm over there….
E: But wait, where did Muhammad live originally?
A: Let’s look at the reading. It says ‘Makkah’
Edward and I tried looking up ‘Makkah’ in VT but didn’t find it, so we searched for it using Google.
A: Ah…it says here ‘The holy city of Makkah’ and it says that it is also called ‘Mecca’
E: Ohhh…so ‘Makkah’ is ‘Mecca’ just spelled differently
A: Right. So I bet if we look up ‘Mecca’…
E: Yeah! I want to do it!
A: So now, it is telling you that Mecca…
E: Is in Saudi Arabia
A: And what does this other definition say?
E: Located in Western Saudi Arabia; as the birthplace of Muhammad it is the holiest city of Islam
A: Okay. Good. So let’s go back to the reading. So that tells us even more about Muhammad. Not only was he poor…
E: He didn’t have a home
Edward very easily moved back and forth from internet searches, VT searches and the article, using what he had learned to make meaning of the paragraph and gain more depth of understanding. He had begun by understanding that Muhammad’s life was challenging because of poverty and the loss of his immediate family, but he also now understood that the loss of his home was also a factor in Muhammad’s early life.
Data Gathering
We gathered data in two ways. Each researcher tape recorded each student. Later, we transcribed selected segments of the tape that illustrated the student’s interaction with the tool.
Data Analysis
We are in process of reviewing the tapes and excerpting relevant sections that demonstrate each student’s ability to understand the meaning of words through the Visual Thesaurus and use this meaning to comprehend the text.
Findings/Conclusions
For these students, who had the basic prerequisite skills, we found that they were:
• All four students were highly motivated to use VT. They verbally expressed their interest in the tool (“cool”) and willingly used it.
• All four students quickly learned how to use the tool. Usually after the first demonstration, they were able to independently use the tool. They were very curious and interested in exploring all of the tool’s features.
• Setting the bar to have prerequisite skills was critical. Students were able to decode the meanings and discuss them.
• It was critical to put the words and meanings back in context to support comprehension of the text.
• Interaction with the tool was not enough; students needed instruction. It was important that the researcher, serving in the role of teacher or tutor, use different instructional strategies to make the most of VT. For example, we all point out contextual information, encouraged the students to use the definitions in context, asked prompting questions, and helped students link pieces of information. When Alise worked with Edward, she helped him to relate the information he learned about ‘Makkah’ (or Mecca) back to Muhammad’s life. He realized that he was originally from Saudi Arabia. Overall, the instructional strategies made the goal (understanding word meanings) explicit and kept the students focused on making meaning of the text.
Recommendations for Next Design Study
Based on this experience with six students, we felt that we should take the next step to develop a design study with the following features:
• We would continue to work with 8th grade students who have reading comprehension problems
• Instead of taking students out of classroom as we did in the first two studies, we would work with an entire social studies class that included struggling students
• We would continue to use the digitized version of the social studies textbook.
• We would be deliberate in identifying a set of instructional strategies that focus on understanding vocabulary to better understand text.
• We would develop these instructional strategies in collaboration with a participating social studies teacher.
• We would study the implementation of these strategies, in conjunction with VT, in the social studies teacher’s classroom.
PROPOSED DESIGN STUDY #3
We are proposing to conduct the third design study over a 4-6 week period in one social studies classroom during the months of November and December.
Research Questions
While research questions for the first two design studies basically focused on students’ ability to use the tool, the questions for this third study begin to address more of the overarching questions posed by NCSeT. These include the following:
• What characteristics of Visual Thesaurus text facilitate or impede access to and learning of social studies content?
• Does use of the Visual Thesaurus improve learning of academic content in actual educational settings with typical resources and levels of teacher support?
• What student characteristics (e.g., disability, technology skills, components of the tool) influence the effectiveness of use of Visual Thesaurus.
• What contextual factors (e.g., teacher training, hardware resources, student groupings) influence the effectiveness of supported e-text?
Participating Students
We will work with one social studies class that includes students with a range of diverse abilities and needs. We will select a heterogeneous class that includes students who struggle to comprehend. For our more intensive data collection, we will select eight – ten “focal” students, half who will be identified as struggling with comprehension (by teacher and MCAS scores) and half who are typical students without comprehension problems.
Content
We will focus on one digitized chapter in the social students textbook. We will begin the study at the start of the chapter and end it when the chapter ends. We were told this could last from 4-6 weeks.
Process
Our process will require plans for ensuring technology access, training, developing the instructional plans, and implementing the plans.
Ensuring Technology Access
• Making sure students will have access to computers
• Making sure students will have access to VT
Training
• Training the teacher to use VT (before the planning)
• Plan for training the students to use VT (during implementation, see below)
Instructional Planning
First, we will engage in collaborative planning with the participating social studies teacher. This will involve
• Agreeing on the chapter to be used from the social studies textbook (digitized)
• Selecting vocabulary words from the chapter that contribute to goals for the understanding key concepts
• Selecting instructional strategies to build vocabulary knowledge
• Planning how to link these strategies to VT
• Planning when and how to introduce the strategies to students
• Developing necessary student materials
Implementation
• Have teacher introduce VT and demonstrate how to use it with the in-class SmartBoard
• Have teacher introduce the vocabulary
• Have teacher introduce the instructional strategies
• Have teacher link the instructional strategies to VT
• Have students work independently or in small groups
• Hold classroom discussions of the words and strategies.
Data Gathering
For this design study, we will collect data in the following ways:
• Gather background information on the focal students who struggle with comprehension
• Administer chapter pre-test to entire class
• Administer chapter post-test to entire class
• During the course of the intervention (4-6 weeks), carry out four classroom observations, focusing on the focal students
• Conduct a “Think aloud” with focal students
• Collect student work when they have used the VT
• Collect the teacher’s plans
• Debrief with students
• Conduct debriefing interview with teacher
Data Analysis
In addition to the previous data analysis strategies used in the first two studies, we will also examine student work, compare pre and post test scores for all students and the focal students.
DESIGN STUDY # 4
At this point, it is too premature to propose a design for a study to take place after the New Year. Our plans will emerge after we conclude Design Study #3.