Xiaojuan Ma will present her preFPO on Wednesday January 13 at 1:30PM in room 402. The members of her committee are: Perry Cook, advisor; Marilyn Tremaine (Rutgers) and Christiane Fellbaum, readers; Szymon Rusinkiewicz and Ken Steiglitz, nonreaders. Everyone is invited to attend her talk. Her abstract follows below. ------------------------------------------------- Title: A Multimedia Augmented Online Language Assistant for People with Aphasia and Other Language Barriers Abstract: Communication among language users take place usually in the spoken or written mode, as in face-to-face conversations, reading of books and newspapers, television watching and Internet browsing. However, for people with language disabilities (e.g. aphasia), people with low literacy, and people with poor command of a language, receiving and expressing information in this way is difficult. In particular, because of the inability to comprehend words and or to find words that express intended concepts, people with language barriers may be faced with great challenges performing everyday tasks such as ordering food in a restaurant and visiting a doctor. As an alternative to words, pictures and sounds have been designed, tested, verified and used to evoke concepts in computer interfaces, education, industry, and advertisement. However, icons created by artists, user-uploaded photos, auditory icons and earcons cannot always satisfy the need for communicating everyday concepts for people with language problems. We are building a Multimedia Augmented Online Language Assistant, which employs various multimedia forms, including web images, icons, animations, videos, and environmental sounds with the goal of assisting comprehension of common concepts. The Language Assistant is implemented as a popup dictionary in the form of a web browser extension. Users can select an unfamiliar word on a webpage to view its associated visual/auditory representation in a popup box. The Language Assistant can enhance concept understanding as people browse information on the Internet, and support face-to-face communication when people want to illustrate a term via a picture, a video or a sound when their conversation partner does not understand the spoken word. The Multimedia Augmented Online Language Assistant consists of a backend lexical network associated with large scale of multimedia data, and web interfaces to access the multimedia-concept associations. To develop the backend multimedia enhanced lexical database, we went through a cycle of design, construction, evaluation, and modification for each stimulus-to-concept associations. As an example we describe the creation of SoundNet, a nonspeech audio-to-concept semantic network which is part of the database. Soundnails, five-second environmental auditory representations, were extracted from sound effects libraries, and assigned to a vocabulary of common concepts. Two large scale studies were conducted via Amazon Mechanical Turk to evaluate the efficacy of soundnail representations. The first was a tagging study which collected human interpretations of the sources, locations, and interactions involved in the sound scene. The second study investigated how well people can understand words encoded in soundnails which are embedded in sentences compared to conventionally used icons and animations. Modification guidelines were proposed based on the study results. Through a series of studies, we verified that web images are as effective as stylized icons in conveying nouns, videos outperformed other stimuli in illustrating verbs, and nonspeech audio clips are better in distinguishing concepts like thunder, alarm, and sneezing. The Multimedia Augmented Online Language Assistant was shown to enhance information comprehension, and we are exploring its application in assisting communication involving medical care issues for people with language barriers and low levels of and medical literacy.