[talks] Xiaojuan Ma preFPO

Melissa Lawson mml at CS.Princeton.EDU
Wed Jan 6 13:43:06 EST 2010

Xiaojuan Ma will present her preFPO on Wednesday January 13 at 1:30PM in room 402.  The 
members of her committee are:  Perry Cook, advisor; Marilyn Tremaine (Rutgers) and
Fellbaum, readers; Szymon Rusinkiewicz and Ken Steiglitz, nonreaders.  Everyone is invited
attend her talk.  Her abstract follows below.


Title: A Multimedia Augmented Online Language Assistant for People with Aphasia and Other
Language Barriers

Communication among language users take place usually in the spoken or written mode, as in
face-to-face conversations, reading of books and newspapers, television watching and
Internet browsing. However, for people with language disabilities (e.g. aphasia), people
with low literacy, and people with poor command of a language, receiving and expressing
information in this way is difficult. In particular, because of the inability to
comprehend words and or to find words that express intended concepts, people with language
barriers may be faced with great challenges performing everyday tasks such as ordering
food in a restaurant and visiting a doctor. As an alternative to words, pictures and
sounds have been designed, tested, verified and used to evoke concepts in computer
interfaces, education, industry, and advertisement. However, icons created by artists,
user-uploaded photos, auditory icons and earcons cannot always satisfy the need for
communicating everyday concepts for people with language problems.  

We are building a Multimedia Augmented Online Language Assistant, which employs various
multimedia forms, including web images, icons, animations, videos, and environmental
sounds with the goal of assisting comprehension of common concepts. The Language Assistant
is implemented as a popup dictionary in the form of a web browser extension. Users can
select an unfamiliar word on a webpage to view its associated visual/auditory
representation in a popup box. The Language Assistant can enhance concept understanding as
people browse information on the Internet, and support face-to-face communication when
people want to illustrate a term via a picture, a video or a sound when their conversation

partner does not understand the spoken word. The Multimedia Augmented Online Language
Assistant consists of a backend lexical network associated with 
large scale of multimedia data, and web interfaces to access the multimedia-concept

To develop the backend multimedia enhanced lexical database, we went through a cycle of
design, construction, evaluation, and modification for each stimulus-to-concept
associations. As an example we describe the creation of SoundNet, a nonspeech
audio-to-concept semantic network which is part of the database. Soundnails, five-second
environmental auditory representations, were extracted from sound effects libraries, and
assigned to a vocabulary of common concepts. Two large scale studies were conducted via
Amazon Mechanical Turk to evaluate the efficacy of soundnail representations. The first
was a tagging study which collected human interpretations of the sources, locations, and
interactions involved in the sound scene. The second study investigated how well people
can understand words encoded in soundnails which are embedded in sentences compared to
conventionally used icons and animations. Modification guidelines were proposed based on
the study results.  

Through a series of studies, we verified that web images are as effective as stylized
icons in conveying nouns, videos outperformed other stimuli in illustrating verbs, and
nonspeech audio clips are better in distinguishing concepts like thunder, alarm, and
sneezing. The Multimedia Augmented Online Language Assistant was shown to enhance
information comprehension, and we are exploring its application in assisting communication
involving medical care issues for people with language barriers and low levels of and
medical literacy.

More information about the talks mailing list