4 Mayıs 2009 Pazartesi

Supporting Feedback and Assessment of Digital Ink Answers to In-Class Exercises
Kimberle Koile, Kevin Chevalier, Michel Rheiz, Adam Rogal, David Singer, Jordan Sorensen, Amanda Smith, Kah Seng Tay, Kenneth Wu -- 2007 (8 Pages)


Summary:
This article discusses Classroom Learning Partner (CLP), a system for classroom teaching based on Classroom Presenter (CP). CLP is a system that interprets and aggregates students answers on Tablet PCs. The aggregation (also used as "clustering" and "grouping" elsewhere) and interpretation parts employ AI techniques.

The system described in the article has four major parts. The first part is the Instructor Authoring Tool (IAT). The IAT is used by the instructor for preparing the questions during the lectures. While preparing the questions, the instructor can also designate where the students will be drawn and what the type of the answer will be (ie string, number, set, sequence, multiple choice etc).

The second part is the ink interpreter part. This part runs on the students PCs. The goal of the interpreter is to extract semantic information from the student's ink. That is, text, arrows and numbers are differentiated from each other. However, the authors say that the current implementation does not differentiate text from sketches and the interpreter passes those inks directly to handwriting recognizer. The handwriting recognizer distinguishes arrows from text. A new interpreter which will segment text and sketches is reported to be under development.

The handwriting recognizer built in the interpreter works in the following way: An ink segmentation module based on Microsoft's ink analyser segments inks into words or arrows. An error correction module fixes errors such as splitting a word into two words or combining two words into one. Then the strokes are passed to Microsoft English recognizer which outputs hypotheses. These hypotheses are sent to the language model and the stroke is labeled as number, string set, set, sequence or Scheme expression.

The recognition accuracy of the ink interpreter is measured with a modified version of Levenstein distance, as opposed to traditional word error rates.

The third part is the aggregator. It uses semantic information produced by the ink interpreter and builds a histogram of the inputs. Two methods are used for this: top-down and bottom-up. Top-down method places all answers into one group and splits the groups into two according to similarities. Bottom-up starts with each answer in its own group and merges the groups until an instructor-specified number of groups are left. The aggregator is responsible for decisions on answers such as "True, true, t, ..." which have similar meanings.

Other than numbers, strings, sets, sequences, true-false and multiple choice questions; new types can be added to the system by defining their similarity measures. It was said that the string comparisons use Levenshtein distance. Likewise, sequences use a modified version of this measure. Sequences are considered as strings and word by word comparisons are made between two.

A test was done on the aggregator where the results of it was compared with human aggregators. It is reported that the human aggregators found CLP's groupings reasonable.

The final part of the system is the results displayer, where answers to questions are displayed via histograms. The structure of the central database where the data are stored is also given in the article.

The evaluation part lists four hypoteses expected from the CLP: increasing student focus and attentiveness, providing immediate feedback, enabling instructor to adjust course material in real-time based on student needs, increasing student satisfaction. The evaluations were done in MIT's computer science classes. An experimental class was given tablet PC's and CLP was run. As a conclusion, the authors argue that fewer students than expected performed poorly. The lowest performing students were the ones in the non-Tablet PC classes. It is also sais that there is a pattern that the poorer performing students benefit the CLP most, and the top students are not benefiting from the technology. The authors think that top students's benefits are either not measured or they are already learning the material more quickly than their counterparts in the control class (non-tablet PC class).

Further work includes more learning studies and further development of CLP. New types such as sequence of characters are planned to be implemented. The authors say that they are working on interpretation and aggregation of sketched answers and marked answers, such as a circle around a multiple-choice answer on a map. These are different than sketches since they are dependent on background image.

Discussion:
A previous work by Richard Anderson, Craig Prince, Beth Simon ("Using Clustering to Assist Understanding of Digital Ink in Low Attention Environments", 2005) focused on sketch clustering but used a very primitive image based clustering algorithm. This study by Koile is a much more complex one, which works on the sketch strokes and semantics. The most important parts of the system are ink interpreter and aggregator. The recognition rate at the end will mostly depend on these two parts. As a result of using semantic interpretation, the system must be fed with new data types when those data types need to be recognized and grouped. This enables the system to group answers which have same meanings but different visual depictions. This means that it is theoretically possible to group answers "9" and "nine". However, almost every answer type should be introduced to the system manually, by defining the similarity metrics of that type.

The current study seems to be more focused on extracting the semantic meanings of the sketches rather than grouping them. It is not yet clear that this is the most efficient method. Other clustering algorithms based on temporal sketch data may yield better and faster results than this approach.

Furthermore, the classroom studies yield interesting results. It is almost obvious that only the poor performing students benefit from clustering. How the better students may benefit from clustering is not yet known. The current situation suggests that clustering only enables poor performing students to learn faster, and it does not dramatically increase the performance beyond a certain point. This may be attributed to the current clustering performance of the CLP which is poor in some answer types such as sequences.

Hiç yorum yok: