From jastenzel@ucdavis.edu Thu Jan 30 17:42:25 1997 Date: Thu, 2 Jan 1997 13:59:06 -0800 (PST) From: John Stenzel To: tcleamy@ucdavis.edu Cc: pgblando@ucdavis.edu Subject: Performance tests; Reconfiguration Tim and Pete: this humongous message has two main parts. The first gives details of performance tests that should be run. The second (longer) part is a combination of status report, critique, and blueprint for a radical rethinking of a computer classroom. I need for you to apply your considerable intelligences to the questions presented, so that I can make appropriate recommendations to the appropriate administrative entities. Some of this is draft that will be incorporated into a more formal report on the state of computer-assisted composition instruction at UC Davis. As such, there's a bit of time pressure; on the other hand, I am more interested in your creative and imaginative investment than in any off-the cuff response. Looking forward to seeing what y'all come up with at every phase of this! --J Baseline Performance Tests: Classroom procedures Here are some basic procedures for the computer classrooms. As students enter, they log in with their usercodes and passwords, and click on the "I have read these instructions" splash screen. Launching MS-Word used to be the first, default step that I suggested from the beginning of the quarter--analogous to taking out a pen and piece of paper for notes or writing assignment. However, with Word 6 the routine is different because the program tied up machines, making them run impossibly slowly and sometimes bringing the entire classroom sub-network down. Just opening the File menu can take minutes in some cases. Launching Daedalus (which has its own primitive word processor) may become the default first step now, although the limits of that program are clear (opening files takes another step, can't double-click on the icon but have to go through file menu, another proprietary format problem, etc.). Still, for comparison purposes, Word is still the default. The most common classroom operation is to have a student open a word-processed document, make some changes in it, save it, and drop it off in an instructor's drop-off area. Thus, the base-line tests I'd like to see are the following: I splash-screen to a ready-to-type Word new-document screen II splash-screen to ready-to-edit Word-opened medium-sized file (say 4 pages) (is there a difference between launching Word by double-clicking on the icon versus selecting from under the Apple icon? Also, should we check on whether having a room full of a mixture of IBM-formatted files and Macs puts a different load on the classroom?) III classroom full of edited files to all saved to disk and copies dropped in appropriate drop-off folder IV splash screen to Daedalus Interchange conference V all students in InterChange conference with a paragraph to submit, to all students able to read the submissions in the conference * * * Tests I and II should be fairly straightforward, along the lines of the tests that Pete has begun (albeit not yet systematically or with complete success). Here are some details of other typical operations in tests III-V Test III, file save to file dropped off: o Save file as (for example) "Student 3's essay" to disk o Access file server > class folders > instructor's folder > drop-off folder o Drag student essay to appropriate folder in drop-off o Click on AppleShare's confirmation box to complete drop-off o Close boxes and return to basic word-processing mode Do this with 20 or 25 students at a time, within say a 3-minute period, and you emulate a typical end-of-class activity. Test IV, Daedalus InterChange launch: o Launch Daedalus from hard drives o Type in name to register with the server o Pull down "Activity" menu and select "Interchange" o Pull down "Interchange" menu and select "Join a Conference" o Select "Test conference" Measure the time from start until everyone has the right blank conference screen in front of them; then go to Test V. Test V, Daedalus InterChange performance test: o Write a paragraph or so (say, 50 words) on every scratchpad in the conference o Within about a minute, send all messages to the conference Measure the time until every screen has every message readable--which may be difficult to ascertain, since the scrolling down is pretty inconsistent and lengthy. Questions: does it slow the network down to have students scolling up and down the message list as the messages come streaming in? [Does changing the refresh or delay rate improve performance?] Might want to try it completely passive and more class-like (with students trying to scroll up and down)--this might help guide instructors, if there's a way to avoid delays. Do this with 20 or 25 machines. In Olson upstairs or other labs where other server activity may compromise performance, we _must_ get a better idea of what kinds of activity slow down classroom response by how much. Numbers of seconds or minutes! Status report--end of Fall Quarter; thoughts for future The perception amongst rank and file English Composition instructors is that the computer classrooms are slower and less reliable than they were last year, and that they have become less attractive places to teach. Some of this is beyond IT-Labs' control, but some of it is due to decisions about architecture and upgrades. One of the reasons we must have better performance testing and baseline figures for classroom-related operations is that improvements will be difficult to demonstrate without them. Even so, experience in the trenches under game conditions will always lag behind the articial test environments, and IT-Labs has to make a bigger effort to ascertain what actually is going on in the classrooms themselves. It's easy to be lulled into a false sense of security when one measures lab performance in terms of complaints received: early this quarter, in the wake of the switch to Word 6, the networks were crashing so often that most instructors didn't bother to use the computers at all. A few instructors documented problems via v-mail and e-mail to me, but they quickly tired of complaining, even though I assured them I was trying not to take the disasters personally. Even when Word was transferred to the hard drives from the central servers, glitches remained that caused still more crashes, further eroding my own and instructors' confidence in the labs' reliability. Thus, even though the end of the quarter may have seen a reduction in the number of complaints reaching Pete and Tim's ears, the damage had already been done. Clearly, we need to reassess the mode of assessing classroom performance. Toward that end I have prepared a set of basic descriptions detailing what I consider the most important and frequent classroom activities, and have urged Pete to conduct baseline performance tests on all classrooms. These tests are only the start of a larger program of quality assurance that must be integrated into IT-Labs' normal duty cycle, if IT-Labs is to serve the pedagogical needs of the Computers in Composition program. The test results from last quarter, as presented by Pete Blando, are clearly unacceptable. During the quarter, in October, IT-Labs personnel were unable to launch MS-Word 6 on all upstairs Olson machines without significant numbers of crashes. Even in the best of the tests, it took more than 12 minutes for all machines to get ready for classroom activities. Even in an 80-minute class period this is far too long, and completely unworkable in the 50-minute slots that some composition courses still take. I still have a difficult time persuading both Tim and Pete that such response times are not compatible with the teaching mission. Technological solutions like a faster network and better cabling are supposedly in the offing, but so far faster machines have paradoxically led to slower overall response. The reasons for this are best left to Pete and Tim to explain. In my meeting on December 20, we discussed the placement of a new 3Com hub upstream of the classrooms, which may help matters, but even here the issues of convenience for IT seem to outweigh what would work better for the teachers in the classroom. Network architecture might have dictated placing this new box in one location rather than another, but the logistics of running a cable set from 241 to 247, with concomitant hassles about getting Physical Plant to do the work in a timely fashion and to make the hole in the right location, seemed to override the advantages. This seems like a minor point, but it actually represents a major stumbling block: we seem to be overlooking the fundamental purpose of these classrooms as classrooms. As I pointed out at the end of our meeting, instead of compromising on performance and making things convenient for the runners and builders of the labs, we should weigh the pedagogical needs of the users more carefully: if we can gain a performance improvement with a few hours' hassle with Physical Plant and some cable-pulling, let's not dismiss it. In effect, however, we seem to be flushing another year--putting up with cobbled-together solutions and mediocre reliability--in hopes that Network 21 improvements will be the magic bullet. So far at least, I remain unconvinced, given the track record. Economics and the Registrar's target numbers There are economic and political overtones to this debate as well: at the same meeting, Tim pointed out that the Registrar is dissatisfied with the occupancy rate of the 241 / 247 classrooms, and would like to see us reach state-mandated targets like 35 hours of classes a week. Though this was the first I had heard of this particular number, I look forward to discussing these matters with the Registrar herself. If the Department's Composition Program is to be responsible for filling these rooms more regularly, we must have a better product to sell, and we must have a greater say in the design and implementation of the rooms. Decisions made years ago, and persisted in through logistical inertia or lack of money, are now showing their consequences in financial as well as pedagogical terms. As the Classroom Design report makes clear, 241 Olson is a terrible place to try to teach composition or have discussion: besides the overcrowding, students are isolated from each other and from the instructor, the traffic patterns are obstructed and discourage collaboration, and short students become increasing invisible as monitors get bulkier. Add the fact that the upstairs Olson classrooms are highly unreliable--as IT-Labs' own test results of Fall 96 clearly confirm--and you have a very tough sell indeed. I cannot in good conscience attempt to schedule more classes in such rooms that offer no advantages over standard classrooms. More profoundly, the very methods by which the classroom computers are networked have pedagogical consequences not apparent to the naked eye or to the untrained observer. For example, having multiple classrooms share small numbers of 10-megabit lines (and having classroom servers handling thousands of pieces of electronic red tape because of bloated software) means that response times and reliability are often woefully inadequate for the teaching mission. In short, if these rooms are to be assessed as classrooms, let us design and configure them, first and foremost, as classrooms. In many cases this will not mean adding lots of expensive equipment, but it may mean spending our money more wisely and changing the scope of what we are trying to do in specialized computer rooms. Configuring Classrooms for Class Use As I have suggested informally to Tim and Pete, and as I more formally do now, the pedagogical needs of composition instructors might be better served by a more radical re-thinking of the lab classrooms as a whole. [This is draft, Tim and Pete, and I need your comments, in writing, now more than ever.] The paragraphs that follow describe a configuration that would work better for teaching English composition than the current configurations. This represents what I believe would be a more effective teaching environment; what Tim and Pete must do is come up with a detailed discussion of what exactly would be entailed in setting up such labs, how much the alterations would cost in terms of equipment and time. If the set-up described is logistically impossible, they should also provide a point-by-point and room-by-room explanation; if logistics are the main issue, I advise that 247 Olson, which is the most composition-intensive room, be considered as the site of a pilot set-up, as soon as possible. From all of the above, and from my own experience over many years of teaching and two-plus years as Coordinator and liaison with IT-Labs, it is clear that the multiple function of labs-as-classrooms--the idea that one roomful of machines must do everything we ask of it--is the root of the pedagogical limitations of these rooms. Here are the features of a composition-centered computer classroom, as opposed to a lab: o The room is a local network unto itself, not dependent upon a general connection to UCDNet or the classroom backbone o Individual student machines are not connected to the wider network, but are connected at Ethernet speed to each other, the server, and the instructor's machine o Only the instructor's Macintosh is connected to the campus backbone and the Internet at large o The hard drives have MS Word 5.1a installed on every drive, including the Word 6 converter; the other software such as Excel would be optional or omitted as inappropriate to the main purpose of this classroom o The hard drives have Daedalus installed, and possibly the Timbuktu screen-view software o Each Macintosh would be able to connect to the classroom server for use with AppleShare file pick-up and drop-off, but the server would only serve one room Some clarifying comments on this configuration I remain unconvinced by IT's arguments for Word 6, and until they can make it run as fast or faster than Word 5.1a (running on individual hard drives, not served from the server--it is unfair to compare Woird 5.1 from the server with Word 6 from hard drives--the comparisons have to be legitimate!). I will send IT-Labs a fuller draft of my report on the disavantages of Word 6--and the critique of the entire mechanism by which the decision to upgrade was made--by the beginning of Winter quarter. No _pedagogical_ reason for upgrading to the newer version has ever been made by IT-Labs, despite clear statistical and anecdotal evidence to the contrary. Until convincing statistical proof is shown to me, I still believe that the most common use of these lab classrooms (whether in lab mode or as classrooms) is basic word processing, and not the high-end features that differentiate Word 6 from Word 5.1a; after word processing, the most common uses are checking electronic mail and web surfing. As presently configured, with multiple machines served by few 10-megabit lines, the classrooms don't work particularly well for simultaneous web surfing either. When the labs are used as classrooms, the foremost application is word processing, with pick-up and drop-off running a close second. If it can be made reliable and fast, Daedalus InterChange is a very effective teaching application, one I use most frequently in my own classes. Student checking of electronic mail is generally not a part of the instructor's planned activities, usually quite the contrary: in the many labs where screens are between students and instructor, there is a significant loss of student attention due to e-mail and web surfing. Even composition instructors who integrate an Internet component into their class activities have assured me (in a mid-November meeting of English lecturers) that they would gladly do away with in-class e-mail and net traffic in exchange for faster and more reliable server performance. The fact that network activity in one of the upstairs Olson rooms (or in other parts of Surge IV) compromises server performance in a classroom is regrettable and inescapable until and unless significant changes occur. I recognize that Tim and Pete are taking steps to reduce the interference the classrooms suffer from their connection to the rest of the campus network. However, this seems to accept the dominant assumption that the classrooms need to be connected to the network at all. Classrooms must quit trying to be all things to all users. I believe we should take a step backwards, and dedicate the classrooms to a specific task. Class activities should have the priority, not out-of-class use, and isolating the classrooms from the rest of the net may be the best solution. How fast would the classroom network run if it did not have Internet traffic clogging the pipes, and if each machine's hard drive had the key software installed to avoid dependence on the server? Just because a room contains computers shouldn't mean that those computers are all connected to the Internet. Students who want to check e-mail will have to go elsewhere, just as students who wanted to use a telephone would have expected to find telephones in the library's typewriter rooms. Your mission, Tim and Pete, should you choose to accept it I would like to know as soon as possible what would be involved in getting such a classroom up and running, even on an experimental basis. IT-Labs is the entity on campus with the most experience and expertise in this sort of design and execution, so I go to you first. But I want to stress that you give full consideration to the concept, and not make fragmentary objections to logistical details. For example, security considerations for the hard drives alone, or software copyright issues, or changeover costs, are all legitimate points of discussion, but they are not argument-stoppers or project-killers. Furthermore, I need to know about the assumptions that drive your objections; if a particular improvement would necessitate the purchase of another server or the bashing of a couple of holes through a wall, don't let the debate get stuck there. My experiences with Katherine Olvera on the issue of classroom layout--her assertion that she never had any idea that the tightness of the computer spacing was an issue to us--makes me extremely careful about what is said and unsaid; hence the specificity of these questions and the specificity of your response. It's important to stress that all conceptions of labs / classrooms are by their nature compromises of cost, performance, reliability, maintenance, and pedagogical effectiveness, and that the current compromise is not working. The current configurations are not effective teaching tools--in fact, in many cases they are making teaching and recruitment difficult. I look forward to working out how we can make teaching the priority, at least in some rooms. At some point soon we should design and build a classroom with the classroom concerns driving physical layout, networking configuration, and software. Emancipating the classroom from the attempt to serve all the needs of all the students would mean decisions about software, for example, would be made not from the upgrade-obsessed mindset, but from the standpoint of effective pedagogy and long-term efficiency. That may be in the future, but at least we can dream!