COMP 440/COMP 557

Artificial Intelligence

Pacwar Report Guidelines

All material on this website is © Devika Subramanian, 2007-2016. Please request permission from devika@rice.edu for use of the material, and please acknowledge this site in your material.

This material was developed with Professor Linda Driskill under the auspices of the Cain Project at Rice University.

Checklist for the comp440 Pacwar Report

Writing a killer report for the Pacwar Project

Introduction

The report you write for Pacwar problem differs from typical science and engineering reports. Those reports feature results rather than the process of discovery. You’re already well aware that the killer gene assignment is not the standard plug-and-chug routine problem; instead, you are faced with a problem that has never been solved. You are placed in the same position as thousands of other scientists, and you may succeed. The problem challenges you to get the most you can from all the resources at your disposal:

The problem puts you at the boundary of the human/machine interface and challenges you to learn as much as you can from the experience by keeping a log and then interrogating what you record. It’s true that this report must be turned in to the instructor, but you are the report’s principal audience and beneficiary. The best outcome will be (1) insight into your problem-solving processes and (2) rules of thumb for future efforts. Writing it will help you glean from your experience the meaning of being an AI scientist.

As you know, the challenge of Pacwar continues over fourteen weeks, pitting your solutions against the best efforts of earlier years’ students. This situation makes it difficult to provide you with report examples; student reports written in other semesters might bias you toward particular approaches. Since the problem has not been solved, none of these approaches is necessarily an ideal model; none of the reports describes a perfect solution to the problem. However, some are much better than others in setting out the assumptions, approaches, dead ends, and wisdom of specific partnerships. This guide will show you how to write a valuable report, but it won’t give specific examples of problem solving from the Pacwar case; we instead will draw on analogies and quotes from which specific information has been deleted.

Pacwar: The Heroic Quest

Many years ago the Museum of Fine Arts Houston presented a Star Wars exhibit. We’re going to rely on this classic series of films from your childhood to help you understand the writing you’ll do in this project. If you haven’t seen the films, you can view the exhibit at the National Air and Space Museum or look at many other Star Wars sites on the Web. The films chronicle a classic myth of a hero’s moral, social, physical, and technological development. As he develops as a warrior, Luke Skywalker (the central character) must learn to master the technologies of his society. The movies emphasize that his ability to use technologies requires more than sheer physical skill or rational analysis. Weapons such as the light saber and the fighter craft, like computers, have capabilities that are almost beyond human abilities to control and direct. To be successful and attain the status of Jedi Knight, Luke must learn to trust an intuitive and moral knowledge called “The Force”.

If you go to the Web, you will doubtless be struck with the artificiality of the costumes, models, and sets. It is, YES, it is hokey. And however fictive its nomenclature-its planets, tribes, and devices, it depicts our sense that problem solving requires a mysterious blending of human modalities. To become expert requires lots of physical practice, intellectual analysis, AND reflection—just like the Pacwar project. So while we aren’t offering you a Jedi Knight pin or a light saber for your work, we want you to think about the writing component as a chance to enjoy an unusually high level of self-awareness, a kind of screen that enables you to capture some of the otherwise fleeting insights into managing the time, the talent, and the physical resources of your tasks. It’s fine to see yourself in a heroic role and fashion a heroic narrative meanwhile.

Don’t Go Here: How Not to Write the Report

As you’ll see from this “shell” of a low-scoring report, a blind recollection of actions separated from any analysis of why those actions were taken or why they produced specific results would be perfectly useless as a guide in the future: 

How can we even begin to describe how we approached this assignment? We used a number of methods to achieve our results. Our log details are fairly scarce, simply because we're just not that organized, but we're sure you'll be able to understand the time and quality of work we put into PacWar by taking a look at the code in <URL>

The most obvious places to start were, of course, random genes and the four genes that contain the same digit for all fifty digits. In playing, I found out that <gene deleted> performs best. I used the following method to obtain a slightly better gene: I used this method for a few days. Although this got me off to a good start, this method lasted only so long. After a while I needed to move on to something more complex. I now decided to use a GA to get me a bit further than where I was. It went like this: .

As you can see, this report did not reveal why anything was done, why any particular strategy worked or failed, or what the overall shape of the problem-solving process was. The metaphor of the journey—starting out, going on in time, and making a change to “move on”-dominates the report and limits its usefulness. The writer expects the reader to make sense out of the code even though the writer has not done so. Equally useless is a high-level, abstract description that contains no details and no evidence to test the generalizations: 

To ensure a good amount of cross breeding so as to counter the effects of specialization, it seems likely that any given gene pool is very susceptible to the problem of -----. That is, a pool killer of a specific class will likely fail to and thus our purpose is to ------. The Method: Each phase of our algorithm goes like this: a b c d and so on . . . . With this scheme we introduce a significant amount of ----- in the gene pool and thus, we should expect to end up with a number of good genes at the end of the night. . ... The winner, in theory, should be a superb gene. Unfortunately, this grand vision of never came to fruition because of the frustration known as ------. The implementation called for a number of such and such, and, probably due to lack of sleep in the previous nights, I was unable to code the thing properly and it kept crashing. But so grand was the vision that I felt it good to report it anyway.

That neither of these reports produced a high contender is not the principal failure of the reports. The real problem is that neither of these teams learned much about their own problem solving, about the way to approach very large problems with the aid of a computer, or about working with a partner to share work and test one another’s ideas. The first writer probably realizes being organized and keeping a record would be helpful; the second may have learned that grand schemes only work if they’re compatible with one’s skill in implementation. But these are superficial gains in knowledge. These excerpts come from real reports turned in to Pacwar only to earn a poor grade. Don’t go there.

What Yoda Would Have Luke Learn

Having glimpsed failures, a developing hero becomes ready for some direct instruction. Here is some for you: To gain the full benefit of the project, you must use your logs to pick out three kinds of evidence that may not be written down in much detail:

It is very likely that your logs are action records, not reflections. Here’s a sample action-only record:

- 2/27: Downloaded all relevant files from comp440 and began devising genetic algorithm 
- 3/3: Started running the first executable on gg. 
- 3/4 Modified macros in program then left it running for a while 
- 3/11: First tested genes.

If your logs look like this, review them now and overwrite the margins, add large post-its, or add separate pages with reflections. For example, the person who wrote this log should ask himself, why did we decide to create a GA first? What were the criteria we were using? How did we intend to proceed? How did we think we would know when we had exhausted the capability of the GA? Here are some more questions for interrogating your short log entries:

Here are a couple of sample comments that you might add to reveal purpose:

- "Modified macros in program so as to run more search paths" 
- "Manually added some of the best genes from Owlnet to linux process to increase the diversity of stagnant search paths"

Like finding patterns in the genes, you would like to find large problem-solving patterns and their implications in your logs and incorporate these into your report. To detect these patterns, you’ll need to create a story with a bigger plot, not just an episodic series of actions and outcomes. The epic context for this narrative involves you and your partner working at the human/machine interface, with a computer for a sidekick. Which missions can the computer do best? How long would it take the computer to undertake the most demanding mission you would like to give it? What will you do if the computer can’t do what you want?

You may know that as an early step to tackling the problem of having computers play chess, there were efforts to program the computer to play a form of checkers. In order to make the whole task more feasible, the checker board was simplified. You may want to think about which tasks have to be modified to adjust for the computer’s or for your own limitations as a processor. You might want to note places where you would have done more if you could have had more time, things you would try next time, and problems with software engineering. It’s also good to note your attitudes and your partner’s, too.

As you look for big patterns of actions in your logs, keep in mind some of the structures that others have found useful for explaining their experience. We’ve taken out the solution-relevant data and concept words. Think heroic quest. Here are three of these macro-plots.

Sample lines: “We thought of our first phase two optimization before we even realized that we were done with phase one: we noticed that we had very and we wondered _____ . To answer that question, we wrote a small program that __

Paragraph or Section Structure in Good Reports

The best reports had a predictable pattern in individual sections or paragraphs:

Here’s the skeleton of a section demonstrating this pattern: 

The next stop on the way to our gene was ... (action) We ended up scrapping this when ... (outcome) The reason was ... (interpretation) At this point we began to question the wisdom of relying strictly on ____ We're still not sure whether moving away from ____ was a good idea, but it was an important step for us and gave us lots of success. Perhaps all we did was climb to a small local maximum, but we still think that our next approach was very helpful to us. (lessons learned)

Style in the heroic narrative (Don’t freeze)

As you might expect, the heroic quest brings knights and scientists to a confrontation with themselves. You must find a style that is comfortable for you and that provides you and your professor with a clear picture of your experience with the project. However, clarity is not a function of formal conventions, although it is closely correlated with good syntax and correct grammar; you can write this report in a variety of styles and be successful. In other words, you can write in the first person singular, the first person plural, or in the third person and it won’t make any significant difference. You can choose a conversational tone, use contractions, and informal expressions. Avoid slang, however, because you can’t count on the faculty knowing “in phrases.” Indeed, a good many colloquialisms, repeated too often, have no meaning at all. Specificity is wanted in this report, but priggish formalism is not valued. Listen in to some very different, but very competent and successful report writers:

So assume that you have freedom to write in a style that suits your taste, but remember that a thoughtless style will reveal little to you or to anyone else. Your tone can be confident, stoic, or gleeful, but it has to be clear and precise.