All that follows is a supplement to your reading of the chapters on design (8, 11, 12, and 13) in the textbook, Hall’s Web-Based Training Cookbook, Wiley 1997.

Those who want to pursue the matter further might review chapters 3, 4, and 5 ("Identity Design," "information Design," and "Interactivity Design") in Clement Mok’s Designing Business, Adobe Press 1996.

Much of discussion of iterative testing is drawn from Nielsen and Mack’s Usability Inspection Methods, Wiley 1994.

All of the information below (and in Mok’s book) should be read in light of the fact that not all of it applies to all of the media choices available in distance learning. Most of what follows assumes the use of the Web as the framework and vehicle for presenting both information and other media. In cases where video or printed text are the primary vehicles, the philosophy presented below will apply but all the details will not.

The process (as described in the first unit on planning

We all know that the process from idea to finished digital product is continuous and recursive. But for our purposes momentarily assume the process consists of discrete tasks organized as follows:

• planning which yields a decision to engage in DL, to create a particular course or program, to market it to (or require it of) a specific audience, to use a specific technology or technologies, commit a specific amount of money, time, staff, and facilities, assume a specific set of educational and/or financial outcomes, and assess in a specific way against specific standards. The planning process yields (1) a strategic plan, (2) a business plan, and (3) a marketing plan.

• Design assumes that the planning process has presented strategic, business, and marketing plans. Based on this information and further information generated during the design process, the design stage develops:

specific content

information modules

sequences of information

media selection

the interface

plans for student assessment

plans for course management

The design plan is developed through an iterative process based in based in:

1. definition of client goals and constraints

2. detailed user analysis

3. designer’s initial creative process

4. tests of designs with clients, users, and peer designers

5. the final articulation of the design in:

- a proposal

- sets of storyboards (for clients and programmers)

- an interactivity flowchart

- voiceover and shooting scripts.

Following approval of the design, the process continues with:

• production which yields media and integration, loading to servers and/or creations of CD’s.

• assessment which yields formative, iterative testing during the design and production process, summative tests at the end of the production process, and continuing formal or informal testing after implementation

• implementation which yields

• ongoing maintenance which involves resolving technical problems, accommodating technology upgrades, interface modifications, addition and clarification of content, and continuing revision of the on-line testing process.

***********

We must:

1. satisfy the client's immediate needs.

2. identify and satisfy client’s long-range and strategic needs

3. satisfy the user

4. document the result

• clients want (and often need) substantial results for limited cost

• client’s immediate and strategic needs may conflict

• the client situation and environment can be too complex for the designer to fully grasp

• user culture (or more likely cultures) pose barriers for DL

• designers find it difficult to identify user cultures because the designers themselves are ethnocentric

Structured, iterative testing is the key to success.

• the designer’s initial assumptions can be refined

• clients, users, and peer designers can become partners in the process.

1. defining projects with clients

2. defining projects with users

3. an initial creative process in which the designer applies his/her designs skills to the defined problem

4. documenting the initial design in flowcharts, storyboards, and scripts

5. iterative testing of group designs with clients & users, modifying the design and documenting the changes

6. iterative testing of group designs with designers, modifying the design and documenting the changes

7. gaining client approval for final flowcharts, storyboards, and scripts

8. preparing complete set of more detailed storyboards for programmers

• Accomplish the client's stated objective to achieve a given learning result within given time and budget constraints

Ideally, the objective can be fully stated as follows: to teach a defined group of students, a defined content, in a defined period of time with a defined learning outcome (which includes a specific level of skill or knowledge at the end of the course, a specific level of retention for a given duration and a specific level of applicability) at a given cost and with a specific technology.

• Accomplish the implied objectives

accommodate the organization’s technology strategy

address technological barriers

accommodate client's sense of user issues

accommodate client/organization management style

accommodate client's technology awareness level

accommodate client /organization success measures

• No knowledge matchup between client & designer

• A set of structured discussions between client and designer reflected in:

an assumed content

a look and feel

a set of information

a set of perspectives

a list of key images and/or words

• as well as other information such as:

a set of benchmarks

assumed installation & maintenance costs

specific assessment methods

interim deadlines on the project

assumed ROI

assumed technology requirements

time allowed for formal usability testing and revision

the client's reading or experience-based expectations for DL

the client's professional assumptions about DL

This involves considerable data gathering. Much of the information will come directly from the client. But the designer should also identify and talk to other key individuals including:

• corporate MIS for course delivery issues and planned technology enhancements that may affect the course

• the outside ad agency for look and feel issues in the organization

• the professional organizations in the course content area (sales, H/R, academic discipline)

Sources of information about the above issues: (See attached biblio)

• professional journals

• journals of industry-specific data

• US demographic studies

• legal reviews for non-lawyers in such areas as intellectual property, liability, and labor-management

• published research studies in corporate use of newmedia (i.e., Project 2000)

• business school journals for case studies and strategic impact

• current books

• industry or company task benchmarks

• client H/R data

• internal documents

The task

Accommodate the users’ cultural and cognitive issues in a way that enhances the learning process and the students’ satisfaction with the DL delivery mode.

• Every organization poses a different set of cultural issues based in the organization’s own culture and the sub-cultures that exist within the organization.

• Each individual learner poses a unique set of cognitive issues which govern how he/she perceives, relates, stores, attaches significance to, and applies information.

• extensive audience analysis using individual interviews, focus groups, surveys, informants

• review of the professional literature describing cultures of similar organizations

• extensive structured iterative testing of the storyboards and flowcharts with individual and groups, sometimes using peer designers to assist the individuals and groups in articulating issues

Organizations, whether schools, undergraduate student bodies, members of a profession, a company, or one of the company’s departments, typically have an organizational culture. It defines what they are and distinguishes them from similar organizations. Evidence of the culture will be found in its literature, whether corporate reports, professional journals, college bulletins, or student newspapers. The culture typically defines where the organization sees itself going and how it intends to get there. The goal may be a new form of public service, the use of a new tool to achieve that end, or a particular manner in which the ends is achieved. It may be penetration of a new market, a acceptance of risk, or an assurance of prudence.

To a greater or lesser degree, depending on the strength and pervasiveness of the culture, the culture will define how members of the organization see any new activity such as DL. When the culture seems to immediately relate to either DL or the course content or course goal, designers can achieve student motivation and commitment by reinforcing this connection. When the culture does not immediately relate, designers must create or articulate a relationship.

In addition to its overall culture, organizations contain sub-cultures based on any one of a number of factors described in more detail below. Like the broader culture, these affect user perception and motivation. Unlike the broader culture, these cultures are not easy to find. They are typically known only to their own members and exist because their members see themselves as margialized within the organization.

In the past we have seen such cultures as primarily defined by gender and race. But to an increasing degree we find them in age groupings, regional origin, ethnicity, disability, and disciplinary background.

Not all of these cultures will exist an any organization. But where such cultures do exist, they define how people respond to training ("Why should I learn this when only younger people will get promoted") or to visual media ("Look at the people in the video, all engineers, nobody from sales!")

These tend to be much more difficult to define usefully. At this stage, suffice it to say that:

• a constructivist approach has merit in virtually every situation

• the constructivist approach seems unknown in much of education and is somewhat controversial in industry

• one of the potential strengths of DL is its potential to use multiple media and multiple presentation sequences to facilitate learning by many different cognitive perspectives

1. With peer designers

• Does project design meet/exceed current best practice?

• How does this design relate to similar projects now in use or forecasted?

• What are two examples of best practice (i.e., award-winning, well-reviewed) in this field or industry? How does this package incorporate their features?

• What are two examples of cutting edge projects? Are any of their features incorporated here? If not, Why not?

• How this specific design can be improved?

- Check modules, text/visual content, navigation system, specific links with a "cognitive walkthrough."

- Check the design against "consistency guidelines."

The "cognitive walkthrough"

1. Prepare sample user task on paper.

2. Define the user subgroup in as much detail as possible.

3. Define the context of use.

4. Select three peers (fellow design experts).

5. Ask the peers, together or separately, to walk through the task aloud, commenting on whether the user will

- recognize choices,

- see the implications of specific choices,

- be able to recognize where he/she is located in the navigation system

Ask the peers also to comment on:

- conformity to user interface conventions

- needed documentation

- ability to back up or cancel a choice

Results of assessments with peers

- major design changes to accommodate best practice

- modifications to insure ease of use

- the designer’s increased ability to explain and justify the design

2. With peer designers and users

Create a situation in which your peers (human factors or design professionals) work with your users to validate and interpret their reactions to the design a "pluralistic assessment"

The pluralistic assessment

1. users are given the paper-based design documents and a scenario

2. professionals are advised of key cultural factors (i.e., collaborative tendencies, risk aversion, etc.) in the user group

3. users are asked to move through the package, explaining their choices at each navigation step

4. professionals and users discuss choices, validating the users' responses and surfacing the reasoning behind them

5. designer records responses and the reasoning process

6. designer creates sets of links to accommodate each of the larger blocks of users and designs a navigation system that facilitates the use of those links

3. Question sessions with users

Circulate flowcharts & storyboards user groups; use their comments to continually clarify their needs, to minimize cultural fractures and reinforce cultural bonds

Techniques for testing flowcharts

- open-ended questions for information blocks and sub-blocks:

What's missing here?

What's wrong with this division?

What information have we left out?

If you had to divide this information, how would you do it

results:

a number of ways of dividing the material

each way preferred by a given percentage of the users

- open-ended questions to define the names/icons for blocks and sub-blocks

What would you call this block of information?

What would others call it?

What would people outside your group call it?

4. Usability tests to define best media to present information in each block and sub-block

user comfort level

user speed

user retention

user application

5. cultural assessments of storyboards (and scripts and music)

- open ended questions with emphasis on cultural fractures and bonds

What would these people look like (age, dress, gender, race, ethnicity) in your organization?

Who should be in this picture? Why?

What is going on in the conversation pictured in this storyboard?

How would you change this picture to make it more realistic?

Ideal results of the user testing process:

1. a description of the three largest groups of users as defined by demographics and cognition

"Group 1 blocks information as follows, names the blocks and sub-blocks as follows, prefers one of the three following media."

"Group 2 blocks information as follows, names the block and sub-blocks as follows ; all members prefer one of the three information media used by group 1."

2. a description of the blocks, sub-blocks, and media choices necessary to meet all of the groups' needs

3. a set of images (and scripts) defining the look needed to reinforce the group culture and the looks which must be avoided.

Design

Peter A. Gloor's Elements of Hypermedia Design : Techniques for Navigation & Visualization in Cyberspace, Springer Verlag, 1966 is heavy going at times but is thorough review of what we know about the process of mapping and linking information.

An excellent and not at all dated book on interactivity is Ray Kristof, Amy Satran's Interactivity by Design: Creating & Communicating With New Media, Hayden Books, 1995.

Brenda Laurel's Computers As Theatre is a standard work on how users interact with the computer. It is still available as a reprint from Addison-Wesley.

Douglas Schuler and Aki Naioka's Participatory Design: Principles and Practice , Lawrence Earlbaum, 1993, does not specifically address newmedia, but it addresses in great detail how one gets users involved in the design process.

Fred Moody’s I Sing the Body Electronic: A Year with Microsoft on the Electronic Frontier, Viking, 1995, is good study of what happens when the design process goes wrong.

User Analysis

Ethnography can be generally defined as the qualitative (as opposed to quantitative) study of the behavior of groups. When combined with the various types of quantitative analysis, it serves as a crucial tool in understanding the users act within specific environments. Useful resources include:

Hammersly and Atkinson's Ethnography: Principles in Practice (2nd edition), Routledge, 1993, is a clear introduction and review of method for the non-specialist.

Two excellent studies of applications of ethnography to technological environments are:

Julian Orr, Talking About Machines: The Ethnography of a Modern Job, Cornell, 1996, an ethnographic study of the world of repair technicians at Xerox Corporation.

Frank Dubinskas, ed., Making Time: Ethnographies of High-Technology Organizations, Temple Univ. Press, 1988, which deals with the worlds of high energy physics, radiology, and genetic engineering.

Sherry Turkle's Identity in the Age of the Internet, Touchstone Books, is a well-known and well-regarded ethnographic study of Internet populations.

Michael H. Agar's The Professional Stranger : An Informal Introduction to Ethnography, Academic Press, 1996 is a revision of an earlier, well regarded work. It's readable and up-to date.

The standard work on focus groups is Richard A. Krueger's Focus Groups: A Practical Guide for Applied Research, Sage Publications, 1994.

A good book on creating user surveys is Floyd Fowler's Improving Survey Questions: Design and Evaluation , Sage Publications, 1995.

A forthcoming book on the use of computers to aid the disabled in the workplace is by Alan Roulstone is Enabling Technology : Disabled People, Work, and New Technology, Open University Press, 1999.

Gavriel Salvendy ed., Social, Ergonomic, and Stress Aspects of Work with Computers, (The proceedings of the 2nd --1987-- International Conference on Human Computer Interaction, is still in large part valid.

The most accessible detailed demographic studies are available in the US Government Documents collections on the Web. Although a bit dated (generally 1995) they are still useful in a variety of analyses.

www.census.gov/population/www/pop-profile/toc.html

provides demographic breakdowns, location, education, income

www.fedstats.gov/

provides detailed data on age, education, gender, race, ethnicity in labor force, disability, employment statistics with detail down to the county level.

A good and very detailed source of demographic information on Web users is

Valuable context for such information, including a discussion of the data collection methods used by the various reporters, is available from Project 2000, an ongoing study of internet commerce by the Owen School of Management at Vanderbilt University

The key problems in the use of cultural analysis at the moment involve (1) expanding the term "culture" beyond race and gender and (2) finding tools for effective cultural analysis.

Two good current studies that engage both issues are:

R. Roosevelt Thomas, Beyond Race and Gender: Unleashing the Power of Your Total Work Force, Managing Diversity, AMACOM, 1992.

George Henderson, Cultural Diversity in the Workplace: Issues and Strategies, Quorum Books, 1994

Managing Multimedia by Elaine England, Andrew Finney, Andy Finney, Addison-Wesley, 1996, is an excellent book covering such issues as client relations, scheduling, and budgeting.

Ron Goldberg's Multimedia Producer's Bible : Managing Projects and Teams, IDG Books Worldwide, 1996, includes some sections on software that are already a bit dated. The rest, covering a range of issues from intellectual property to team management, is excellent.

Clement Mok’s Designing Business (Adobe Press, 1996), is an excellent book, widely reviewed in the business press and probably the best-known newmedia design book in the corporate community.

Avoiding Potential Pitfalls

Anyone planning a DL application gives some thought to the much publicized risks in its use. Both legal and public relations problems have arisen when employees have accessed, downloaded and displayed within the corporation text and images from sexually explicit or hate sites. In addition, there is a growing and well founded fear of data security risks in allowing access outside the corporate firewall. Some worthwhile books that explore and clarify these risk areas are:

Nancy Tamositis, net.sex , Ziff-Davis, 1995 which both reviews the problem and provides information on where and how sex exists in cyberspace and how one could access it.

Kenneth Dam's edited collection of essays Cryptography's Role in Securing the Information Society, 1996, is a thorough and scholarly effort.

Jean Guisnel's Cyberwars: Espionage on the Internet, 1997, is a popular, readable treatment of the same issue.

J. Dianne Brinson, Mark F. Radcliffe's Multimedia : Law and Business Handbook: A Practical Guide for Developers and Publishers, Ladera Press, 1997 is what the authors describe as a "preventative guide," pointing out potential problems to the non-lawyer.

Context and Competition

A useful benchmarking tool is the Critical Guide to Management Training Videos and Selected Multimedia, 1966, edited by William Ellet and Laura Winig, a publication in the Harvard Business Reference series.

The best book I've seen on newmedia training is Angus Reynolds and Thomas Iwinski's Multimedia Training: Developing Technology-Based Systems 1966, part of the McGraw-Hill Series on Visual Technology. It includes a CD with examples and has a number of extensive case studies.

Thomas Duffy and David Jonassen's Constructivism and the Technology of Instruction, Lawrence Earlbaum, 1992, is a collection of essays and dialogs between training professionals and academics on how a constructivist perspective changes training in general and especially the use of technology in instruction. It's a very thought-provoking book.

Two excellent Web sites on training and executive development that contain extensive material on newmedia and training are

www.muohio.edu/~wheeleba/mgtexdev.htmlx

www.muohio.edu/~wheeleba/centers.htmlx

..

A key issue in Internet sales is financial transaction management for consumers. (Corporate purchasers already have systems in place.) A good book on the current issues and solutions is Peter Wayner's Digital Cash: Commerce on the Net, 1997.

To market the course on the Web, see Margo Komenar's Electronic Marketing, Wiley, 1996. It is much recommended by sales and marketing professionals and includes extensive case studies.

Despite its importance, there is not much available on this subject. A good general study is Jim Sterne's Customer Service on the Internet : Building Relationships, Increasing Loyalty and Staying Competitive, Wiley, 1996. It includes a number of case studies.

The best book on scripting is Timothy Gararnd, Writing for Multimedia: Entertainment, Education, Training, Advertising, and the World Wide Web, Focal Press, 1966.

The bible on iterative testing of software is Jacob Nielsen and Robert Mack's Usability Inspection Methods, a conference based collection of essays, John Wiley, 1994.

Nielson's Usability Engineering, AP Professional 1994, includes a section on "usability assessment beyond testing," an overview of surveys, questionairs, and focus groups.

Metrics, including number of users, use time and frequency, and behavior patterns, can be generated for any computer-mediated technology. The most popular software for such purposes is made by Ipro. For more information see the Web site at:

www.ipro.com/