Earning the Stamp of Approval: How
to Achieve Optimal Usability
by Susan Makar
Since the NIST Virtual Library (NVL) was first launched in 1994, scientists
and engineers at the National Institute of Standards and Technology have relied
on it for their research needs. The NVL offers access to valuable scientific
databases and electronic journals, and the Web site always received positive
feedback from the scientific researchers who used it. But by the new millennium,
the NIST Information Services Division (ISD) realized it was time to set a
new standard for the NVL. Our customers wanted the same great resources but
a better navigation system for finding information. As an ISD staff member,
it was my job to help ensure that the changes to the NVL reflected the needs
and desires of NIST researchers.
NIST is a nonregulatory federal agency within the U.S. Department of Commerce.
Its mission is to develop and promote measurements, standards, and technology
to enhance productivity, facilitate trade, and improve the quality of life.
The Research Library supports the research activities of the NIST scientific
and technological community. The library, which resides on the main NIST campus
in Gaithersburg, Md., serves a population of 2,700 and has a collection of
more than 200,000 items. As a research consultant and reference librarian,
I was a member of the NVL Redesign Team, which undertook the mission of redesigning
our Web site in the spring of 2001.
The NVL's Focus on Success
To publicize this major redesign of the NVL, we borrowed a playful and appropriate
theme from the popular TV series Star Trek: The Next Generation. We
designed a poster to showcase the "Next Generation NVL" as it was transformed
from the old to the new in Star-Trek-like fashion. Ours is a research institution
of physicists, chemists, and engineers who could easily relate to Star Trek,
transporters, and anything having to do with quantum science.
When word went out that the NVL was to undergo a major redesign, there was
concern among some NIST scientists that the NVL might lose some of its value
or that its resources might become difficult to find. "Don't mess with the
NVL!" was how one customer put it. To make sure that we didn't stray from what
was good and worked well on the NVL, ISD staff conducted focus groups to learn
what our customers liked about the NVL and what they wanted to see changed.
Later on, we conducted Web usability testing on the redesigned site to ensure
that we had indeed made the right changes. We knew from the beginning that,
although we'd be testing users' ability to navigate and find information on
our redesigned site, our users would also be comparing the new site's functionality
and usability with the old one's. So it would be important for us to prepare
carefully and earn our users' approval!
Although there are many elements and phases to redesigning a large-scale
Web site such as the NVL, in this article I'll be focusing on Web site usability
and the testing that we did to make sure our Web pages allow our customers
to navigate the site easily and to find the information they want.
Listening to Our Customers
When the Web site was originally designed in 1994, frames were popular but
presented some usability problems. On the left side of the page was a navigation
bar made up of image buttons (see Figure 1). The main body of the page contained
content that focused on what was new on the NVL. No one kept up with this What's
New section, and eventually the lack of attention caused it to become dated.
This section, and others like it, also lacked a well-defined system for archiving.
To redesign our Web pages, we needed to make sure that we knew what makes
a site "usable." Close to the time of the NVL redesign project, Web sites throughout
our parent organization, Technology Services, were also undergoing redesigns.
As a part of the learning process, Web usability training was arranged for
a small group of Technology Services staff members. I was one of about 30 people
who attended the 2-day, on-site training session, Web Sites that Work: Designing
with Your Eyes Open, in January 2000. In February 2001, my colleagues and I
attended a 1-day refresher course in Web usability training.
Both workshops, conducted by Jared Spool of User Interface Engineering, were
lecture/discussion sessions emphasizing audience interaction and hands-on participation.
During these courses we learned how to design and conduct Web usability tests.
We also learned how users follow the "scent of information" to navigate sites
and how to design links that increase that scent. Spool explained that links
on a Web site give off "scent," or provide visual or verbal cues that users
follow. If a link's scent is strong, customers will more quickly locate the
information they want.
"When they heard our site would undergo a major redesign,
some of our customers were worried that it would become harder to use."
Before we began any redesign efforts, we wanted to learn what our customers
liked about the NVL and what changes they wanted to see. To create a more customer-oriented
Web site, we conducted two focus group sessions in April 2001. The focus group
results showed that customers were generally happy with the many resources
accessible through the NVL but that they wanted more. They wanted specific
databases and always more e-journalsthe more full-text information, the
better. They also wanted a streamlined process to get to the information. In
essence, they wanted one-stop shopping. The site's navigation also needed attention,
and at least one or two participants said that the site was not very attractive.
So, we prepared to set a new standard!
Based on our understanding of site usability from the workshops and the results
of our focus group sessions, we began to design a new NVL home page in the
summer of 2001. The new home page linked to most of our top-level content,
to achieve the "one-stop shopping" that users had requested. We placed the
navigation bar on the right-hand side of our secondary pages, the same side
as the mouse for our right-handed user majority. This was an experiment, in
response to some users' frustration with the old NVL's left-side navigation
bar. (Research at the National Cancer Institute has shown that users click
with greater efficiency on topics that are located on the right, closer to
the scroll bar.) At this stage, we also carefully selected names or labels
for the links. A subgroup of the NVL Redesign Team tackled this project and
came up with an overall design similar to our current home page (see Figure
The Usability Test: Our Users Inspect the New Site
Once we had redesigned the site and moved content into a new content management
system, we were ready to test the new NVL to see how well it worked. Our Web
usability training had given us the background and foundation of knowledge
to create the right testing conditions for our Web site.
To further my understanding, I attended the ASIS&T 2002 Information Architecture
Summit held in March in Portland, Ore. I found one particular program by Jason
Withrow to be especially useful in helping design Web usability tests. His
presentation, Choosing the Best Path: Techniques for Assessing and Improving
Scent, focused on developing good information scent on Web sites. There, I
also learned about quantitative and qualitative user test metrics, which I'll
get to later in this article.
"Our Web usability testing demonstrated that, overall, the
new Web site was well-organized and easy to navigate."
Our first step in doing Web usability testing was designing the list of questions
to ask our subjects. Our aim was to keep the usability testing session shortno
more than half an hourso we included just nine questions. As in any research
institution, our scientists are busy and we wanted to make participation as
appealing and as simple as possible.
Here is a list of some of the questions that we asked our test subjects.
Although the questions are unique to the information found on our Web site,
we tried to focus on some of our more popular products and services.
What journal articles have been published on quantum computing
over the last 3 years?
Does the NIST Research Library provide electronic access to
the journal High Performance Polymers?
Find the full text of the following patent: Nanoelectrode arrays,
What is the exchange rate converting 2,000 euros into U.S.
What is the proper format for citing a book in the Journal
of Research of NIST?
What meetings are being held at NIST today?
Whom do I contact for help with printing my NIST Special Publication?
We conducted usability test sessions with five NIST scientists. We selected
the five subjects from a list of over 30 possible participants. We had learned
from Jared Spool that we did not have to have a large set of test subjects
to learn if our redesign efforts had been successful. Working from the longer
list of possible participants, it didn't take long to find five individuals
who were willing to test the new NVL. Most seemed eager to see the changes
Although our focus group sessions had pulled from a mix of NIST staff members,
including support staff and researchers, participants in our smaller Web usability
test group were scientists, chosen because NIST scientists and researchers
are the primary customers and users of our Web site. From the list of 30, I
selected individuals whom I knew would be supportive of our efforts to improve
the Web site. I knew most of these people to be open in both their praise and
criticism of our site. Not all of them were frequent users, which offered a
We gave participants the choice of holding the usability testing in their
offices or in the library. Three of the five chose their offices while the
other two came to the library, where they were located in a private area away
from most distractions.
How Users Rated Our Site
We used quantitative and qualitative test metrics to measure the success
of our redesign project. In quantitative terms, we measured path directness,
path frequency, and time and completion rates. To measure path directness,
we determined the "optimal path" to the right answer and the number of clicks
it should take to reach that point. We then calculated the number of clicks
it took our test subjects to reach the destination and compared that number
to the optimal path. We also looked at what paths were chosen most frequently,
how long users spent on each question, and whether they found the appropriate
answers. Qualitative metrics included recording verbal user comments and looking
for signs of indecision or indications of frustration or confusion. The form
that I used to record test results is represented in Figure 3.
How quickly did our usability test subjects find answers to the questions?
The speed of locating answers was, in most cases, quite fast. I found that
either the subjects were able to quickly reach their destinationsor they
were completely stumped and were never able to find the answers. Our results
showed that some questions were answered in as little as 10 seconds, while
in a couple of cases no answer was found and the task ended without success
after 5 minutes.
One question had all but one person stumped. Subjects were asked to convert
euros to dollars, which required an exchange rate converter.At the time the
usability testing was being conducted, the answer to this question could be
found under the topic link labeledDictionaries, ETC. We thought this category
nicely described a section where customers could find a wide range ofWeb resources
such as dictionaries, a thesaurus, and phone directories. However, this was
not apparent to our test subjects. We quickly learned that we needed to find
another label for this category of resources. Reference Shelf was a natural.
Our Web usability testing demonstrated that, overall, the new Web site was
well-organized and easy to navigate. In most situations, our customers seemed
to be able to move around the site quickly and find needed information easily.
Even our experimental right-hand navigation bar seemed to work well. The testing
did uncover a few problems, mainly with the labeling of links that didn't precisely
convey content. Based on these results, we renamed some of the labels and changed
the locations of some items on the site. But we were pleased that for the most
part the test results indicated that our new design was a success.
Important Lessons Learned
One of the valuable lessons that we learned from the process of redesigning
our Web site was the importance of customer input. It seems only logical that
to design a more customer-focused Web site, you need to know your customers,
what they want, and how they find information on the Web. Their direct involvement
from the beginning to the end was critical. We believe that a large part of
our success was due to our careful use of focus groups before we began redesigning
the NVL and to the Web usability testing we performed near the end of the process,
which redirected some of our work to make the final product even better. In
the end, we all agreed that we'd won our customers' stamp of approval at NISTthanks
to careful planning and our customers' valuable input!
Fleming, Jennifer. Web Navigation: Designing the User Experience.
Sebastopol, CA: O'Reilly & Associates, Inc., 1998.
Morville, Peter. Information Architecture & Organizational Strategy. ASIS&T
Annual Meeting. Washington, DC, November 2001.
National Cancer Institute. Research-Based Web Design and Usability
Rosenfeld, Louis, and Morville, Peter. Information Architecture
for the World Wide Web. Cambridge; Sebastopol, CA: O'Reilly & Associates,
Inc., 1998. Now available in its 2nd edition, 2002.
Spool, Jared. Web Sites that Work: Designing with Your Eyes Open. User
Interface Engineering, 2000. Workshop manual used during NIST on-site
Spool, Jared, Tara Scanlon, Will Schroeder, Carolyn Snyder, and Terri
DeAngelo. Web Site Usability: A Designer's Guide. San Francisco,
CA: Morgan Kaufmann Publishers, Inc., 1999.
Withrow, Jason. Choosing the Best Path: Techniques for Assessing and
Improving Information Scent. ASIS&T 2002 Information Architecture
Summit, Baltimore, MD, March 2002.
Susan Makar is a research consultant and reference librarian in the
NIST Research Library at the National Institute of Standards and Technology
in Gaithersburg, Md. She holds an M.A. in library and information science
from the University of Iowa. As Webmaster of the NIST Virtual Library,
she is responsible for the overall design and management of the NVL. Her
e-mail address is firstname.lastname@example.org.