Evidence Based Library and Information Practice 2010, 5.2

Evidence Based Library and Information Practice

Using Evidence in Practice
Mining E‐mail to Improve Information Literacy Instruction
Margy MacMillan
Instruction Librarian/Associate Professor
Mount Royal University
Calgary, Alberta, Canada
Email: MMacMillan@mtroyal.ca
Received: 29 Mar. 2010

Accepted: 06 Apr. 2010

2010 MacMillian. This is an Open Access article distributed under the terms of the Creative Commons‐
Attribution‐Noncommercial‐Share Alike License 2.5 Canada (http://creativecommons.org/licenses/by‐nc‐
sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original
work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is
redistributed under the same or similar license to this one.

Setting
Mount Royal University (MRU) is an
undergraduate institution with an enrollment
of 8,000 FTE, located in Calgary, Alberta,
Canada. The Library provides a range of
course and program‐integrated instruction in
information literacy (IL) conducted by liaison
librarians in collaboration with faculty in the
disciplines. The Faculty of Communication
Studies currently serves students in three
degree programs, the Bachelor of
Communication in Journalism, Public
Relations, or Information Design, and a
Diploma program in Broadcasting.
The liaison librarian works with students in all
Communication programs, both in the
development and provision of IL classes, and
in the provision of one‐to‐one assistance
through appointments, e‐mail, and regular
reference hours at both the main library
information desk and in the Centre for
Communications learning space. There are IL
classes in most first‐year courses, and

increasingly specialized sessions in later years
that focus on particular resources, course
content and higher‐level concepts in
information literacy. The librarian is known by
the students to be somewhat compulsive
about answering e‐mails, and often responds
during evenings and weekends.
Problem
While there was no defined problem with the
services being provided, the librarian was
curious about the amount and type of e‐mail
questions students were asking. Patterns
within the types of questions being asked, the
programs, levels, courses or assignments that
were generating the questions, or the
individuals asking the questions might reveal
areas where IL might be improved to better
meet student needs. The librarian was also
interested in the time of day students asked
questions as an indication of when students
were working on their assignments.

103

Evidence Based Library and Information Practice 2010, 5.2

Evidence
In consultation with members of MRU’s
Human Research Ethics Board, the author
determined that unless consent was obtained
from those students who e‐mailed the
librarian, no data could be published. As
seeking such consent might unduly constrain
students from contacting the librarian, and
might therefore be detrimental to their studies,
consent was not sought. Therefore, while the
process of the study can be described as an
encouragement to other librarians to mine this
rich supply of data, I cannot share the results.
Beginning in August of 2008, the librarian
collected e‐mail from students and placed it in
a folder. Excluded were e‐mails that were
required, such as assignments that were e‐
mailed to the librarian, and, for the first part of
the year, communications through Facebook.
These last messages were routinely deleted by
the librarian, until it became evident how
many of them were reference questions. In
July 2009, the librarian extracted the file of
student e‐mails into a spreadsheet and coded
it by type, such as search strategy help,
citation questions, requests for appointments,
and help with statistics. She also recorded data
about what might have triggered the
questions, including the program, year of
study, and, where determinable, course and
assignment. This was possible due to the
librarian’s knowledge of the curriculum and of
the students. Information on the time and date
that questions were sent was also part of the
data. Coding a year’s worth of questions
(n=543) took approximately eight hours.
The results regarding the type and timing of
questions would not have surprised any
academic librarian. In fact, results were similar
to larger scale studies carried out to evaluate
e‐mail and chat reference services (Lee, 2004;
Leykam & Perkins, 2008; Rourke & Lupien,
2007). There were interesting patterns in the
levels, courses and assignments that generated
questions. For example, there was a significant
lack of questions from students in the second
year of any Communications program. This
could have been read a number of ways: the

students had learned everything they needed
to in the first year (optimistic but unlikely); the
students had no significant research
assignments (on investigation, not the case); or
the students, who did not meet with the
librarian in as many courses in the second
year, forgot about this resource (sadly, the
most probable cause). In investigating the
assignments required in the second year, the
librarian ascertained that these had not
prompted in‐person consultations either.
Assessing students’ needs by examining only
one form of interaction is problematic. There
may be students who are not comfortable
using e‐mail, some preferring to phone or
meet face‐to‐face. Some may be getting all the
help they need at the information desk, where
only very broad statistics are kept, or from
their peers and instructors. A study like this
can only be used to measure one aspect of
interaction with users, and should be used in
concert with other assessments to see the
whole picture of who is or is not requesting
help, and what kind of help they are
requesting.
Quite apart from evidence about the kinds of
questions students were asking, the study
provided data about the librarian’s work
activities, including how significant an aspect
of teaching the e‐mail communication
represented.
Implementation
In the 2009‐10 academic year, a number of
changes were made to IL delivery based on
the evidence of the e‐mails. The content of
some sessions was updated and clarified in
response to common areas of difficulty.
In response to the problem of ”missing”
second‐year students, the librarian asked for
15 minutes in the main second‐year class of
the degree disciplines, Information Design,
Journalism, and Public Relations, at the
beginning of the Fall semester of 2009. She
used this time to review key materials
students might use for assignments, to
highlight new resources and changes to the

104

Evidence Based Library and Information Practice 2010, 5.2

library website, and mostly to remind the
students that they could ask for assistance. In
each of these “promotional” talks, the librarian
spoke directly of the evidence from e‐mails as
the reason for the session. There was generally
a brief period for questions and answers.
Information from the study also contributed to
a better understanding and a better balancing
of the librarian’s workload. In part because of
data from the study, the librarian reduced the
number of shifts at the main reference desk
and added regular weekly reference hours in
the Centre for Communication studies, where
many of the students do their work.

Further examination of the data might reveal
demographic patterns or individual usage
patterns that might also be enlightening. It
would be interesting, for example, to find out
if students asked the same type of question
more than once, or if questions from
individual students showed development of
skill and grew more complex over time. In the
current study, the librarian is also more careful
about including reference questions from
Facebook, and it may be that over time, those
form a higher proportion of the
communication. Deeper discourse or content
analysis might reveal other aspects of
communication, especially if the study is
replicated over a longer period of time.

Outcome
While the full impact of implementing the
changes will only be evident after reviewing
the current year’s student e‐mails, there has
been some noticeable effect. Students
apparently found the promotional talks in the
second‐year courses helpful, and instructors
certainly did. As a result of discussions with
faculty about the ”missing” second‐year
students, several faculty members have
booked longer sessions on particular aspects
of information resources, or specific skills such
as using statistics, required by assignments in
these courses.
Reflection
The study has been very successful as a means
of satisfying the librarian’s curiosity about an
aspect of her work. It was relatively easy and
efficient to carry out, requiring comparatively
little time to develop rich results. These results
provide evidence where there had been only
suspicion and anecdote, evidence that
provided clear directions for change. The
broad outcomes of those changes have yet to
be discovered, but the process of examining
work on the basis of communications from
students has been useful in and of itself. So
useful, in fact, that the librarian has repeated
and expanded the study for the 2009‐2010
academic year, to include files of e‐mails from
faculty, staff, and other groups, in addition to
students.

The most frustrating aspect of the study is the
difficulty in presenting results. Just asking for
prior consent of everyone who e‐mailed the
librarian would likely reduce the e‐mail, and
potentially disadvantage those who would
otherwise seek help. If consent was requested,
the populations who gave or did not give it
might be different enough to further confound
the data, and certainly it would complicate the
process to the point where the usefulness of
the study would be compromised. However,
while in terms of dissemination this kind of
study has limited worth, it has been very
productive in terms of informing the work of
the librarian at a local level. I would encourage
others to make use of their e‐mail as evidence
for refining practice.

References
Lee, I. J. (2004). Do virtual reference librarians
dream of digital reference questions?:
A qualitative and quantitative analysis
of email and chat reference. Australian
Academic & Research Libraries, 35(2), 95‐
110.
Leykam, A., & Perkins, C. (2008). Is this the
right tool for our library? A look at e‐
mail virtual reference use patterns. The
Reference Librarian, 48(1), 1‐17.
doi:10.1300/J120v48n99_01

105

Evidence Based Library and Information Practice 2010, 5.2

Rourke, L., & Lupien, P. (2007). Out of the
question!...How we are using our
studentsʹ virtual reference questions
to add a personal touch to a virtual
world. Evidence Based Library And
Information Practice, 2(2). Retrieved 2
Apr. 2010, from
http://ejournals.library.ualberta.ca/ind
ex.php/EBLIP/article/view/236/421

106