From the Doc Side : survey

Q3 Documentation Survey Results

Justin Ward — Tue, 03 Nov 2009 20:25:00 GMT

The Q3 documentation survey results have been compiled. Here are some of the key stats:

We received 183 responses.
For our five overall rating categories - Accuracy, Completeness, Easy to Understand, Accessibility, and Usefulness - the two most frequent ratings were a 4 and 5 (out of 5).
Our highest overall rating category was ‘Accuracy’ (mean 4.30).
Our lowest overall rating category was ‘Easy to Understand’ (mean 3.98).
86.3% of survey respondents access documentation often or sometimes.
47.5% of survey respondents prefer to access documentation as user guide PDFs online.

Additionally, we have begun tracking user guide downloads with Google Analytics. This quarter we had 25,145 downloads from our User Guide pages. That number includes our user guides, video tutorials, and print tutorials. The number does not include hits on our hosted BBNC help files, which we track separately by topic.

And last but not least, each quarter we enter the respondents of our survey in a drawing for a $25 Amazon.com gift certificate. The Q3 winner for 2009 was Heather Flynn of Olive Crest Homes and Services for Abused Children. Congratulations, Heather!

To take the survey (and be entered in our 2009 fourth quarter drawing), click here. It takes only a few minutes. To send us feedback about our user guides or help file documentation at any time, email documentationcomments@blackbaud.com.

Documentation Survey Results and Responses

Justin Ward — Wed, 29 Apr 2009 14:44:00 GMT

The 2009 Q1 results for the documentation survey have been compiled. Here is a quick summary of our key findings for Q1:

We received 153 responses in Q1 2009. That, my friends, is a new record. And with all those responses comes a ton of great feedback (more on that in a minute).
For our five overall rating categories - Accuracy, Completeness, Easy to Understand, Accessibility, and Usefulness - the most frequent rating was a 4 or 5 (out of 5).
Our highest overall rating category was Accuracy. (Mean = 4.26)
Our lowest overall rating category was Easy to Understand. (Mean = 3.82)
72.4% of survey respondents were able to find the answer to their question in the documentation the last time they accessed it.
The last time our respondents had a question, we were able to help 62.5% resolve the issue on their own with documentation instead of them having to contact Customer Support.

I like compiling the statistical data and comparing trends from quarter to quarter. Those metrics are important and help us understand our weaknesses and strengths. But my favorite thing about reviewing survey results is reading all the user feedback. You can really get a feel for the struggles and pains users experience when they can't complete tasks. For example, when a user says he can’t run an import because the guide doesn't explain which fields are required, that's great information. We can work with that. It requires no trend analysis. And when we call the user, even those who sound upset or angry in the survey, they are typically appreciative that we responded, and usually have even more input for us. It's a great example of how this survey helps us improve our documentation.

Likewise, you can sense the relief when a user is able to find information he needs and perform his job better. Knowing that the documentation is helping users and doing what it's supposed to be doing is great to hear. It validates a lot of hard work and effort.

Here are some of the responses collected this quarter (names withheld to protect privacy):

"PLEASE expand the documentation for Dashboards! There is not enough information about functionality with Share Dashboards and other descriptions are not detailed enough. I've had to do a lot of trial and error and testing on my own."

"Create a guide to explain Actions and Action Track creation."

"I would recommend that when the user guides are updated that an email be sent out to inform us."

“I really appreciate having the user guides. I go there first. Yes, I am pleased, too, with the Knowledgebase, and the telephone support. However, I will almost always first go to the user guides, because I want to know as much as possible about the topic with which I'm dealing, and the user guides point out features I may either not understand, or features that our organization is not yet using. So, typically do I not only find an answer to my question, but I learn something else I can apply.”

That’s just a sample of the responses we got last quarter. I look forward to reading the survey responses every quarter because I know I’ll learn different things about our users and how they use our documentation. So if you've ever wanted to know if users really do read our documentation, the answer is a resounding yes. We hear from them every day.

To be entered in our 2009 second quarter drawing, complete our online survey. It takes only a few minutes. To send us feedback about our user guides or help file documentation at any time, email documentationcomments@blackbaud.com.

Documentation Survey 3rd Quarter Results

Lindsey Robbins — Mon, 27 Oct 2008 14:37:00 GMT

On October 9th, Justin announced our third quarter documentation survey winner. I thought it might be nice to discuss some of our results from the survey with our blog readers.

A link to our survey appears on the Blackbaud User Guides pages (accessed from the Support menu), this blog, EE/FE/SIS Help files (a test run), and in the support newsletters once each quarter. We received 73 responses in Q3 2008.

Overall Key Findings:

For our five overall rating categories – Accuracy, Completeness, Easy to Understand, Accessibility, and Usefulness – the most frequent rating was a 4 or 5 (out of 5).
Our best overall rating category was Accuracy. Out of 73 responses, 62 rated us 4 or 5.
72.6% of survey respondents were able to find the answer to their question in the documentation the last time they accessed it.

Question: Please give an overall rating for the following aspects of documentation (user guides and help files only) on a scale of 1 (least favorable) to 5 (most favorable)?

Discussion:
While we are pleased our numbers are mostly 4’s and 5’s we are actively looking for opportunities to improve and we even received some great suggestions from the open ended responses on the survey.

Question: Which Blackbaud product were you using when you accessed the documentation (user guides and help files)?

Product	Responses	Percentage
The Raiser's Edge	51	69.9%
The Financial Edge	11	15.1%
The Education Edge	4	5.5%
BSIS	1	1.4%
BBNC	5	6.8%
The Patron Edge	1	1.4%
The Information Edge	0	0.0%
BBDM	0	0.0%
ResearchPoint	0	0.0%

Discussion: While we are always pleased with the number of responses for The Raiser’s Edge, we are looking for ways to improve the amount of feedback we receive for our other products. While we know we don’t have as many users on The Patron Edge as The Raiser's Edge, we value each user’s feedback equally.

Question: The last time you had a question, did you find the answer in the documentation (user guides and help files)?

Discussion: We are happy to have so many “yes” answers to this question. We hope it means you find your answer quickly and get back to completing your tasks. However, we also see that 27.4% of you did not find your answer. This question is one of our critical ones. We’d love for to you always find your answer. Our numbers have been improving in this category but there is always more room for improvement.

Here are some sample user comments and suggestions pulled from the survey (only initials to protect the privacy of our clients):

User guides are enormous. Small, more streamlined versions, maybe for more discrete tasks would be helpful. I frequently find myself selectively printing only the relevant sections of a guide.
TH

The user guides are easy to understand and are helpful. I like being able to find an answer without having to call customer support.
JH

Love the support offered by Blackbaud, especially easy access to the User Guides - don't have to buy a hard copy, and don't have to deal with free documentation that is purposely incomplete in effort to get me to buy the manual! Good job Blackbaud. Always happy to recommend your company.
AB

Your User Guides are very extensive and frequently help me answer my questions. However,
sometimes when searching I get results that are not dealing with the program I specified.
JF

I especially like the guides when I want an initial overview of the module. They are well organized and explain the technical aspects very clearly.
MT

Discussion: I think the open ended comments are my favorite part of the survey. It's not easy to quantify the information but hearing exactly what you think is critical. And, we take what you say to heart. If you allow us too, we try to follow up with as many people as possible especially when they offer up suggestions, areas for improvements, or mention problems. For example, to the user who mentioned the guides are enormous we would try to see if it would have been better for the user to search for the answer in the help file versus the user guides. Sometimes it can be a matter of choosing the right resource in the situation. And, we do realize that we are unable to help every user every time because there can be situations when contacting the help desk is the right decision.

Thanks again to our Q3 survey respondents! And remember, if you take our survey, every quarter we enter the respondents into a drawing for a $25 Amazon.com gift card.

3rd Quarter Documentation Survey Winner

Justin Ward — Thu, 09 Oct 2008 19:24:00 GMT

Each quarter we enter the respondents of our online survey in a drawing for a $25 Amazon.com gift certificate. The 3rd quarter winner for 2008 is Jennifer Sharp of the Foundation for Physical Therapy. Congratulations, Jennifer!

Last quarter we had 73 responses to the survey. That’s right in line with our 50-75 response average per quarter. Moving forward, this number might grow now that we’ve placed a link to the survey in our AES help files (7.77 release).

And remember, to be entered in our 2008 fourth quarter drawing, complete our online survey. It takes only a few minutes. To send us feedback about our user guides or help file documentation at any time, email documentationcomments@blackbaud.com.

Documentation Survey Announcements

Justin Ward — Mon, 14 Jul 2008 16:55:00 GMT

Each quarter we enter the respondents of our online survey in a drawing for a $25 Amazon.com gift certificate. The 2nd quarter winner for 2008 is Christopher Raspe of Oberlin College. Congratulations, Christopher!

Last quarter we had 98 responses to the survey. That’s a new record for us. The survey continues to provide us with lots of great information and it’s also initiated a number conversations with clients we might not have had otherwise.

For the past year, we’ve averaged approximately 50-75 responses, so this quarter saw a substantial increase. This is a trend that we want to continue. In fact, with the 7.77 release of Education Edge, Financial Edge, and Blackbaud Student Information System, we will begin placing a link to the survey in the help files.

Up to now, we’ve captured feedback only from clients visiting our website and using the online user guides. By adding a link to the survey within the help files, we hope to tap into a whole new segment of documentation users. We are very excited about this opportunity.

And remember, to be entered in our 2008 third quarter drawing, complete our online survey. It takes only a few minutes. To send us feedback about our user guides or help file documentation at any time, email documentationcomments@blackbaud.com.

Documentation’s Online Survey

Justin Ward — Tue, 13 May 2008 18:36:00 GMT

Last year, the documentation team created an online survey to capture client feedback about our help content. We’ve done surveys in the past, and while we typically came away with valuable information, we always wished we’d gotten more responses. So this time around, we tried something a little different. Instead of the survey being a one-hit-wonder, we decided to leave the survey up indefinitely -- or as long as our users keep responding.

So far, this approach has worked well. The survey went live last year, and to date, we’ve received over 300 responses. Now, when we evaluate the data, we have more information to base our decisions on, and we can compare response trends from quarter to quarter.

What do we do with the information?
The whole purpose of this project is to improve our help content, both in the help files and user guides. We review survey responses on a weekly basis. When we need more clarification about something a user said, or when users have problems with our documentation, we contact them (just the users who give us permission). All this feedback is then compiled and shared with the team. When we analyze the data, we look for trends in responses so that we can focus our discussions.

For example, we recently focused our improvement efforts on five specific Raiser’s Edge user guides. By reviewing the survey data from the past year, we knew where the problem areas were in these guides. I don’t think we could have honed in on these guides and made meaningful changes without having such a high volume of client feedback over an extended period of time.

We've restructured and reorganized guides for other products as well, improved indexes, and added additional cross references to make our content easier to use. We found that in many cases, we had the information users needed, but they couldn't find it!

Right now, we are averaging 50 to 75 responses per quarter, and I hope this trend continues. If you haven’t taken our survey and would like to, click here. And don’t forget, each quarter we randomly select one of our survey respondents for a $25 Amazon gift card.