Introduction to the IRB Survey Results

In their continuing effort to improve, the IRB and the Office of Research Compliance (ORC) supported the use of a survey to get feedback from faculty investigators. The Office of the Vice Provost for Research (VPR) administered the survey and the IRB Monitor and the VPR's IT staff helped to analyze and present the results. The survey included the IRB-Researchers' Assessment Tool (IRB-RAT) and other questions about the functioning of the IRB, the IRB Office and the IRB application and review process. The anonymous survey was given in June, 2007 in paper and pencil format. The survey does not meet the definition of research under the federal regulations because it is being used for program evaluation and improvement only and not for generalizable purposes.


IRB-Rat Results

The Institutional Review Board - Researcher's Assessment Tool (IRB-RAT) was developed by Patricia Keith-Spiegel and Gerald P. Koocher of Children's Hospital, Boston and Harvard Medical School, respectively. The national norming data was based on 2283 randomly selected Biomedical and Social behavioral science recipients of Public Health Service funding and members of the American Psychological Society. They had a 39% return rate, N=886. The intent of the UConn IRB in using this survey is for self assessment/evaluation and not for generalizable or research purposes. However, it is useful to compare our results to the national norming data. The response rate at UConn was 33%, N=114 faculty principal investigators (PIs).

The IRB-RAT consists of 45 questions about IRB characteristics. The National participating investigators were asked to rate the importance of each of the 45 IRB characteristics to their own research, (1 = not important, 7 = essential). In addition, they were asked to rate how their "Ideal IRB" would measure against each characteristic and how they would rate their own institutions' IRBs on each characteristic. UConn Principal Investigators were only asked to rate their Ideal IRB and the UConn IRB on the same 45 characteristics. These answers were also given on a 1-7 scale where 1 = low, 7 = high and 4 = the mid-point.

IRB-RAT Survey Results are reported as follows:

  1. Lists:
    • 12 Most Important IRB Characteristics
    • 5 Least Important IRB Characteristics.

  2. Graphs: The comparisons of results for individual questions are shown in Graphs Q1 -Q45. The Graphs show the mean ratings of how the UConn PIs rated:
    • the UConn IRB,
    • their ideal IRB, and
    • the National ideal IRB for the IRB characteristic addressed in the question.
    The standard deviations are represented by a vertical line extending through the mean.

Some Observations:
The mean for all characteristics rated for the UConn IRB was 5.04 on the 7 point scale compared with the National Ideal IRB where the mean was 5.07. However, the mean rating for all characteristics of an Ideal IRB as rated by the UConn PIs was 5.93. Therefore, while the overall rating of the UConn IRB was only .03 away from the National Ideal, it was .89 points away from the UConn investigators Ideal IRB.

83% of the UConn answers about their Ideal IRB were within 1.5 standard deviations of the mean. 63% of the National norming group's answers about their Ideal IRB were within 1.5 standard deviations of the mean. 53% of the UConn answers about the UConn IRB were within 1.5 standard deviations of the mean.


Additional IRB Survey Results

UConn investigators were also asked for feedback about the IRB Staff and particular aspects of the UConn IRB program like the CITI training in human subject protections. All questions were asked using a 1-7 point scale, where 1=least satisfied and 7=most satisfied unless otherwise indicated.

CITI Training:

  • Open-ended feedback
  • 1-7 scale questions
  • Alternate methods

Open-ended Feedback
Investigators were asked if they could change one thing about the CITI Training program on human research protections, what it would be. Out of 114 returned surveys, we received 40 identifiable areas that investigators would like to change about the CITI Training program. Repeat comments are categorized below:

  • 15 respondents said it should be shorter,
    • 3 of those specified shorter for experienced researchers only.

  • 18 criticized the course content,
    • 6 specified too broad, should be limited to a researcher's area of research.
    • 3 testing aspect was weak,
    • 3 it was repetitious,
    • 3 case method would be of more value
    • 3 miscellaneous: Pretty good overall; hard to pick modules; change UConn part.

  • 7 requested alternatives or be able to test out of the requirement.

Scaled Questions
7 Questions asked respondents to rate different aspects of the CITI course on a scale of 1-7 (1=least, 7=most). 4 out of these 7 questions asked if the CITI course increased understanding of risks and protections for human subjects in research. There were no statistical differences in the answers received on this group of 4 questions.

  • 53% rated this group 5 or above
  • 16% rated this group 4, moderate
  • 31% rated this group 3 or below

Similar rates were received for overall satisfaction with the CITI course:

  • 54% rated it 5 or above
  • 9% rated it 4, moderate
  • 37% rated it 3 or below

The course did appear to have an impact on the respondent's understanding of the Federal Regulations. On this criteria,

  • 72% rated it 5 or above
  • 4% rated it 4, moderate
  • 24% rated it 3 or below

The course had a negative impact on the respondents' willingness to join an IRB:

  • 29% rated it 5 or above
  • 13% rated it 4, moderate
  • 58% rated it 3 or below

Alternate Methods
Investigators were asked if they would prefer a different method of training (assuming an equal time commitment).

Alternate MethodNumber Preferred
Audio1
Video12
Classroom11
No Change74
No Opinion12

IRB Staff Satisfaction
Overall, satisfaction with the assistance provided by the staff.

  • 79% rated it 5 or above
  • 11% rated it 4, moderate
  • 10% rated it 3 or below

Satisfaction with how promptly the staff returns my phone calls:

  • 83% rated it 5 or above
  • 8% rated it 4, moderate
  • 9% rated it 3 or below

Satisfaction with the clarity and knowledge of answers given to questions.

  • 80% rated it 5 or above
  • 9% rated it 4, moderate
  • 11% rated it 3 or below

IRB Website Satisfaction

Use of Website
OccasionallyWeeklyMonthlyNever
61348

Satisfaction with information:

  • 74% rated it 5 or higher
  • 17% rated it 4, moderately satisfied
  • 9% rated it 3 or below

Satisfaction/ ease of finding information:

  • 66% rated it 5 or higher;
  • 17% rated it 4, moderately satisfied
  • 17% rated it 3 or below

Open-ended comments on IRB website - 21 received
In order of number of comments received (most to least) the investigators requested:
more examples of completed forms, a search engine, FAQ section or easier format for investigators to find information and some content topic clarifications.

More Examples:

  • completed protocol IRB-1, IRB-5 and IRB-7 applications
  • projects that meet IRB-5 criteria
  • projects that do not require any IRB application at all
  • Consent Forms, Assent Forms, Information Sheets
  • Debriefing Forms

Content topic clarifications:

  • Criteria for "exempt research"
  • Criteria for projects that are not "human subject research"
  • Criteria for determining when a study is "closed"
  • Use of the terms, "amended" vs. "revised"

Additional Comments or Suggestions for IRB

16 responded with both positive and negative comments

11 comments were negative: 6 regarding Forms and Process, notably Forms are too long, complex and Bio-medically oriented; simplify renewal process. 5 regarding IRB-creep, lack of expertise, and perceived bias against certain research methodologies.

7 comments were positive: regarding improvements seen over the past few years (2005-2007) notably in faster turn around times and being more responsive to PI needs, one example was the IRB-7 application form for Research Methods class projects.

Changes to the IRB Program

Even before the survey was conducted and the results were shared with the IRB and the Office of Research Compliance, efforts were made to meet with departments and address investigators' questions and concerns about IRB requirements and procedures. Both the staff and the program began to undergo changes. The IRB Office was reorganized, to include an additional person to perform the monitoring and education function that would include regular help for faculty and graduate students and help relieve the IRB Staff of some of those responsibilities so they could focus on processing IRB applications. A new Assistant Director who is a Certified IRB Professional (CIP) with experience in a clinical setting was hired to manage the IRB Office and another staff member also became certified as a CIP. A new administrative IRB person became responsible for the initial application and data processing. These efforts helped produce shorter turn around time for reviews and more outreach and support for researchers and students; some of these changes were noted positively by respondents to the survey.

As a result of the survey feed back, additional work has been accomplished or is in progress to put all of the IRB policies into writing, to redesign a more user friendly website with more specific guidelines and examples for researchers, and to move toward electronic submission of applications. We at the ORC appreciate those who took time to answer the surveys and remain interested in listening to investigators and in continuing to improve the program.