Meeting Decisions

Assessment Database Advisory Committee Meeting
Kellogg Biological Station
November 9 - 11, 2007

Decisions made

1. A working group session at the 2008 ASM annual meeting in May will introduce the database to the microbiologists.
2. The database is and is not a test bank.
• The primary function of the database is to facilitate and encourage research in undergraduate biology education.
• A secondary function is to act as a test or question bank.
3. Categorizing assessment questions will be done in two ways:
• Bloom’s Taxonomy (e.g. 1 – 6) or an abbreviated Bloom’s (e.g. higher-level thinking, lower-level thinking); and,
• Concept categories.
4. The database must be able to store multiple raters and their interpretations of questions/items.
5. Concept categorization
• We should start with a pre-existing taxonomy.
• NBII.gov – National Biological Information Infrastructure. This will be our starting point.
• Categorization must evolve to keep pace with cutting-edge research.
6. Metadata collection
• The metadata forms must be streamlined and seeded with more forced (selected) choice categories.
• We agreed to replace questions about: texts used, pre-requisites, required majors, instructional strategies, assessments used with a copy of the class syllabus.
• All data are section data. We therefore changed the wording of several questions to eliminate confusion about course versus section.
• We promoted the last question (What is the primary reason you submitted your data) to the top of the page.
• Language needs to follow IMS language standards.
• A question will be added to the assessment-level metadata asking whether this assessment (or any part of the assessment) is part of a concept inventory, published assessment or otherwise protected.
• Additional data on the assessment form: in-class/outside of class; open/closed book; test/quiz/homework/clicker question/project/other.
• We changed the total number of points of the assessment to percentage of the final grade.
7. Issues for future rounds of funding
• IRB issues are beyond the scope of this project/round of funding. However, the ICPSR is a model we will continue to investigate.
• Machine driven de-identification of constructed student response data is a non-trivial issue. The prototype database will not allow for constructed responses to be uploaded during this funding period. Although it might be possible for the name to be removed from an individual student response, identifying information may still appear within the constructed answer. As a result, we chose to not include any constructed response data.
8. Terms of use
• Many assessment items are published and/or copyrighted. Faculty must agree to upload only original work, credit original sources or obtain permission to upload items.
• We will investigate the Creative Commons licensing agreements for one that is appropriate to our database. http://creativecommons.org/
9. Upload interface
• Assessment items/questions will be number-less (ie the number from the exam should be removed).
• Exams will be saved as PDFs and will be archived in their original format.
• Quality control: outside reviewers from a professional society (e.g. ASM, American Society of Microbiologists) will review questions and tag those that are considered ‘good’.
10. Storage and search interface
• We will author a vocabulary that constrains certain fields of the metadata.
11. Minimum metadata requirements
• All metadata are required, except for Bloom’s categorization and student response.