Rubric for AI Tool Evaluation

We translated the above framework into a rubric to help institutions and instructors evaluate AI tools for their effectiveness, usability, and ethical considerations in teaching and learning contexts. Readers can also access and adapt for their own contexts the Google Form version of the rubric.

Assessment 1: Functionality & Pedagogical Usefulness

Score the tool on each of the items below using the scale provided.

• Does the tool enhance teaching and learning (i.e., is it either designed for or adaptable to teaching/learning)?
• Does it support instructional strategies such as active learning, personalized learning, or formative assessment?
• Does it align with faculty needs in various disciplines?

• 1 – No clear educational application/value
• 2 – Limited usefulness in teaching and learning contexts
• 3 – Neutral—or I haven’t tried it
• 4 – Strong educational value, albeit use-case dependent
• 5 – Highly valuable across instructional contexts

Optional: Please comment on your top one or two choices in terms of “Functionality & Pedagogical Usefulness,” sharing one or two reasons (each) about why you “recommend” faculty (or students) to adopt/explore the AI tool(s).

Assessment 2: Accessibility & Inclusivity

Score the tool on each of the items below using the scale provided.

• Is the tool accessible to students with disabilities (e.g., screen reader compatibility, closed captions, keyboard navigation)?
• Does it support multilingual learners (e.g., translation, speech-to-text, adaptive features)?
• Do its underlying dataset and algorithm seem culturally/globally informed and sensitive?
• Does it offer flexible options for diverse learning needs?

• 1 – Not at all
• 2 – Somewhat
• 3 – Neutral—or I haven’t tried it
• 4 – Good
• 5 – Excellent

Optional: Please comment on your top one or two choices in terms of “Accessibility & Inclusivity,” sharing one or two reasons (each) about why you “recommend” faculty (or students) to adopt/explore the AI tool(s).

Assessment 3: Ease of Use & Faculty Adoption

Score the tool on each of the items below using the scale provided.

• Is the tool intuitive for instructors and students?
• Does it require significant training to use effectively?
• Is it easy to integrate into existing teaching workflows (e.g., LMS, productivity tools)?

• 1 – Very difficult to use or implement
• 2 – Steep learning curve, requiring substantial training
• 3 – Neutral—or I haven’t tried it
• 4 – Mostly intuitive and easy to integrate and train users
• 5 – Very user-friendly with seamless integration

Optional: Please comment on your top one or two choices in terms of “Ease of Use & Faculty Adoption,” sharing one or two reasons (each) about why you “recommend” faculty (or students) to adopt/explore the AI tool(s).

Assessment 4: Ethical Considerations & Data Privacy

Score the tool on each of the items below using the scale provided.

• Does the tool respect student privacy and comply with FERPA, GDPR, or other regulations?
• Beyond being merely legal, will the tool potentially undermine students’ privacy/confidentiality and intellectual property rights?
• Does it have clear policies on data storage, AI-generated outputs, and user tracking?
• Are there risks of bias, misinformation, or unethical use?

• 1 – Major privacy or ethical concerns
• 2 – Some ethical or privacy concerns
• 3 – Neutral—or I haven’t tried it
• 4 – Clear ethical and privacy protections in design and user policy
• 5 – Strong policy and design to ensure ethical use and data security

Optional: Please comment on your top one or two choices in terms of “Ethical Considerations & Data Privacy,” sharing one or two reasons (each) about why you “recommend” faculty (or students) to adopt/explore the AI tool(s).

Assessment 5: Cost & Sustainability

Score the tool on each of the items below using the scale provided.

• Is the tool free, subscription-based, or pay-per-use?
• Is it sustainable for long-term faculty and student use?
• How significant are hidden costs (e.g., user data is used for training, free usage is limited and premium features are required for being fairly useful, ads disrupt use, information asked for creating an account)?

• 1 – Prohibitive in cost or unsustainable
• 2 – Somewhat costly or unclear long-term viability
• 3 – Neutral—or I haven’t tried it
• 4 – Generally cost-effective and sustainable
• 5 – Free (monetarily) or highly sustainable for academic use

Optional: Please comment on your top one or two choices in terms of “Cost & Sustainability,” sharing one or two reasons (each) about why you “recommend” faculty (or students) to adopt/explore the AI tool(s).

Assessment 6: AI Transparency & Explainability

Score the tool on each of the items below using the scale provided.

• Does the tool link to and stay true to human sources (instead of merely generating plausible patterns that require expertise, time, and labor on the user’s part to verify)?
• Can faculty and students fairly easily validate AI-generated outputs?
• Does the tool frequently “hallucinate” or otherwise fail to do what it promises?

• 1 – Opaque/no transparency or fault-prone
• 2 – Limited insight into data/process or often faulty
• 3 – Neutral—or I haven’t tried it
• 4 – Mostly clear AI reasoning and reliable outputs
• 5 – Highly transparent, strong safeguards, and reliable response

Optional: Please comment on your top one or two choices in terms of “AI Transparency & Explainability,” sharing one or two reasons (each) about why you “recommend” faculty (or students) to adopt/explore the AI tool(s).

Assessment 7: Institutional & Instructional Design Support

Score the tool on each of the items below using the scale provided.

• How easy is the tool to support at the institutional level (e.g., IT, instructional design, faculty development)?
• Does the tool have adequate documentation, training materials, and support resources?
• Can the tool be easily integrated into existing instructional design frameworks and pedagogical models?
• Does the tool require significant ongoing maintenance or technical support?

• 1 – Very difficult to support; requires extensive technical expertise or custom infrastructure
• 2 – Challenging to support; limited documentation or high maintenance needs
• 3 – Moderate support required; some training or infrastructure adjustments needed—or I haven’t tried it
• 4 – Generally easy to support with existing IT and instructional design resources
• 5 – Minimal support required; well-documented with strong institutional alignment

Optional: Please comment on your top one or two choices in terms of “Institutional & Instructional Design Support,” sharing one or two reasons (each) about why you “recommend” faculty (or students) to adopt/explore the AI tool(s).

Of course, not all criteria and variables will be relevant for instructors, researchers, and other users; our goal is to prompt similar development of criteria and to assess both tools and tool uses in context. If the use of AI tools hurts more than helps the task at hand, any broader purpose or cause, or the overall mission of education or its contribution to society, then the tools’ “helpfulness” in itself must be judged accordingly. With this in mind, when developing the case studies summarized and discussed below, we asked evaluators to contextualize their evaluation in their disciplines. We also asked them to rank and comment on the tools they recommended. We hope that academic leaders and staff, faculty scholars, and students will benefit from the general ideas of evaluating AI tools and their uses, as well as from adapting the framework and rubric here. The next section illustrates our evaluation of a few AI tools.