3. Research Question and Research Methods

The motivation of the research question is threefold:

1. The need to tailor and adapt guidelines to specific creative disciplines
2. The need to communicate clear expectations to students for the use of AI in advance of assessments.
3. The need to reduce technical barriers for learners to be AI-literate (demystify AI).

Research Question

How can we create practical guidelines on the use of AI coding tools for BSc Data Science and AI students at the Creative Computing Institute, to reduce the barrier to AI literacy in a technical programming context?

Research Method

I am inspired by the use of technology probes in action research (Hutchinson et al., 2003), that is, creating a digital intervention and then evaluating users’ responses to it in their own environment (Madden et al., 2014).

In this case, I use a GenAI Checklist and a “Rubber Duck” Chat Mode as probes to be introduced to the learners, seen as probes in a field-testing setting, taken to the learners’ workspace:

1. A GenAI Checklist tailored from the UAL Student Guide to Generative AI. It aims to provide guidelines on the responsible use of AI coding assistants, including more detailed instructions on:
   • How to use AI coding tools in an academic context,
     • How to keep track of code generation/editing done by AI,
     • How to add the “Generative AI Disclosure” in coding.
2. A “Rubber Duck” Chat Mode will be implemented in GitHub Copilot (the AI coding tool integrated into the software used by CCI students) and provided to students in the unit. Chat Mode is a set of “pre-task instructions” for Copilot, to tailor its behaviours by setting the overall goal. I hope this can better support learners, and potentially help students identify what knowledge/skills are needed.

Detailed write-up of the above two elements are described in a separated blog post: The GenAI Checklist and the Rubber Duck Chat Mode.

Participants

Students in the BSc Year 1 Mathematics and Statistics for Data Science (Math&Stats) unit who gave consent to join the research project. The unit runs from Sep 2025 to Jan 2026.

Dissemination (roll out) of the technology probes

A GenAI workshop (~60mins) will take place during the class time of Math&Stats, workshop schedule:

• 0 – 15mins: I give a brief on the use of AI and AI coding tools
• 15 – 30mins: I walk through the principles and guidelines of using GenAI coding tools in class, the adapted GenAI checklist, and examples of how to keep track of the use of AI for coding.
• 30 – 40mins: Practical notes on AI coding tools, including how they work, how to set up, applying for educational benefits (free access for students), and limitations of AI coding tools.
• 40 – 60mins: Several programming tasks are prepared. In this activity, students get into groups of two, choose a task, use the Rubber Duck chat mode and prompt the AI to do the task, keep a chat log of their use according to the GenAI checklist, and put their results in Miro and share with the class.

Slides for the GenAI workshop:

Data Collection

First, I lead a group discussion with the student participants, about the perceptions toward AI in academia, I use a Miro board to collect their anonymised answers to:

• What did you use AI for? (What was the model? What questions/prompts did you ask? How did it go?)
• What excites you about using AI in the university? What worries you about using AI in the university?

Second, students are required to submit a written chat log, following the GenAI checklist, if AI is used in their assignment. I collect completed chat logs submitted by student participants in their final assignment, aim to observe students’ response of these two elements in practice.

Data Analysis

I’ll deliver the Miro discussion in the form of a word cloud.

I’ll reflect on the sample chat logs to discuss whether the GenAI checklist has been well-received by participants, I’ll discuss on the following points:

• What patterns and themes can be identified?
• What do they reveal about students’ perception of the GenAI checklist?
• How to improve? What might be useful to gather next?

Next step

I put my ethical action plan and participant-facing document at 4. Ethical Action Plan and Participant-Facing Documents

I wrote a detailed description, and a demo of the two interventions (probes) at 5. The GenAI Checklist and the Rubber Duck Chat Mode