Research Question and Research Methods

The motivation of the research question is threefold:

1. The need to tailor and adapt guidelines to specific creative disciplines
2. The need to communicate clear expectations to students for the use of AI in advance of assessments.
3. The need for technical programming skills creates barriers for learners to be AI-literate.

Research Question

How can we create practical guidelines on the use of AI coding tools for BSc Data Science and AI students at the Creative Computing Institute, to reduce the barrier to AI literacy in a technical programming context?

Research Method

I am inspired by the use of technology probes in action research (Madden et al., 2014), that is, creating a digital intervention and then evaluating users’ responses to it in their own environment. In this case, I use a GenAI Checklist and a “Rubber Duck” Chat Mode as probes to be introduced to the learners, seen as probes in a field-testing setting, taken to the learners’ workspace, to observe their responses:

1. A GenAI Checklist tailored from the UAL Student Guide to Generative AI. It aims to provide guidelines on the responsible use of AI coding assistants, including more detailed instructions on:
   • How to use AI coding tools in an academic context,
     • How to keep track of code generation/editing done by AI,
     • How to add the “Generative AI Disclosure” in coding.
2. A “Rubber Duck” Chat Mode will be implemented in GitHub Copilot (the AI coding tool integrated into the software used by CCI students) and provided to students in the unit. Chat Mode is a set of “pre-task instructions” for Copilot, to tailor its behaviours by setting the overall goal. I hope this can better support learners, and potentially help students identify what knowledge/skills are needed.

Details of the above two elements are described in a separated blog post: The GenAI Checklist and the Rubber Duck Chat Mode.

Participants

Students in the BSc Year 1 Mathematics and Statistics for Data Science (Math&Stats) unit who gave consent to join the research project. The unit runs from Sep 2025 to Jan 2026.

Dissemination (roll out) of the technology probes

A GenAI workshop (~60mins) will take place during the class time of Math&Stats, workshop schedule:

• 0 – 15mins: I give a brief on the use of AI and AI coding tools
• 15 – 30mins: I walk through the principles and guidelines of using GenAI coding tools in class, the adapted GenAI checklist, and examples of how to keep track of the use of AI for coding.
• 30 – 40mins: Practical notes on AI coding tools, including how they work, how to set up, applying for educational benefits (free access for students), and limitations of AI coding tools.
• 40 – 60mins: Several programming tasks are prepared. In this activity, students get into groups of two, choose a task, use the Rubber Duck chat mode and prompt the AI to do the task, keep a chat log of their use according to the GenAI checklist, and put their results in Miro and share with the class.

Slides for the GenAI workshop:

Data Collection

Group discussion on participants’ perceptions toward AI in academia, including anonymised answers to:

• What comes to mind when you hear the term AI? (Words/Feelings, what excites you about using AI? What worries you about using AI)
• What did you use AI for? (What was the model? What questions/prompts did you ask? How did it go?)

I collect sample chat logs submitted by participants in their final assignment, aim to capture how the checklist and the rubber duck work in practice, and students’ responses to them.

Data Analysis

I’ll deliver the Miro discussion in the form of a word cloud.

I’ll reflect on the sample chat logs to discuss whether the GenAI checklist has been well-received by participants, I’ll discuss on the following points: