To the survey: https://docs.google.com/forms/d/e/1FAIpQLSfujaTdkiyrwagGO4Cv_JjGQ_IIWt9v4tO54aVBSiF7J6a_pw/viewform?usp=header
Once I had analysed the overall results, I became curious about the individual images. The average recognition rate already showed that identifying AI-generated images is difficult, but I wanted to understand what was behind those numbers. Which images confused people the most? Which ones were recognised more easily? Looking at each image one by one turned out to be one of the most interesting parts of the whole project.
As I mentioned in my previous blog post, authentic photographs were generally easier to recognise than AI-generated images. But once I started looking at the individual images, I realised that the averages only told part of the story.
Some authentic photographs were recognised by more than 70% of the participants, and one image was even recognised by almost 80%. At the same time, several fully AI-generated images were correctly identified by only around 20% of participants.
One result immediately caught my attention. Image 13 was an authentic photograph, but only 14.3% of participants identified it correctly. Image 20, which was fully AI-generated, reached exactly the same recognition rate. I found it fascinating that two completely different image types produced almost identical results. So, looking at each image individually revealed details that would otherwise have been easy to miss.
Another interesting part of the analysis was the confidence ratings (1 = not confident at all, 5 = very confident). Before analysing the data, I expected participants to feel much more confident when they answered correctly. Instead, the difference was surprisingly small.
On average, participants rated their confidence as 3.13 when they answered correctly and 2.99 when they answered incorrectly. Both values are very close to the middle of the scale, meaning that participants generally felt only moderately confident about their decisions. What surprised me most was how similar these two values were. Whether participants were right or wrong, their confidence hardly changed.
I also noticed a change in how participants viewed their own ability to recognise AI-generated images. Since participants rated their own ability before and after the survey, I was able to compare both answers. At the beginning of the survey, many participants rated themselves as being quite good at identifying AI-generated content. After completing the survey, several participants rated themselves as less confident than before. Although this pilot study is too small to draw firm conclusions, it seems that taking part in the survey made some people realise that distinguishing between authentic and AI-generated images is more challenging than they had expected.
The second part of my pilot study was a little different. This time, participants did not have to classify a single image. Instead, they saw two similar images and had to decide which one they thought was authentic. I was curious to find out if comparing two images directly would make the task easier.
The average recognition rate was 55.4%, which is slightly higher than in the first part of the survey. Even so, I was surprised that the results were still so close to chance. Some image pairs were much easier than others. Pair 3 was recognised correctly by 71.4% of the participants, while Pair 4 was the most difficult, with only 42.9% choosing the authentic image. Most of the other image pairs were recognised correctly by about half of the participants.
For me, this part of the study added another interesting perspective. It showed that even when participants could compare two images directly, deciding which one was authentic was often still difficult.
Analysing the results in this level of detail helped me understand much more than the overall recognition rate alone. It showed me which images were particularly successful for the study and which ones challenged participants the most. These observations will surely help me improve the image selection for my main study.
!!Attention!!
!!Please only look at the documents if you have already completed the study or if you do not intend to participate in it!!
