ChatGPT

Like many other people (particularly in academia), I’ve been playing a bit with the new large language model from OpenAI, ChatGPT:  https://chat.openai.com/chat.  This artificial-intelligence program is going to change the game substantially in college courses, as it writes better than many college students, though it tends to write a lot of bullshit and fabulates citations.

Most of my experimenting was triggered by posts or comments on the r/Professors subreddit, where professors have been expressing concern about how this new tool will be used for cheating and how it will affect their courses.

My experience is that ChatGPT writes grammatical prose that sounds good, but doesn’t really understand what it is saying.  When I gave it quiz questions from my electronics course, it provided answers that would sound good to a non-expert, but which are clearly wrong, in the same ways that failing students in the course would be wrong. For example, the prompt “What is the impedance of 20kΩ resistor in parallel with a 50pF capacitor at 1kHz?” results in

The impedance of a resistor is simply its resistance, so the impedance of a 20kΩ resistor at any frequency is 20kΩ. The impedance of a capacitor, on the other hand, is determined by its capacitance and the frequency of the current passing through it. The impedance of a capacitor is given by the following formula:

Z = 1 / (2 * pi * f * C)

where Z is the impedance, f is the frequency of the current, and C is the capacitance of the capacitor. In this case, the impedance of the 50pF capacitor at 1kHz would be:

Z = 1 / (2 * pi * 1000 * 50 * 10^-12) = 1.59e5Ω

To find the total impedance of the 20kΩ resistor and the 50pF capacitor in parallel, we need to use the formula for the parallel combination of impedances:

1 / Ztotal = 1 / Z1 + 1 / Z2

where Z1 is the impedance of the first component (in this case, the 20kΩ resistor), Z2 is the impedance of the second component (in this case, the 50pF capacitor), and Ztotal is the total impedance of the two components in parallel. Plugging in the values from above, we get:

1 / Ztotal = 1 / 20,000 + 1 / 1.59e5

Solving for Ztotal, we get:

Ztotal = 1.19e5Ω

So the total impedance of the 20kΩ resistor and the 50pF capacitor in parallel at 1kHz is approximately 119,000Ω.

This “answer” replicates two of the common mistakes of failing students: it neglects that the impedance of a capacitor is a purely imaginary number and it gets the arithmetic of fractions wrong. Until the large language models can be appropriately merged with the math AIs (like Photomath or Wolfram Alpha), engineering faculty do not have much to fear from ChatGPT.  I guess that it will take 5–10 years to do that sort of merging, as the two approaches to AI are not very compatible.

I tried some more “creative” prompts also and tweaking the prompts to get better results. I tried one series of prompts (I won’t show the results for any but the last iteration):

• Write a song to be sung by students with the theme and title “When will I ever need this?”
• Try again but with rhyming verses.
• There is no rhyme there.
• Try again with real rhymes.
• Try again with a blues-song format and general air of depression.
• Try again without the upbeat bridge.
• Remove the current outro and end with singer being lost and forlorn.
• Try again incorporating “1812”, “square root of 1066”, and “the quadratic formula”.

resulting in 

Verse 1: Sittin’ in this classroom
Feelin’ like a waste of space
Teacher’s talkin’ at me
But I can’t hear a word he’s sayin’

Chorus: When will I ever need this?
Will it help me find my way?
Will it make me rich or famous?
When will I ever need this?

Verse 2: I ain’t sayin’ it ain’t important
But it just don’t make no sense
Memorizin’ 1812 and the square root of 1066
And the quadratic formula too

Chorus: When will I ever need this?
Will it help me find my way?
Will it make me rich or famous?
When will I ever need this?

Outro: Feelin’ lost and forlorn
In this never-endin’ maze
I just don’t know where to turn
When will I ever need this?

Chorus: When will I ever need this?
Will it help me find my way?
Will it make me rich or famous?
When will I ever need this?

The main use case for faculty seems to be creating email templates or assignment prompts,  Here is an example of an email template (with the prompt provided from a question asked by a professor on the subreddit). Prompt: “Write an email telling a student that I will not change your grade because that is the grade you earned, and that there are no more opportunities for extra credit as the term is over.”  I had to reprompt with “Try again without inviting the student to reply.”

Dear Student,

I want to let you know that I have carefully reviewed your work and have determined that the grade you received is an accurate reflection of your performance in the course. As such, I will not be changing your grade.

Furthermore, the term is now over, so there are no more opportunities for extra credit. I understand that you may be disappointed with your grade, but please know that it is my belief that you earned the grade you received through your work in the class.

Sincerely,

[Your Name]

The main ways that people have thought of to avoid or detect ChatGPT cheating have included highly technical content that requires math skill; checking citations (ChatGPT likes to fabulate them); making prompts that use a visual image (it will be a while before the large language models are merged with computer vision); requiring web lookup of current events (newer than the model’s training—the models currently do no search); requiring live, in-class writing; and requiring multiple stages of writing process (proposal, outline, draft, edited draft).  I don’t think that the last approach would work, as ChatGPT seems capable of doing outlines and editing drafts.

Someone suggested that real student writing could be detected by the cliched “hooks” that students use, but that is easily faked also: “Write a 200-word essay about cheating using AI, starting with ‘Since the dawn of education, students …’ “

Since the dawn of education, students have always looked for ways to cheat in order to get ahead. With the rise of technology, cheating has become even easier with the use of artificial intelligence (AI). AI can be used to generate answers to tests or homework assignments, or even to write essays.

While using AI to cheat may seem like an easy way to get good grades, it is ultimately a dishonest and unethical way to succeed. Not only does it go against the principles of fair competition and hard work, it also undermines the value of education. By cheating, students are not truly learning and are not gaining the skills and knowledge necessary for future success.

Furthermore, using AI to cheat can also have serious consequences. Students caught cheating can face disciplinary action from their schools, and in some cases, it can even result in expulsion. In addition, using AI to cheat can also have long-term consequences on a student’s future, as it can damage their reputation and make it more difficult to get into college or find a job.

In conclusion, while using AI to cheat may seem like an easy way to get ahead, it is ultimately a dishonest and unethical action that can have serious consequences. It is important for students to understand the importance of hard work and honesty in education, and to resist the temptation to cheat.

I’m convinced that Chegg will be buying a large language model and retraining it on their large database of cheating materials, resulting in an AI that can replace the “experts” that they hire in India. The results from ChatGPT are already about as good as Chegg answers, and training a dedicated CheatGPT bot will undoubtedly increase Chegg’s profit margin (no labor costs for “experts”).

For my Spring quarter class, I will have the students use ChatGPT (assuming it is still free then) to create the summaries that they need to write for the class—they will have to provide ChatGPT with the info and have it turn it into readable prose of about the right length.  I think that this approach will result in more readable summaries than in the past, while still challenging students to come up with the key points that need to be communicated. I suspect that a lot of writing in future will be done by a combination of human and artificial intelligence, and I think that this 2-unit, non-required, P/NP course will be a good way for students to experiment with that approach to writing (with no academic-integrity issues interfering).