What Happens When You “Set and Forget” with ChatGPT

For many, ChatGPT and other AI tools and software have been a blessing for productivity. There are loads of tasks that have been sped up or even automated by different Artificial Intelligence bots and frameworks, from internal processes to customer service interactions and so much more. With this new and growing power of AI, it can be tempting and seem like the obvious decision to just fully send into automation across applications. However, there are a few issues that can come up if you aren’t careful, and they have a lot to do with not fully knowing what’s happening underneath the hood. Here are some things to keep in mind, so you don’t “set and forget” with ChatGPT and end up with some avoidable problems.

Potential Inaccuracies

Although there have been significant improvements with the latest iterations of the ChatGPT models, OpenAI has been rather transparent about how the models can and do fall short. In some sense, the ChatGPT models construct the “most likely” string of words to fit in a response to a prompt. It’s quite a bit more sophisticated than just a statistically likely string of characters, but the idea is the same.

ChatGPT and the relevant neural networks don’t “think” per se, and for this reason, they don’t necessarily reflect on what they’re saying. This means that the AI sometimes presents incorrect information, and it won’t catch itself if you’re not checking. “Setting and forgetting” when this happens can snowball into bigger problems.

For example, let’s say you’ve automated a navigation system with AI. When you make a wrong turn using Google Maps, the app recalculates the route and corrects for the mistake. If the automation app you’ve created churns out a wrong turn, and it doesn’t catch itself, it might not correct for its mistake, and it might end up sending you to an incorrect location. If there are more errors, you’ll end up in a much more obscure location. Hopefully you aren’t navigating with an AI model that is known to be inaccurate from time to time, but the point here should be clear: setting and forgetting with ChatGPT can lead to oversight of some pretty serious issues.

A Decision Black Box

Another issue that can come up with this level of automation is that, often, AI models are a bit of a black box. You might be getting good results for your prompts, and if you’ve automated some processes you might get some good results in your framework, but there might be a crucial issue: replication.

If you pull up ChatGPT and put in the same prompt twice, you’re likely to get different responses. If there are mistakes, as mentioned above, that’s bound to give you some unwanted variance, but there will be variance even without mistakes. This is great for making interesting conversations with a chatbot, but maybe not so great when you want to achieve consistent and high-quality results. One big issue with ChatGPT and similar tools is that there’s little-to-no way to look under the hood and see what’s being done at all. Even if the results are good, if you don’t know how decisions are being made, there’s a limit to what you can trust the software to do.

Without being able to see what’s actually going on, trust isn’t the only relevant issue. There’s also just a limited degree to which you’ll be able to understand how and why good results are achieved. If nothing else, this means that although you might get a great result one day, you might find yourself struggling to produce similar results later. There’s no “good performance button,” which can be frustrating when things turn into a guessing game of how to make this mysterious tool act the way that you want it to. Without being able to finely tune the model yourself, letting it run unsupervised is very true to the word unsupervised.

Further Limitations

None of these points are to say that ChatGPT and AI can’t be trusted. There are very useful and powerful applications of AI that are promising and even in practice now. However, if we want to talk about “setting and forgetting” with ChatGPT, there is one pretty striking obstacle: ChatGPT is prompt-based. In other words, at least at face value, ChatGPT isn’t something you can program to run in the background. It uses its Large Language Model to generate responses to each user-input prompt, or a string of prompts. This means that it’ll take a non-insignificant amount of effort to fully “automate” ChatGPT to the point of being able to “set and forget.” You might build a bot that feeds ChatGPT prompts automatically, but depending on the application, this might be a pretty significant task.

There are ways of getting around this “limitation” and, in some applications, it might not be seen as a limitation at all. For example, ChatGPT has already been offered as a repackaged customer assistance bot for brands and companies. With AI customer service chatbots, customers and guests already feed the bot prompts. While you’ll have to keep tabs and make sure the bot is behaving appropriately, this is very much a situation in which “automation” can be accomplished while being prompt-based. There are still dangers to navigate, though. ChatGPT has offered more realistic and reliable interactions in both customer service chats and otherwise than most other AI bots ever before, but limiting customers to AI assistance can be a treacherous path. If the bot doesn’t behave as well as you’re expecting, and customers don’t have access to real customer support humans, you could be playing with the trust and patience of your customer base.

All in all, it’s important to consider both the strengths and limitations of tools like ChatGPT. It can be incredibly powerful and helpful in certain situations, but certain limitations can also paint drawbacks and weaknesses for some applications. Although neural networks are more powerful than ever before, your biggest strength will be putting your good ol’ human intuition first.