Should we build custom chatbots?

Custom chatbots for personalized learning is all the hype.

But lets start with the context and reality of OpenAI’s custom chatbots and why bluntly, I think it is a gimmick for educators planning to use it in the classroom, even if you are rich enough to pay for your entire class/ schools’s ChatGPT plus subscription.

Cost: Usage of Large Language Models is not free

Even as GPT-4 becomes more price competitive, Large Language Models (LLMs) in the near foreseeable future will continue to require sizeable computing power, remain energy intensive and hence be inevitably costly.

The cost of sustaining infrastructure is necessarily passed on to consumers. It is no surprise that free credits/ trials from OpenAI ended since Jan 2023 and rarely do you find any AI-tool given out without a paid subscription plan.

Generally, any free use will inevitably have a usage cap given resource constraints.

Cost: Custom Chatbots by OpenAI will not be universally free

The economics of custom chatbots is a way to drive adoption of ChatGPT plus subscriptions instead of making custom chatbots universally available. Do not expect it to be free.

Especially when bandwidth is constrained and servers are overloaded, custom chatbots will just not be available to all users.

As an educator, creating custom chatbots using the OpenAI frame will be a gimmick even with students possess ChatGPT plus accounts (which is easily $20USD/month/student which adds to $800USD/month/student for a class of 40). Who pays? Is the cost justified, what exactly is the pedagogical value of having such chatbots?

Independent of costs, what are other considerations?

Unpopular opinion, but fixation or being overly starry-eyed with chatbots, especially in this iteration of custom bots by OpenAI (caa 17 Nov 2023 at the time of writing), as the solution-driven approach to teaching and learning may be misguided for educators.

Let’s take a few steps back: what exactly is the problem that we are trying to solve with chatbots?

This is probably a few things:

1. Teacher is overstretched and may not be able to address all queries while rushing the syllabus, especially for graduating classes with impending national exams.

2. Teacher does not have the capacity to draw insights from high volumes of student data to personalize learning - if a teacher is overwhelmed by queries, one cannot process such information or personalize learning effectively

3. Students need but do not have the resources or aptitude to enjoy personalized learning (no money to get a private tutor; too shy to ask questions, etc)

If you build chatbots in the context of education:

1. The scale of impact (number of users, number of expected queries) must justify the cost of maintenance/ development. Any tech solution really need to land the impact of being able to scale across users. The design of custom GPT bots given the context above, provides clear costs constraints that hinders this.

2. Handover-to-agent process/ have a human-in-the-loop - frustrated with the responses? Not sure if the output is accurate? Ideally there is still a handover process to involve the educator in the same way unsatisfactory support by customer service bots have recourse by seeking a live agent.

3. There is a feedback loop for iteration - in this case, just blindly extending custom ChatGPT we are offering a blackbox for students and hoping that responses are generally accurate. Given the state of LLMs and GPT, responses may be generally accurate, but what about instances in which it is not?

4. Have consolidated learning analytics - chatbots, beyond pumping answers efficiently, is a tool for data collection. We synthesize queries of students as part of the profession and do it consciously/ unconsciously. We get a sense of the room. Ideally, a good chatbot can plough through the input of students and help us to better teaching and learning. For instance, it summarize/ highlight common misconceptions by class or note frequently asked questions. The segmentation of data is ideally by class. We profile our classes all the time and adapt our teaching styles accordingly. Could data be collected in the same manner? Data is not fed back to the bot owner at this point which, in my opinion, significantly diminishes the pedagogical value of a chatbot in the context of education.

I think before jumping in to get your-own-custom-chatbot, it might be worth contemplating what is the data pipeline for educators? What are the requirements?

Chatbots are probably part of the future. But what ChatGPT custom bots offer in its iteration, while exploratory and fun, does not fit the specs of what we need in education.

Personalized learning is recursive, iterative in an ideal tech implementation rather than just document uploads and hoping that the learning experience will be meaningful or complementary to the classroom teaching experience. The issue of prompt design, socializing students to use a custom chatbot effectively are other further considerations.

This is not to dismiss chatbots for personalised learning entirely - I do believe good learning assistants (see Duolingo, among others) can be implemented really well to improve T&L and hope we can have a deeper discussion on what constitutes good returns on value/ cost in education.

In the next article, we will share a prototype/ proof-of-concept of how this could work in place of what ChatGPT custom bots offer by first revisiting the problem and specs of an ideal product. The User Interface (UI) will not be as good but let’s see if it withstands the scrutiny.