Anyone using Skilljar's AI Tutor yet?

Hey ​@efoley.

We evaluated the AI Tutor when it was released and decided not to turn it on for similar reasons.

• One of my main concerns was whether the chatbot would restrict its answer to the content of the current lesson. We would prefer it if it did, because LLMs in general (especially older models or cheaper ones) are not good at discussing our product. Many of them are trained on outdated information or hallucinate because our product is relatively niche. 
  • In our testing, we found that the chatbot, unfortunately, will try (and often fail) to answer questions outside the context of the current lesson.
  • The chatbot's habit of adding follow-up options with every response (which many people might recognize from OpenAI models) quickly leads the chat to abandon the lesson context, as the bot suggests discussing topics outside the lesson scope. In my case, this led to a pretty unrelated attempt to explain aspects of our platform that were not 100% wrong but were outdated and not related to the lesson topic. This tendency could be useful if the chatbot were accurate (explaining related topics or pointing the learner towards resources), but because it both does this and is often wrong, it's not helpful.
• We find the chatbot behaves a lot like the alpha version of our own chatbot that we deploy in our product. We had to do quite a bit of work internally to fix some of the problems. 
  • For example, without help, most LLMs hallucinate things on our platform that don't exist. This problem is very evident in the AI Tutor. It will make up tools that do not exist and very confidently tell you how to add and configure them. This kind of advice is counterproductive to learning and can be frustrating for learners. Especially since learning the tools is a big part of the initial learning curve!
  • It linked to out-of-date documentation in many instances. I assume this is because it was trained on an older version of the internet and does not actively search for up-to-date resources using tool calls like some LLMs do now.
• While it actually did an OK job suggesting troubleshooting steps during an exercise, it ended up providing "Recommended content" that was just the page we were already on, which seemed a bit redundant. Though funnily enough, it's kind of true: "I recommend reading and following the exercise instructions if you are confused." 
• I wonder whether users will be annoyed that their chat history isn't saved per lesson, i.e., that if you have a chat in a lesson, navigate away from that lesson, then come back, your chat is reset. We all expected it to resume our last conversation on that lesson.

So, on balance, we found the AI Tutor was more often unhelpful than helpful. In order to turn this on, we'd need to see some mix of the following:

• Only linking to up-to-date resources.
• Not hallucinating elements of our product.
• If the above is not possible, restricting the answers very specifically to the content of the lesson would hopefully restrict the degree to which it could be wrong.

I will also point out another issue I think we are all encountering: because pretty much every product is adding a chatbot, users now have to choose which chatbot to use. We already provide three user-facing chatbots:

• In-product chatbot we built ourselves
• Support site chatbot powered by Zendesk
• Marketing chatbot powered by https://expertise.ai

Of these, we vastly prefer that users access our in-product chatbot for troubleshooting and learning. This chatbot has access to their workflows and has been specifically designed to avoid common LLM shortcomings regarding our product. Most learners taking our training should have access to this chatbot. Since it is much more accurate than the AI Tutor, we will focus on directing learners to use it instead.

​@sam.walker Super helpful reply, as usual. Thank you!