Do we Really Need Another AI Chatbot?

Socrates might ask, what do you mean by “need”? There are many uses for AI chatbots that are not yet addressed by commercial chatbots. We have created our chatbots to demonstrate some of these scenarios; the first two have been contributed to the Fabric open-source project. Socrates is a model that has been trained on the Socratic Method through extensive analysis of some of the more popular works of Plato and Xenophon (many thanks to Project Gutenberg for the source materials). Duke is a model that has been tailored to the software development community specifically in the technologies used in and with Java programming.

Personalities

Our artificial intelligence chatbots are given distinct personalities to make them more relatable; this has been so successful that it actually created some discomfort during development when we almost felt bad about sending Duke simple messages repeatedly during testing. As a Question & Answer chatbot, Duke was initially all business; if you weren’t asking him something about software development, he would basically just tell you to focus on what he wanted to discuss. Socrates, on the other hand, is a conversational chatbot. Initially, he also was somewhat oblivious. He did not possess the ability to temper his responses to the tone of the user. We have now given them both a bit more awareness of concepts such as humour.

Process

Socrates

Socrates was our first foray into the world of artificial intelligence. Initially, we just fed him the works of Plato and Xenophon and asked him questions. His responses were what we would now expect, excerpts of the source material that pertained to the request. While that may be useful for a student of philosophy, our goal was to create a more conversational and engaging experience.

We then used a generic AI chatbot to extract the top five themes represented in the source material. With that in hand, we used the AI chatbot to analyze the responses and identify patterns in how Socrates handled different types of questions. Now we had something! Finally, we asked the chatbot to identify the style of Socrates’ sense of humour.

All of that was then considered to produce the System Prompt which is prepended to the prompt provided by the user.

Duke

As a Question & Answer chatbot, we were presented with the challenge of accuracy. In the field of software development, the correct answer does not remain “correct” indefinitely. It might be said that the only constant in software development is “change”. Oracle publishes new Long-Term Support (LTS) releases of Java every two years. VMWare releases a new major version of the Spring Framework on a similar schedule, while Spring Boot, is released every six months. This ever-changing landscape makes it difficult to provide answers that are correct for the environment in which the user is working.

Initially, Duke was like other chatbots and only giving us useful results about ten or twenty percent of the time. Part of the problem was that he was only trained on data up to 2021. Enter Retrieval-Augmented Generation (RAG). Duke will now reference the official documentation from Oracle and VMWare prior to providing an answer. (This feature is still in beta.) Next, the users will be given the ability to isolate Duke’s responses to specific versions through the click of the mouse.

Privacy

With our background in security, we take privacy very seriously. For organizations this is more than a concern as it is increasingly included in regulatory requirements. Our models are running on an oLlama server locally within our lab. This ensures that no data that is submitted to our AI system leaves our network.

Value (Cost vs. Quality)

Commercially viable AI chatbots face a problem... the size. In order to get quality responses from an AI model, one can re-train a model to focus its knowledge, or create verbose system prompts (as discussed above). Our research produced some quality results however they required system prompts of over 8000 characters! That’s roughly 1600 tokens… and doesn’t yet include the request from the user! Clearly, our chatbots needed to go on a diet.

We started with Duke. Duke’s weight comes in part from us wanting him to provide assistance with more that simple syntactic knowledge. We also wanted Duke to be able to pull together RESEARCH material, and conduct CODE REVIEWS. Our system prompt included instructions on the methodology to conduct each of these tasks in addition to the process of standard interactions and output formatting. We needed to find a way to only give Duke the instructions required for a specific type of request.

As you may know, this site is built using Java and Spring. The Spring AI project has introduced the concept of Advisors. We created an advisor that will first categorize the user’s request, and then update the system prompt to include only the relevant instructions for the type of request submitted.

Next Steps

Our success to date reveals that there remains much to be explored.

• Our chatbots need to learn from their mistakes;
• They also need to be able to search the internet for specific results, using anonymized queries;
• Creating proactive agents;
• Creating plugins for various existing applications;

These are exciting times, stay tuned…