Of Course There Was AI at Splunk .conf23

Every product announcement in 2023 is required to include AI. It’s the law. However most of them are vague, fluffy and just to keep the investors happy. Only a few offer meaningful benefits. Splunk’s new offerings are in this latter category, eschewing wild claims and big noise, but respecting the importance of humans and the real issues that SREs face in their daily working lives. And refreshingly they are not all generative AI.

Not AI Newbies

The new AI capabilities were presented by Min Wang, who became Splunk CTO in April 2023 after five years as an ED at Google where she worked on the Google Assistant. While that consumer-focused product may seem adjacent to SEIM, that’s not the point. It’s the experience of rapidly and easily assisting people that counts, augmenting their productivity.

This is also not Splunk’s first sortie into machine learning and AI, having launched Splunk ML in 2015. This means that the company not only has deep AI-related experience, it already has a mass of data on which to draw, whether for threat and attack detection, automation or analysis. This means that Splunk have domain-specific models built from a solid base, unlike a generic tool such as ChatGPT which lacks any contextual knowledge.

Responsible AI, Really

The current boom in AI has lead to a crazy amount of AI washing, irresponsible releases and endless foolishly exaggerated claims. There are a number of basic rules that should be applied to launching any AI product, and Splunk is one of the few organisations actually following them.

The first rule of responsible AI is to always maintain a human in the loop. All too often cost-fixated management try to use AI as a means of eliminating staff, often with catastrophic results. Since security and incident management is a cost that doesn’t directly drive revenue, all too many executives will consider it a grudge purchase.

This means that the Splunk AI tools do not take action directly, always confirming via a human what action, if any, should be taken. This approach maximizes the likelihood of the correct action being taken by combining human insight with the scale of machine detection, while minimizing the risk of humans missing something and machines jumping to the wrong conclusion.

Explain Yourself, Machine

Another rule that should be respected by AI systems but all to often is not even considered is auditability. This means the ability for the machine to explain why it did something or recommended a particular course of action. Just like with a human, you can ask the reasons behind choices. This is particularly important when there is a chance of legal exposure or an insurance claim, where every little detail of an incident will be dissected and examined.

The Splunk AI Assistant is exemplary in this aspect. Its main purpose seems to be helping SREs write custom filtering code in SPL2. Most AI-enabled tools which simply belch out a bunch of stuff and expect the human to check if it is correct.

The Splunk AI Assistant uses a conversational approach to go way beyond this. The user can ask it to create a query in plain language (presumably only English at present), and the tool returns the code with a line-by-line explanation of what it does and why. This serves three vital functions:

• Providing an audit history that explains why the actions were taken.
• Allows experienced engineers to spot errors.
• Trains junior engineers in developing SPL2 scripts.

Most of us would say that handling an outage of any kind is a learning experience, with the AI Assistant Splunk can turn any operation, routine or emergency, into an educational opportunity. This helps those who may be experienced operators but lack programming skills to develop an understanding of scripting. And given the shortage of staff in the area, this is a useful tool for skills transfer and onboarding new SREs.

Revenge of the Command Line

I found it amusing that amongst all the gorgeously designed, highly-visual user experience tools offered by Splunk, the latest and greatest tool was essentially a return to a command line. Chat interfaces, however, turn the traditional command line interface on its head as the AI Assistant learns what the user wants and needs, not the reverse. No more figuring out the right command and its options, just use plain language.

AI and ML Transfusion

ML and AI capabilities are being rolled out across security, observability and platform Splunk products. Given the existing ML heritage this is no sudden move to keep the vultures on Wall Street happy, but part of a longer roadmap to AI-enable a broad range of capabilities. This careful approach enhances the product line appropriately where AI and ML will add the most benefit to security and reliability professionals.