Machine learning (ML), artificial intelligence (AI), and data science are paving the way for progress in different business sectors. AI adoption is in full swing for many organizations, Gartner predicts an increase in 2022 AI software revenue by 21.3% from 2021.
Using and applying these technologies to various use cases, and helping the different departments succeed, lies in understanding the concepts. On the eleventh episode of the Decisions Now podcast, Joshua Starmer, Founder, and CEO of StatQuest, and Lead AI Educator at Lightning AI dives into the nuts and bolts of ML algorithms, updating the technology, educating teams on ML within organizations, and more.
Don’t miss this engaging episode as co-hosts, Rigvinath Chevala, EVS chief technology officer, and Erin Pearson, VP of Marketing get the scoop from Starmer, in the most musical manner. Subscribe to the Decisions Now podcast today, you can find us on Spotify, Apple Podcasts, and Amazon Music among other platforms.
Verifying Your Models and Data
When running different models and different interpretations of data, teams must know to ask the right questions to verify and trust the AI and results, they land on. Starmer sheds light on what it is teams should know.
While there are many things to do when it comes to statistics, ML, and data science that we don’t do anymore.
“When in doubt we can always basically try to calculate error or error bars,” Starmer adds. “We don’t often associate errors with the output, and I think that’s something that needs to come into fashion is people need to be a little more rigorous of with their machine learning.”
He recommends and discusses SHAP values, as they give you a sense of which variables play a significant role in your models. In the past, teams used decision trees which were helpful because of how explainable they were.
“With these relatively new SHAP values and other things, what we can do is we can apply that same interrogation to even so-called black box models, like neural networks or support vector machines and things like that. We’ve got newer tools, we’ve got old tools such as statistics, and we’ve got newer tools like these Shapley things that we could maybe use a little bit more in both cases,” Starmer said.
Knowing the background of data sets, and what the statistics are made of, is critical to understanding the output and then driving decisions through it, Pearson said.
Some questions Starmer said teams must ask are:
-Does the sample size represent the right population?
-Where is the data coming from?
-What do you want to do with the data?
-How much data do we have?
-What do we want the models to do?
Understanding and Updating Data and Algorithms
As important as it is to know your data, and ask the right questions, teams need to ensure they’re updating the data and models, too.
Before a recent trip to Europe, and looking for many international flights, he noticed that all the travel websites had price predictors optimized for North America only, and not for international flights because they probably have collected most of their data from American flights, Chevala said.
“So, every time I checked it just said, ‘It’s going to go down. It’s going to go down,’ but it never went down. I pulled the trigger and bought my tickets,” he added.
Even if the data is not from where it’s supposed to be, it doesn’t relate to what it’s trying to say. Sure, it can be super confident because it’s got tons of domestic data, but still really a terrible model, Starmer said responding to Chevala.
“How do you actually create this consistency of evaluation and how often do you need to change that?” Pearson asked in reference to models drifting over time.
Starmer said the short answer is, that there’s no rest for the weary. Once the models are finished being built and teams have spent months or a year collecting and curating data, it’s not the time to sit back but the time to collect data on how well your model is performing in the wild.
“I would think it’d be a continuous thing where you’d always be evaluating and collecting more data, even though you collected so much data to train that thing, you got to keep collecting it. Just like you said, in order to make sure you’re not drifting and that the model still is doing what you think it is,” he added.
Corporates must realize there’s going to be a drift, and not deploy the model and let it go, Chevala said.
“ML is becoming easier to do and more cost-effective or price the price for doing it is much more reasonable than it used to be. And so yeah, so there’s going to be smaller companies with fewer resources diving into the deep end,” Starmer added. “It’s like a pet, you got to keep taking care of it. Go take it for a walk every day. Not many people think of it as something that requires care.”
Hashtag modeling is never done, Pearson added. jokingly.
Syncing Your Model with Reality
Pearson asked if there was a best practice to tell if your model is actually in sync with reality?
Have a good definition of your models and know your target audience so you can benchmark against it, and find out if that audience is even using the model, Starmer advised.
As of a couple years ago, Thanksgiving sale wasn’t a thing on the e-commerce side, Chevala mentioned. The algorithms picked up on consumer behaviors, like waiting till November to buy say a TV, knowing not to expect high sales in Sept. or Oct.
“At some point the behavior as a result of the algorithm being used, that needs to be fed back, right? Because the behavior changed as a result of that,” Chevala said.
Calling it Quits: Sync or Sink
Companies and teams invest tremendous amounts of time and resources when building their models.
“It’s great if you achieve the sync or the reality with the model. What happens if you don’t? What’s the opposite? When do to kill it or not use it?”, Chevala asked Starmer.
“Before killing the model and walking away, teams should ask themselves if it can be repurposed in a different, yet useful manner,” Starmer said. “If your target audience originally was adults, but it’s only kids using it and they keep returning, well maybe you’re like, okay, well this is for kids.”
Educating Teams on ML
You can apply that concept to most things in business, including change management with organizations and training your employees, Pearson said.
Starmer said he’s now working with companies to train their management in becoming more data science savvy and learn about ML.
The idea is, you can work with the people you have, you don’t always have to hire from the outside and you can, if people are interested and they’re willing to gain some new skills that they didn’t have before it, he said.
When teaching, if talking to executives, he focuses on educating them on asking the right questions, and gets more technical with smaller, specific teams, he said.
“The idea is to build people up from the bottom and just to make very little or next to no assumptions about what they know and just build them up until they can tackle whatever they need to tackle,” Starmer said.
Listen to this episode of Decisions Now podcast featuring Joshua Starmer today to learn more about Machine Learning. Subscribe today!
