Aurich Lawson | Getty Images
I’m not a data scientist. And while I know a Jupyter notebook and have written a fair amount of Python code, I don’t claim to be anything close to a machine learning expert. So when I ran the first part of our no-code/low-code machine learning experiment and achieved an accuracy rate of over 90% on one model, I suspected that I had done something wrong.
If you haven’t followed along so far, here’s a quick recap before heading back to the first two articles in this series. To see how far machine learning tools had advanced for the rest of us, and to redeem myself for my machine learning assignment last year, I took a dataset of a well-worn heart attack from a University of California-Irvine archive and attempted to outperform data science students using the “easy button” of Amazon Web Services’ low-code and no-code tools .
The purpose of this experiment was to see:
- If a novice family member could use these tools effectively and accurately
- If the tools were more cost effective than finding someone who knew what the hell they were doing and handing it to them
This is not exactly a true picture of how machine learning projects usually go. And, as I’ve found, the “no code” option offered by Amazon Web Services (SageMaker Canvas) is meant to work hand-in-hand with SageMaker Studio’s more data-scientific approach. But Canvas surpassed what I was able to do with Studio’s low-code approach, though probably due to my unskilled data handling hands.
(For those who haven’t read the previous two articles, now’s the time to catch up: here’s part one and here’s part two.)
Evaluation of the robot’s work
Canvas allowed me to export a shareable link that opened the model I created with my full build from the 590+ rows of patient data from the Cleveland Clinic and the Hungarian Cardiology Institute. This link gave me a little more insight into what was going on inside the very black box of Canvas with Studio, a Jupyter-based platform for doing data science and machine learning experiments.
As the name suggests, Jupyter is based on Python. It’s a web-based interface to a container environment that lets you spin kernels based on different Python implementations, depending on the task.
Examples of the different kernel containers available in Studio.
Cores can be populated with any modules the project requires when doing code-centric explorations, such as the Python data analysis library (pandas) and SciKit-Learn (sklearn). I used a local version of Jupyter Lab to do most of my initial data analysis to save AWS compute time.
The Studio environment created with the Canvas link included pre-built content that provided information about the produced Canvas model, some of which I discussed briefly in the last article:
Expand / Details of the model of the best exhibition of Canvas in Studio.
Some of the details included the hyperparameters used by the best-fit version of the model created by Canvas:
Expand / Model hyperparameters.
Hyperparameters are adjustments AutoML made to calculations by the algorithm to improve accuracy, as well as some basic maintenance: the SageMaker instance parameters, the fit metric (“F1”, which we’ll discuss in a moment) and other inputs. These are all pretty standard for a binary classification like ours.
The model overview in Studio provided basic information about the model produced by Canvas, including the algorithm used (XGBoost) and the relative importance of each of the columns valued with something called SHAP values. SHAP is a really horrible acronym that stands for “SHapley Additive exPlanations”, which is a game theory-based method for extracting the contribution of each data feature to a change in model output. It turns out that “maximum heart rate achieved” had a negligible impact on the model, while thalassemia results (“thall”) and angiogram (“caa”), data points for which we were missing significant data, they had more of an impact than I wanted them to. . He couldn’t drop them, apparently. So I downloaded a model performance report to get more detailed information on how the model held up: