Questions You Should Ask Before Starting Your ML Project
Guidelines and questions to hedge the risk associated with an ML project
The upside that ML models can learn from data without explicitly defining a wide range of unmanageable instructions opens up the doors for a broad range of applications. On top of that, there is a sentiment that VCs and Angels pounce on “ML-powered” organizations which clouds our head from seeing through the risks associated with ML projects.
Another impediment in evaluating a solution is when technical founders view their “interesting” ML approach in a vacuum irrespective of how appropriate that approach is for the product.
There is no surefire process to predict the success of an ML project but there are guidelines and practices that can help in reducing the associated risk and that is the sweet spot where this week’s newsletter fits in.
Why am I writing about this?
Think through before dive deep!
I’m working on three ML Engineering projects at the time of writing this article and choosing the project was the single hardest thing I’ve had to do as of now. I went through dozens of articles and books learning about design patterns, guidelines, and the right questions to ask to better define the ML problem.
Here’s what I’ve learned:
2-Step approach to frame your ML problem
If you want to evaluate an ML approach for your product, you should consider following these two successive steps:
Define your product goal in an ML paradigm: We start thinking about a product by finding what value it would generate for the users. The ML paradigm works differently. With ML problems, you are basically trying to learn a function from data. For example, you want to learn which segment of customers like to see political news based on past interactions. There are more than one ML formulations for one product goal.
Evaluate the ML-feasibility: As we all know, building an Image classifier has become a matter of hours now but the same is not true for creating a completely open conversational chatbot which is an open research problem. The feasibility of an ML problem is evaluated by looking at the two main aspects: data and models.
Is the problem supervised or unsupervised? Most of the time, it is better to start with a supervised problem as it is easier to validate the model using the available labels.
Questions you should answer to lay out your ML project
Following are the five(can be more based on your use case) questions that you should answer before you kick off your ML project:
Q1: How will I scope out this ML project?
Firstly, get rid of any bias of using ML in your solution. The goal is to use the best tools to solve a problem, and only use ML if it makes sense.
Let’s say you wanted to predict a user’s behavior on your application to suggest to them the right TV show/Clothing line/Groceries or whatever your product offers. Yous should start by discussing the product goal and the type of modeling it would need. This should include discussions on the system’s ability to handle ML failures gracefully.
An important metric to take into account is the confidence the chosen model has in its prediction. Based on the confidence score, you could then formulate your suggestions differently. If you have a good confidence score of more than 90%, you display the suggestions prominently; if it is between 50–90%, you still display it but with less emphasis, and you do not display anything if the confidence is below 50%.
Q2: What should be my main area of focus in this project?
The question requires you to do a risk-reward analysis. You must learn the cost of the prediction/inference. Then, you should find the impact bottleneck, the main pain point that could add the most value. Oftentimes, you’ll find that you are not working on the right problem or if you are running a team, you are not at the right growth stage to take it up.
A common issue that pops up is choosing an overly complex modeling technique. The best workaround is to replace the model with something more explainable and debug the whole pipeline. A common finding is that even if the accuracy of your model is state-of-the-art, your product is dead because of the overheads that come along with the model.
Q3: What is the quickest way to test the feasibility of the project?
The goal of this whole exercise is to come up with a plan that should de-risk your project somehow. You need to get to the stage where you are able to run as many iterations as possible in unit time.
The best approach is to start with a “strawman baseline” to evaluate worst-case performance. This could be simply suggesting the latest action of the user for our previous example.
Doing this exercise will tell you how annoying would your model be to the user if the predictions/suggestions were wrong? Assuming that your model could not do much better than the baseline, would your product still be valuable?
For example, in natural language applications serving summarization of text, simply extracting the top keywords and categories is enough to serve most users’ needs.
Q4: What is the value proposition of my ML solution? How do I identify the impact bottleneck?
To figure out the value proposition, imagine that the impact bottleneck is solved and you are about to announce/launch the product in the market. Compose your social media post or press release before you even start the project. In the process of writing that post, you’ll realize the true worth of that project. This puts the value of the results into context based on the effort.
Q5: How do I decide which modeling technique to use?
The first block in the process of choosing the modeling technique is looking at the data yourself. Spending some manual hours to look at the inputs and outputs of your model. Skim through a bunch of examples to see if anything looks weird. The mantra is to put in a few hours doing the job yourself before you ask a machine to efficiently produce the output.
This exercise helps you think of good heuristics, appropriate models, and ways to reframe the product. If you sort instances in your dataset by frequency, you might even be able to quickly identify and label 80% of your use cases.
The last step is to have a diverse workforce that looks at the results. This helps in catching instances where a model is exhibiting discriminative behavior, such as tagging your friends as a monkey.
Interesting read of the Week
We are going through grim and testing times in India. My prayers go out to all those who are battling this brutal curse on humanity and health workers who are working 24x7 to get things back to normalcy.
So, I was going through some scholarly articles to study the correlation of Vitamin D with the inflammatory markers of COVID-19 and I came across this scientific report on nature.com which will give you some insights into how vitamin D levels affect the severity of disease(s) that you might develop as a result of the infection.
Here you go: