> "It's a catch 22 with ML though. What you wrote is completely true however with ML you cannot say "We will get to 98% precision and 92% recall by Q4"."
This applies also to ML - it applies to all tech projects, though yeah, it's harder in ML. But not figuring out the intermediate products is not an option though - your stuff will get killed prematurely if you don't.
The trick with ML is not to promise "98% precision and 92% recall by Q4", it's to figure out what kind of product is shippable with lower precision and recall. Or perhaps a stepping-stone model that allows some simpler use case, but gives you progress towards the greater goal.
It's always case-specific, but as a ML team you do need to figure out what your intermediate checkpoints are. You need to demonstrate not only progress, but that your progress is contributing to the company's goals.
I think of it as building a staircase. If we need to hit 98% eventually, we need to make certain progress now. It is possible that a new breakthrough could appear in 2 years to spike that, but the safer assumption is that our near term progress will harvest the low hanging fruit.
Therefore, intermediate and measurable milestones are to derisk something. Even if you are doing a moonshot, there are still steps you can identify. The namesake, Apollo, didn’t try for the moon on the first launch!
Exactly this. Intermediate shippables are derisking. There are a few things that this strategy represents to executives:
- "You have already received benefit X for your investment in our team, and X has been well worth it. Continued investment in us will yield benefit Y."
- "Even if the project is terminated early, or later stages become blocked due to business or technical impracticalities, the company will have gained a tangible benefit already in the form of X. There is a payout, even if it is not the full thing we wanted."
- "The intermediate products validate the technology/product/business model, such that every incremental deliverable means an overall reduction in risk to the final goal."
100%. And the sooner low level folks like me can provide this signal, the sooner the execs can make decisions. That saves cycle time and is the real secret sauce for growth.
That’s why my favorite interview question (I’ve done hundreds of interviews) is “tell me about a time you cut corners on a project”. My role as a data person is to provide enough signal as early as possible - not necessarily to give a precise answer to everything we want to know.
Right but the "how" is the tricky part. For example, wrt to ML, what is the milestone and how do you know you will hit it by a particular date? Very difficult to say with ML.
This is where smart analytics teams are super useful. It is a really hard projection estimation! Even understanding what the F1 score should be to be "good enough" is nearly impossible, much less understanding what it takes to go from "where we are" to "good enough".
What you can do is estimate something like "we think we need $measurement_value on $metric by $date, we are at $current_value. If we're not at 75% of the gap in 50% of the project time (when we have low hanging fruit), we cancel."
Agreed, it's very hard, I'd like to see how many orgs have actually done this successfully and what those shippable intermediate products actually were.
This applies also to ML - it applies to all tech projects, though yeah, it's harder in ML. But not figuring out the intermediate products is not an option though - your stuff will get killed prematurely if you don't.
The trick with ML is not to promise "98% precision and 92% recall by Q4", it's to figure out what kind of product is shippable with lower precision and recall. Or perhaps a stepping-stone model that allows some simpler use case, but gives you progress towards the greater goal.
It's always case-specific, but as a ML team you do need to figure out what your intermediate checkpoints are. You need to demonstrate not only progress, but that your progress is contributing to the company's goals.