A few lessons learned from a few years in ML research
1. Project selection is more important than work ethic.
If we work really hard, we can increase productivity by 10–50%. If we choose good projects, our productivity can increase by an unbounded amount. The goal of graduate study is to gain skills and do more unique work, not to produce more work.
2. Work ethic makes project selection better.
Of course, we can’t choose projects well until we are an expert, so we need to work hard to become an expert. Reading relentlessly can help here. For myself, I’ve found it use to explicitly give myself permission to bail on boring papers. I think about reading as a pyramid of commitment:
- 1 full paper per week
- 10 abstracts per day
- 100 titles per day
3. The longer and harder you work to show a technique works, the worse it probably is.
If the method is really novel and useful, it doesn’t take that much work to show it. Be very careful if you are spending a lot of time trying to prove that an idea is novel/useful — the difficult part should be coming up with the idea, not showing that it is great.
4. Invalidate your own hypotheses.
It’s easy to want to show that our idea is a good one by trying to confirm our hypothesis, but it’s no fun to invest a lot of time in an idea that will later be proven wrong. Constantly asking ourselves “what’s the simplest experiment that would invalidate this hypothesis?” can lead us down paths to more interesting and robust conclusions.
5. Document Everything — You won’t remember anything.
Even if you are the only person working on a project, you are collaborating with future you — and past you doesn’t answer emails.
6. Outline the paper first, then fill in the placeholder figures.
Expert researchers know that a paper tells a specific story; novice researchers plunge into the void without a map. Defining the signposts along the way helps us to know whether our results support or reject experimental hypotheses. When possible, experiments should be designed so that either answer is interesting.
7. Three projects at a time.
Being able to switch projects during a difficult period is great for mental rejuvenation. Too many projects and the overhead for context switching becomes unbearable. Personally, I struggle with keeping the number low.
8. Focus on becoming the person, then accomplish the goal.
Instead of focusing on accomplishing a large goal, we should focus on becoming the type of person that accomplishes that goal. For instance, suppose we want to run a marathon. We could either focus on the race itself, or we could focus on becoming a marathoner. Focusing on personal development leads to daily habits of excellence that build into character and make our goals routine.
9. Celebrate the small wins.
If we follow the previous tip, our goals will feel small when they are accomplished because we have already become the type of person that accomplishes that goal. We must take the time to celebrate small wins, because every win will feel small.
10. Be a closer.
Nothing counts until it’s shipped.
