How I got into Machine Learning
My ML journey started from a naive interest in applying my math knowledge to something practical. During my studies at the Faculty of Mechanics and Mathematics at Moscow State University, I grew tired of too many abstractions. I was taught a lot of advanced math, but was never taught how to apply it.
That is when I discovered Coursera and Andrew Ng’s ML Course. I discovered it, tried doing it and soon abandoned it: my English was poor, my programming was poor, and so I struggled to learn anything. It took me a couple of years to improve my English (for clarity, I was not motivated by Coursera to do this, but that is a topic for another story), and eventually I re-discovered this course.
Ng used Octave, which was very similar to MATLAB, a tool I was familiar with. He showed a lot of practical examples, one of which was about eBay price-listing prediction, if my memory serves me well. Equipped with knowledge of regression and classification tasks, I started thinking about where I could apply that knowledge. Or, so to speak, with the hammer in my hand, I was on the lookout for nails. For me at that time, the most obvious nails were centred around football, the English Premier League and, specifically, Liverpool FC. It was around 2013-2014, and like other Liverpool fans around the world, I was in a dreamland for a period of time.
I started by dipping my toes into data scraping, data visualisation, data exploration and some simple predictive tasks. I visualised the number of home and away goals scored and conceded by each team across all Premier League seasons, the number of professional clubs registered per country, I tried to find any correlation between whether being on top of the league at any particular month combination from August to December would guarantee you the title at the end of May, I visualised transfer spending and sales. I enjoyed doing all of that; I had a lot of fun, and while not achieving anything groundbreaking (coinciding with the unsuccessful end of the 2013-2014 season for Liverpool), I learnt a lot, and potentially, for the first time in my life, I felt like my math knowledge could have some purpose.
My interactive visualisation attempt. Can you recognise which summer transfer window is shown here?
I continued doing more data science-related courses on Coursera. I stumbled upon Duke University’s series of courses and got introduced to R. I had always hated programming. I had done Pascal, C, C++ in lyceum and university, but I never liked doing that, and I did not think of MATLAB/Octave scripts as proper programming anyway. But it all changed with R for me, as I genuinely enjoyed writing scripts in R. RStudio with its code, data exploration, and data visualisation layout was groundbreaking. Somehow right from the start, I was always interested in visualisation and interactivity. So RStudio and Shiny became my go-to tools. For me it was absolutely mind-blowing to create some interactive HTML pages from R with some simple dropdown menus, where you could select something and see your plots change in real-time. That summer I even read some JavaScript, HTML and CSS books to get a better understanding of what was going on behind the page. There was (maybe still is, not sure) this package in R called Rcharts, and I truly believed that the person who wrote it was a genius.
It is funny how years later, during my PhD days and Skydio work, I did become known as the demo guy: at University I was showcasing real-time dense per-pixel networks to various visitors, while at Skydio I was building a lot of various Streamlit apps to play with models and with data.
After becoming somewhat fluent in R and somewhat familiar with general data science and machine learning approaches, I decided to search for jobs where I could practice those skills and learn even more. I landed a data analyst internship at Svyaznoy Bank - a bank affiliated with Svyaznoy, a major electronics retailer in Russia at that time. I was a part of the team analysing customer data, trying to answer questions such as “if a customer uses our card at a partner business A, which is in proximity to another partner business B, what are the chances this customer might become interested in business B if we were to promote said business?”. It was not really an ML job; we were not building sophisticated models, but I was still glad to be able to apply some of the knowledge I had learnt fully on my own on some real-world data. The team was using Excel and some simple regression curve fitting; I remember how awesome it was when I loaded, processed and visualised the data fully in R, producing an interactive map and some simple clustering analysis on top of that.
As my internship was ending, I learnt that at Svyaznoy they actually did have a dedicated ML team. So I decided to apply for a role there. At that time, it was probably the hardest interview I had ever had. I felt grilled on theory and on practical questions. I thought I totally failed my interview. That evening, I was attending an ML-related talk when HR from Svyaznoy called me. I was mentally preparing myself for a rejection, but, to my surprise, she said that the team liked me and liked my enthusiasm, and they wanted me to join, offering me more than 2x what I was paid at my internship. I was still a full-time student at that time, so that was a lot of money for me.
The team I joined was a proper ML team of those days - these were people with PhDs, published papers, even books written, professors, founders, you name it. They talked about big data, Hadoop, Spark, clusters, training, Python, C++, Scala, and a multitude of other topics that I had barely heard about prior to that. At Svyaznoy, they shipped some huge ML-powered deduplication work, which, when described to me, sounded very interesting and also very convoluted - fair to say, it was my first exposure to the actual complexity and obscurity of real-world data. The project that I was assigned to was on listings recommendations. Once a customer landed on the website, Svyaznoy wanted to display a list of suitable recommendations – personalised if the user was logged in. In other words, a classic recommendation systems task as I know it now. Back then, I knew nothing and I had to learn about recommendation systems, the famous Netflix competition, matrix factorisations and some early deep learning approaches. I was still able to use R for some data exploration work, but I also had to learn Python and Scala (I actually really loved working in Scala, and I do miss it). I also had to learn about Hadoop, remote clusters, and a ton of other things.
I was surrounded by extremely smart and extremely ambitious people. I was learning a lot and I felt like I wanted to learn even more. Combining near full-time work with full-time studies was a difficult period, and as I was approaching my graduation, I started to realise that learning at work versus learning on your own had its differences: on my own, I did not have a focus; one day I could have been learning about JavaScript, the next day – about regression. At work, I had to think about the problem at hand and focus on the tools required to solve said problem. Ironically, while the breadth of knowledge was one difference, so was the depth – I felt like I heard of many things, maybe even used them, but if I had been asked to implement from scratch or explain in more detail, I would have failed to do so. For me, with a rigorous math background, it was important to understand the details and to know what was behind various tools at my disposal.
Hence, I thought that a natural next step for me was to continue my studies, but now in the ML domain, not just math or mechanics. For me at that time, two realistic choices were either Skoltech or HSE, both still in Moscow. I did not get into Skoltech, but got into HSE’s data science master’s program. I declined the offer and instead began looking for an opportunity abroad. Previously, I had already tried reaching out to some professors abroad, but it never came to fruition. This time, though, I did receive responses and two concrete options materialised for me.
One of them was in South Korea, another – in Australia. The Australian opportunity was in the NLP domain, which did look interesting to me and probably closer to what I already knew. However, the application process felt to me a little bit too complicated and too bureaucratic. The option in Korea was much more straightforward – they asked me to participate in a Kaggle challenge. Based on the results of that challenge, where I did well (reaching top-3 %), they invited me to do an internship at their laboratory for 6 months before I could apply for a PhD position.
My internship was on medical image segmentation in collaboration with a Korean industry partner. It was my first experience working on deep learning and computer vision, and I had to rely much more on Python and slowly move away from R. For deep learning, I used Caffe at that time. I remember when the ResNet paper by Kaiming He et al was released, and there was no official code yet; I implemented some pieces from that paper via custom layers, then trained the model on a local cluster on a subset of ImageNet that we had at that time and, in the end, got some reasonable results. I thought that experience was very cool, and it probably played a big part in my interest later to focus some of my time on open-source development, and, specifically, on reproducing and porting popular semantic segmentation networks from Caffe to Python/TensorFlow (which, of course, led to me being known as drsleep on GitHub and, occasionally, in real life).
Eventually, while I was still in my internship, I managed to secure an offer for the PhD program – even without having a published deep learning paper at that time. However, right around the end of the internship, I did write a paper as the first author together with my labmate and supervisor, which we submitted to ICML. I was very stressed during that period and barely slept the night of the submission; I remember how the following morning I took a bus to go to the other part of the city, where I ventured into some fancy supermarket and bought a box of potatoes to celebrate - I do love potatoes!
When I was already a PhD student, we received the rejection news from ICML, which did hurt, but I was probably expecting that to happen as the ICML bar is quite high and the paper might not have been a good fit for the conference in the first place. However, the feedback we received was good and helpful, so we revised the paper and resubmitted it to a computer vision-focused conference – BMVC in the UK. Thankfully, this time around it got accepted, and luckily I was invited to travel to England, coincidentally fulfilling my dream of visiting Liverpool and seeing Anfield in real life.
Obviously, it was my first experience attending an international computer vision conference, so I did not have any expectations and just thoroughly enjoyed my time there. When I got off the train in York and walked to the bus stop, there was another person there waiting for the bus. It turned out that he was also going to the conference, and what was even more surprising was that his alma mater was also Lomonosov Moscow State University (though a different faculty).
I met more people at the conference, and I got this impression that at least in Europe there was this tight circle of folks working on similar topics, and there was a lot of collaboration between them. What I had in Korea was very different from that – there was only 1 post-doc in our lab, and his expertise was not in deep learning or computer vision, whilst my supervisor was split across 10+ other master’s and PhD students. I felt isolated in Korea, and in addition to my inability to learn the Korean language, at times it was hard for me to fit in. I never regretted the year and a half that I spent in Korea, as it was extremely helpful for my professional and personal development, but soon after returning from BMVC, I realised that I was not happy in my studies and I needed to move elsewhere.
This time around, I did know what I wanted to do – deep learning, semantic segmentation; and I knew what I was looking for – a large laboratory, ideally, a SOTA or near SOTA in semantic segmentation, and a country where English was the primary language. Ironically, Australia did satisfy those desiderata of mine, and the computer vision laboratory at the University of Adelaide was the laboratory I contacted this time. As I had previously imagined, applying for a PhD in Australia was (probably still is) a tedious bureaucratic process, further complicated by my nationality. Overall, it took me nearly 10 months from the moment I first reached out to them until I finally arrived in Australia (some of my labmates in China had it much worse, waiting 1-2 years for all formalities to be resolved).
Prior to me joining the Adelaide lab, I had obviously familiarised myself with their research in semantic segmentation, and I was very much looking forward to meeting with and learning from the first author of many of their papers at that time - Guosheng Lin. Unfortunately, when I arrived in Adelaide, Guosheng had already left for his new position in Singapore. Fortunately, though, his knowledge and all subtle implementation and training tricks had been passed on to other PhD students in the lab, who became my good friends and mentors during my first months there.
I would be lying if I said that I had it all figured out about my PhD at the time of my arrival. Even though my time in Korea did help me in narrowing down my field of interest, the concrete direction was still a blank. In my first months, my supervisors tasked me with reproducing and porting Guosheng’s RefineNet training and implementation from MatConvNet to TensorFlow; I went even further and ported it to PyTorch, which was just becoming a popular deep learning framework. With that implementation in place, my supervisor, Prof. I. Reid, asked me to profile the PyTorch inference of RefineNet and see whether I could demo it to some external visitors to the lab. I did do that successfully, but the network was certainly not running in real-time, at least not on some realistic resolution inputs. This did appear to me as a challenge to be solved.
At that time, I was not familiar with almost any of the inference tricks nor any potential frameworks (and I doubt many of the good ones even existed at that time), so I felt like one of the few options available to me was tweaking the model architecture. I wanted to avoid making any changes that would require the model to be retrained, though, so I started looking into existing structures of RefineNet. That is how I stumbled upon a discovery that some building blocks of it were not contributing much to the overall result and could be simply dropped without a significant drop in accuracy but with a significant speed-up factor – this work eventually led to Light-Weight RefineNet, and one more BMVC conference to attend. Naturally, soon after, I extended Light-Weight RefineNet to other dense per-pixel tasks and continued my PhD with a focus on efficient networks for dense-per-pixel tasks - and that was how I finally discovered the direction of my PhD.
As I finished writing this post, memories of those years came rushing back. I always enjoyed mathematics, which led me to attend a physics and mathematics lyceum during the final four years of school, and later to study at the Faculty of Mechanics and Mathematics at university. Even then, though, a PhD was never part of the plan (and, let’s be honest, there wasn’t much of a grand plan to begin with). And while I dreamed of living abroad one day, I never imagined it would actually happen.
Looking back, I realise that my journey would not have been possible without a fair amount of luck. It was shaped by a multitude of opportunities (some that I fumbled and others that I was fortunate enough to seize), as well as by the many people who influenced me along the way. I am deeply grateful for all of it.
I found my way into machine learning because I wanted to do something practical. It all started with a few fun football analytics and visualisation projects that I built simply out of curiosity. I enjoyed every step of that process, and I am glad it all turned out the way it did.
Comments