Building a successful, self-managed machine learning team

Maciej Adamiak — SoftwareMill’s ML Lead, Senior Software Engineer, Machine Learning Engineer with a passion for data engineering, machine learning, and research, expanding his expertise in remote sensing; co-author of a number of scientific papers within Earth sciences and machine learning.

What makes a good Machine Learning Engineer? How to build a self-managed, interdisciplinary team that delivers? I asked Maciej Adamiak about managing ML projects, finding the right talents to work with, and building relationships with coworkers and clients.

Can you share some details about the machine learning projects that you've worked on so far?

One of the most interesting commercial projects I have participated in is machine learning for the biomedical sector, lesion detection. In this project, using various types of body scans, we detected where cancerous changes were. I had an opportunity to work with experts in this difficult domain. From my perspective as an ML engineer, this project is fascinating, and it’s great to take part in the creation of a meaningful solution that helps people. I also work on multiple non-commercial projects, and there are a lot of interesting ones, but maybe I'll just tell you about a few of them.

In recent years, I have been focusing on research for my Ph.D. thesis related to utilizing deep learning in Earth Sciences. For this, with different research teams, we selected research problems that are interesting from a geographical point of view and combined them with machine learning. These projects are really developmental and, since they’re part of basic research, give a greater understanding. They’re also enjoyable — watching satellite pictures of the Earth is calming.

The first scientific project focused on the use of convolutional neural networks in the classification of areas with land abandonment. Land abandonment is often associated with the imagery of a post-agricultural wasteland, but every such area undergoes plant succession, that is: nature begins to reclaim an area once farmed and then abandoned. Our model showed that in the Łódzkie Voivodeship, there are many places where this succession is already advanced; these places slowly turn into areas similar to meadows – in the future, they will start to turn into young forests.

Our subsequent projects were also related to forests, for example, we were looking into whether it is possible to detect forests from satellite images. This is a surprisingly non-trivial problem. We came up with the idea that we would create a machine learning model that would take all satellite photos from Poland and draw us a map of the forest in the entire country. This model is able to tell what is and what is not a forest and to delineate the area very precisely, which means it successfully designates these regions and helps achieve the results much faster than before.

My third scientific paper within this area of research focused on the use of generative adversarial networks, that is networks that generate and synthesize images. In synthesizing the images, I got the idea that maybe it is possible to synthesize satellite images. I was thinking: we have some photos, reference data, e.g. a forest area with a stream, and we want to get 100,000 photos similar to those but non-existent; realistic, but synthetic For example, they might be needed for a different machine learning model, where more information is necessary than we can acquire at that time — after all, the area of the Earth is finite, and some phenomena have a low frequency of occurrence. So data can be synthesized and we can create artificial sets. We managed to achieve this goal and, in one of our research projects, we have an algorithm that synthesizes photos and these synthetic photos are, by and large, indistinguishable from real ones.

I think that remote sensing has a chance to develop a lot in the coming years because machine learning is so popular now, and, in this domain, you work with tons of data. These projects are very interesting from my point of view as a computer vision specialist because it is not recognizing cats, it's detecting very complicated spatial features, natural phenomena. What is more, one can verify the model in a real scenario to see how it actually does work. When we examined land abandonment, we selected test samples and went out into the field, checked everything manually and it turned out that the model is working properly. With our synthetic data, we sent it to experts who deal with geographic information systems, asking them to tell apart the real images from artificial ones. We sent the images over just for comparison and were pleasantly surprised with the results. It's amazing when you can test your solution in practice.

How do you approach project management? Do you think it's difficult?

Management is not easy. The word may be easy to understand, but it is a difficult discipline. After years of practice, I am sure that it should not be underestimated at any stage. You can be an excellent programmer, but without management and without a good plan, you will easily lose track.

I studied IT management for 5 years and worked for a long time as a software engineer, and in the first stages of my professional career, I had such an impression that it would be easy, that it all just happens along the way. But no, it doesn't just happen, you have to put in an effort to ensure that the project is well-run. It is complicated and requires specialist know-how, and a lot of experience, so it's not easy at all.

How do you run a project in a team where there is no project manager?

Our team members have an "inner locus of control" and don't need outside management, which I think is very important. Having a certain vision, going to a given point, we make great use of self-management. I think this is an interesting approach because when I worked in corporations, plans came from above, and now there is a general outline of what we want to achieve, while the plans come from us and are confronted with the inner need of another person. It's as if we're protons and meet on the same path: we crash around for so long until we finally reach a consensus and grow together as a team. This path takes a tough start, but it works.

I have very good memories of working with Paweł, our senior backend engineer, on one of the projects. In the beginning, we basically didn't know each other, I was new to the team, and we needed some time to work out our ways. Paweł is an amazing specialist, I also have my experience, so it was quite intense, and once we reached that understanding, figured out what to do and who is responsible for what, it was an indestructible team. The platform we built is really great, programming-wise: very good code and architecture, backed by great ideas that came from the client. We also needed to work out efficient ways of working together with the in-house team members, and as a result, we became part of the client's team, and they became a part of ours.

In the beginning, the client came to us with an initiative, we turned it around in all possible ways to find the best solutions, and we went hand in hand with the client in the process. When specialists come together as a team, everyone approaches the project professionally and has an inner drive to do it, the client receives more than our hard-done work, we go beyond that because we are then a part of this project. The motivation to solve difficult things is a catalyst for this work. In that particular project, we joined a strong client team, found this shared motivation and we all understood that we were part of something important, and we were doing it for another human being.

Just looking at the example of this project, I see that the process of management is difficult because it is not only skill management but also how a person approaches the project internally, that they choose something that they can relate with, what they find interesting. There are also all organizational issues: finances, code quality, tests, deployment — that's a lot of variables and it's not easy to control them all.

Then there are the "soft" issues like motivation, communication — they play a significant role, too. We all have this thing called the comfort zone, it is not my favorite topic to talk about (don’t you hate being told to get out of your comfort zone? I know I do), but it seems to me that when you enter a project and want to do it as well as possible, you do it according to your standards and to what's known to you, so it's somewhere inside this comfort zone. It's also something worth taking into consideration, when there's a leader, they will naturally focus most on what's their domain.

In small projects, everyone can be a leader within their area of expertise. If we work in a 5-person team where everyone has their domain and that's communicated clearly, it's not a barrier, rather an advantage. We all had our perspectives and priorities:

• For the client, it was very important that the project reached production.
• Another team member wanted the product to be explainable, to build a simple, good solution.
• For the DevOps engineer, it was important that it can be swiftly deployed.
• To Paweł, it was clear that the code must be readable, tested, and made in accordance with the highest standards.
• For me, the most important thing was that it should be an interesting, non-trivial architecture that precisely solves the client's problem.

When each team member speaks openly about their priorities and exposes their ideas to the criticism of others and is not afraid of that criticism - then there's no need to bring in an additional person to watch over the process. The concept is what drives the work. It's key to talk, discuss ideas, exchange thoughts, and the project benefits from that. But I think it's a solution for small teams because a large team needs someone to tie it all together. 5 people can talk about the entire infrastructure over tea, but a team of 100 can't do the same thing.

More in the 'Managing Machine Learning Projects' series:

Does this mean that you do not need any project management system, but rather should adapt to the client and their practices?

The most important thing in our work is that the client feels good with us and is satisfied with our work. If we had a fixed management system, we would have to clash it against the fixed system of the company we work for. We want to write code and build good solutions, not to focus on how to integrate two ways of project work.

When we work in a team extension model, we also learn the client's organizational culture. This is something that's a part of the package, we begin to understand how the client works, and even if we have some good practices, we implement them when appropriate or try to show the client how we work. I think this is a good approach because the client pays us for building the software, not for coming and changing the project management system. They want us to be flexible in what we do. Each of us has several years of experience, and in various projects, we encountered various types of project management. Now, we're able to make use of each of them when they suit the project’s needs. It comes with experience, and it works for us because we have high seniority in the team. Our role is to enrich the client's project, not change it, and that is the most essential thing.

Does it ever happen that you have a completely different approach to management and work organization than the client?

Sure, it does happen sometimes. But you also have to fight such things internally because nobody said that what I am doing is correct. I don't believe that universal solutions exist. Absolutely not, every solution is different. I like Scrum and iterative work but that doesn't mean that it will be a perfect fit for every project, or that another solution won't work for a given client. You always have to be able to confront your views with what is happening around you.

Adjusting to changing conditions is the essence of my work: when the client tells me that we'll do this and that, I'm allowed to have some doubts, ask questions, or even share criticism. But this is also a discussion and if there is any tension in the project and I see that something is not going as it should, being a professional, I have to let other team members know that I see a problem. Above all, you have to communicate — if we're to build something meaningful together, then everyone has to be on the same page. I believe that the right to doubt is important in the work of a programmer: to ask questions, reevaluate what you know, and learn business empathy. Think about it this way: if we enter a large corporation, we will not do everything as we wish, because the point is to implement the project efficiently, not turn it around — although sometimes, you can meet halfway.

Are there any elements that are necessary for every project? You've already mentioned communication: it would mean that you also have to start with an in-depth conversation and not jump into the project straight away, right?

Sure. It's the code that does the job but what that job is — that's a result of some need that has to be defined. It's fun to understand the motivation behind the project, and understanding the project's goal is crucial: to grasp when, for whom, and why it was done. It is also important to understand the person you are working with: their experience and background. To start talking about code and solutions, you have to speak the same conceptual language. Each of us works in a different place, sometimes there are cultural differences, we received different education, we read different articles and books. It's good to sit down with this person and just talk, human to human, to find out who is working on the other side. Just issuing and assigning tickets dehumanizes the process.

In addition, you must also remember that all of us make mistakes and these have to be resolved. To do that quickly and efficiently, you can't look for who's to blame, you need to understand the other person and why it happened. In the beginning, I like focusing on learning more about the team, the company, the product, and only then can I go into how it's written. It's not just code that makes a product great.

You may also be interested in:

You also mentioned that there are projects in which everyone is a leader in their domain. Isn't it then easy to forget about this aspect of team building and talk only about tasks, report what has been done?

I am a fan of teams where people are, let's use an RPG term, "multi-class". It means that, for example, I am a programmer but I also work in data science. My client codes too, he works in data science, but also has a managerial role. It's a good idea to make sure there is a common denominator so that there is no isolation in tasks. It was the same in our case — Paweł is quite proficient in DevOps, so he spoke to the DevOps engineer from the client's team. It is important that any ticket to make is not just for me, that there is also someone else who can solve it. This approach really works great for us. In another project, where all team members were ML engineers and data scientists, it was noticeable that the team was missing software developers to bring in a different perspective, too.

And how do you choose people for the team and allocate them to projects? How do you know that in a given project, they will find this common denominator with the rest of the team?

Everyone has some areas in which they want to develop and, above all, we are looking for projects tailored to these interests. Because our ML engineers have a software engineering background, they are “multi-class”: they understand data science, statistics, modeling, but they are also great programmers. So whatever task they're given, they will always find a common language with that other person, either focusing on their data science or software engineering experience.

For example, I enjoy working on projects with other scientists, but in addition to that scientific part, I am also a software engineer and ML engineer, because that's my professional experience. We understand each other perfectly with my colleagues at the university because we are geographers, and although e.g. I do not understand all the nuances of socio-economic geography, in which I sometimes work, we have the same vocabulary. It can be said that we received the same education. Projects go better if there is someone in the team who understands everyone.

I am thinking about the need to have a project manager in the team because companies often assign their own PM to the project, even if the team is small, and put emphasis on their own good practices. From what you say, however, it seems like such a PM will not be needed sometimes, and that you have to be flexible in these practices.

Precisely. For small projects, this is really not needed, and self-organization works then. We are a healthy example of this. I also like working in smaller teams, and when you work with really good specialists, you can even support large projects with such a team.

Let's get back to the topic of building a team for a moment: are you able to assess whether someone is a match during the interview? What skills are you looking for in people who are about to work with you on ML projects?

The first project is the best test, but well, in standard processes, it's a little too late. Recruitment interviews, however well conducted, screen out people with whom we do not want to work, but that's not the goal, really. When it comes to hiring new people, we simply hire the best person who applied to us, and it does not have to be the best person for a given project, because it is not so easy to assess it so early on. Of course, we are always looking for experts, but what kind of person will find us depends on many things: who recruited, how the recruitment campaign went, how good the HR processes are, what brand we have. That's a lot of different factors that influence who's responding to our job postings.

On the other hand, I believe that everyone should be given a chance: if someone says that they are good, maybe the recruitment process didn't go ideally, but you can see that there is some chemistry, that you can chat with this person, I think you should give them a chance. We can reject a candidate because, for example, they are not good at some algorithms, but this way, we could also reject good people who, even if they have some shortcomings, are able to make up for them quickly. This first project is an opportunity, a trial period that can dispel any doubts. The problem with recruiting is that you can make a poor impression because you have a bad day, you're stressed, or the questions didn't work. Is there anyone who has done well on absolutely every job interview? I don't think that's possible. I would like the recruitment processes in all companies to be based on checking people in action and not asking them tricky questions.

This reminds me of a Digital Poland report where many companies indicated that they have a problem with employing ML specialists. It is not easy for them to verify the candidates well, to assess their skills. It is also difficult because machine learning projects are very diverse. What skills do ML engineers need to have?

It's best when someone has references, a nice Github, maybe some scientific papers that you can read — then it's easier, but that’s a situation when you employ an expert. It's a tough situation for entry-level jobs. For me, the most important thing about team members is that they can learn fast and do it on their own. A very important skill is that a person is aware of their shortcomings. If you know what you are missing and can make up for it efficiently, this is already a big advantage. After all, no one is born with the knowledge of all programming languages. Understanding what you don't understand is important, and when you add the willingness to learn, it solves the problem. I am an advocate of the approach that everything can be learned, as long as you are motivated to do so.

I guess the main focus is on technical skills anyway? It's something that's in the CV, even a recruiter with no tech know-how can somehow check it, then probably there's some recruitment task. What you are talking about requires more involvement in this process, doesn't it?

Yes. I would prefer such a process because there are people who are, for example, great programmers, but do not understand how machine learning works. On the other hand, there may be someone who understands neural networks very well but isn't proficient at coding. You can learn that and catch up. Machine learning is an open and accessible discipline.

We could test who knows what algorithm, but this is not the point at all, it's about whether you are able to materialize the idea you have in your mind. This is a very important skill in the work of a programmer. Are you able to translate the idea into code? Abstract thinking needs to be brought into the IDE. This is the set of skills that I am looking for, that's why I am looking for programmers — they only need to understand ML, and this is no secret knowledge. On the other hand, if I found, for example, a good statistician who's not yet great with coding nor deployment, it's also something that they can catch up on. I'm not looking for the full package, just someone with talent and motivation. You can always work on the rest.

If we're dealing with a non-trivial problem, how can we prepare the client for the risks associated with a research project? How can you introduce a client who has not dealt with similar projects yet to this process?

We have implemented the workshop format, i.e. a low-budget way to start a project. If the client knows very well what they want, they are able to fill us in on it quickly, already during the first technical calls. But if we have a client who has to work through a few things, it is better to meet and find out what they need done, check feasibility, talk about solutions and alternative approaches. This workshop is also an important stage for us so that we can understand what this project is supposed to do, what is the motivation, what is our goal. I believe that it is important to discuss and analyze everything first, and not to enter the project right away, because it is beneficial for both parties: the client learns what they can expect and what the process looks like, we get a chance to gain a more thorough understanding of their business, goals, and needs. I really like this workshop concept.

Does this also help to regulate expectations, to let the client know what they can achieve and what's impossible?

Yes. It is important to be well prepared, to have a look at some literature, to try to find out a little more about what the client's needs are and whether they are feasible and within what timeframe — because we have to let the client know when and how we can build their solution.

And how do you estimate that?

After the workshop, we make rough estimates of how long the development will take. During the workshop, we present the concept of what we can do. We tell the client exactly what process we need to go through, and it is often a research process to validate whether a solution is correct or not. If the client comes to us with uncertainty about whether this problem can be solved, but wants to invest in it, it is better to first check if something will work, do workshops, define the scope of work, prepare a set of hypotheses that we have to confirm or reject, and build a PoC. This is, in a way, a trial period for us and the client's idea and a great solution for companies that do not have an exact concept for the implementation of the project. Of course, if the idea is great, it will most probably all work out, and if something is wrong, then it’s better to fail fast. You need to check a concept out efficiently before putting in more money. I think that’s a fair approach.

For more information on our approach to Machine Learning development, visit this site.