Before You Invest in an ML Project - Case Study

More and more companies are beginning to consider machine learning – whether to increase process automation, improve user experience, or keep up with the competition. An idea appears – sometimes inspired by a conference, a conversation with the tech team, or a signal from the market that "others already have something with AI, and we don’t yet."

At that point, there’s often a temptation to jump straight into execution: hire a team, pick a tool, start training something – just to "see how it goes." From my perspective, as someone who has been working on machine learning projects for years, that’s a straight path to frustration and a burned budget.

Machine learning is not a plug-and-play technology. Its effectiveness doesn’t come down to the model alone but to the entire environment: the data, processes, integrations, and, above all, the correct problem definition. That’s why a machine learning workshop is not a formality but a critical project phase. As a proof, take a look at our success story from one of our projects, which started as an ML workshop.

The workshop is the moment when expectations meet reality:

• We confront the idea with the actual historical data the company has (or doesn’t have)
• We verify whether the problem the company wants to solve makes sense from an ML perspective
• We define how to measure success, both technically and from a business point of view
• We present alternative ways of reaching the project goal and estimated budget; sometimes simpler, cheaper, or more scalable
• We define scope, timeline, and costs - so you can evaluate ROI and verify if the project fits your budget

This blog post explains how we ran a tailored machine learning workshop for a client building a data extraction platform for accounting professionals.

In the first part, you'll learn about the client's business context: who the client is, what problem they wanted to solve, and why they reached out to us. Next, I describe how we structured and prepared the workshop: from designing the template to drafting a realistic project timeline.

We then show how the workshop unfolded in practice: what data we reviewed, which technical and business constraints surfaced, and what insights we gathered about the users and platform. Finally, we explain how this input shaped the final proposal, including cost estimates, success metrics, and a deployment-ready roadmap.

If you're wondering what kind of value a well-run ML workshop can bring, and how it directly translates into smarter investment decisions, this article gives you a complete, real-world example.

It is a long read, so grab a coffee and set aside about 15 minutes. It’s a small investment but definitely worth your time.

Business context: client, need, technology

The client

We were approached by a company developing a platform for accounting professionals. Its core value lies in automating the process of extracting data from invoices and feeding it into ERP systems or finance management platforms.

How did they find us?

The initial contact came through a team of developers from SoftwareMill who had been working on the platform's backend for several months. Their work was well-received, leading to team expansion and broader responsibilities. Pleased with the collaboration, the client decided to tap into our artificial intelligence expertise to assess feasibility and potential ROI.

The problem to solve

The client had many ideas for using machine learning but hadn’t previously launched any initiatives in this field. The most pressing issue identified was data extraction from invoices. The current legacy AI system only supports documents in one language, while expanding to multilingual support is a key business requirement.

An example of the system that performs data extraction from the semi-structured documents. Source: Microsoft.

The legacy system is challenging to maintain, requiring significant effort to add minor improvements. The goal is to replace the old solution with a new one that leverages modern machine learning technology.

From a machine learning engineer’s perspective, this is a classic key information extraction problem from visually rich documents. I worked on a similar project two years ago for another client, so I already understood the situation and constraints involved.

How we prepared for this ML workshop?

Questions

Ahead of the workshop, I prepared a list of questions for the client. The questions were designed so that answering all of them would cover roughly 50% of the workshop’s goals. But realistically, most clients can't provide complete answers up front - and that’s fine. The point of the list was to prompt internal discussion and reflection before we met.

Data presentation

I asked the client to prepare a few representative samples to illustrate the problem we wanted to solve. This is usually the most engaging and insight-rich part of the workshop – and often the most time-consuming.

Key stakeholders

Last but not least, I recommended what kind of people I would need to conduct the workshop on the client side. First and foremost, a key project stakeholder with decision-making power who understands the limitations and motivations for introducing artificial intelligence from the business side.

Equally important is a person who knows the current shape of the platform, understands how data flows through the system, what happens to it, and what goes in and out of the system. And ultimately, a person who understands how the legacy system used for document processing works.

How to prepare for a machine learning workshop?

If you plan to work with an external ML team, the outcome of your workshop will heavily depend on how well you prepare. Here are the four key areas worth thinking through in advance:

1. Define your budget and cost expectations

AI projects require clear financial boundaries; both in terms of initial investment and long-term operational costs.

• What’s your project budget for prototyping or an MVP?
• What OPEX level is sustainable once the system is in production?

This clarity is essential for selecting the right architecture. Whether it’s SaaS, a managed service, or a custom in-house solution.

Be realistic: artificial intelligence is expensive. Custom ML solutions have infrastructure, engineering, and maintenance costs that can scale quickly. If the business case isn’t clear, the investment won’t justify itself.

AI should be implemented because it solves a real problem, not because it’s trendy.

Start by validating:

• What’s the tangible business impact?
• What cost or time savings are expected?
• Is this solving a bottleneck, enabling scale, or unlocking new capabilities?

With these answers in place, resource allocation stops being speculative; and becomes a matter of ROI planning.

2. Define what a "good performance" means to you

Machine learning is never 100% accurate. But what level of accuracy is good enough for your product?

• Is achieving 99% precision a must-have, or is 80–85% acceptable?
• How costly is a model’s mistake in your business context?
• Are false positives or false negatives more problematic?

Clear expectations around performance help align technical solutions with business priorities, especially when trade-offs are involved (e.g., between cost, speed, and accuracy). It's one step towards risk management planning.

3. Identify technical and business constraints

Some constraints can rule out entire classes of solutions; it's important to surface them early.

• Do you need real-time responses?
• Do you operate in a multilingual environment (e.g., Europe, Asia) and require language support beyond English?
• Are there regulatory or security requirements (e.g., on-premise deployment, GDPR, ISO)?

Being upfront about these limitations helps the ML team focus on the right design space from the beginning.

4. Prepare your product context and sample data

To make the workshop actionable, prepare a simple demo or walkthrough of how your platform works.

• How will the ML model fit into your product’s workflow?
• What are the real-world data examples, user actions, or decisions where ML would provide value?

Providing access to sample data points (even anonymized) and showing the product in context helps uncover edge cases, integration constraints, and user experience considerations that affect the design of the solution's design.

Project scoping: from raw data to data-driven decisions

We scheduled the workshop, an element of the project planning phase, at the client’s office. Once on site, we got straight to work.

1. Kick-off

I started with an overview of how ML projects differ from traditional IT initiatives. I explained what makes them unique, what to look out for before and during implementation, and what our methodology looks like - from workshop to delivery. I then walked the team through how our engagements typically unfold.

2. Sample data

After the intro, we moved on to data analysis. The client’s team shared sample documents and described what wasn’t working or what they expected from the machine learning model. In my head, I was already spotting hidden patterns and edge cases I’d encountered in similar projects – those tricky details that are easy to overlook but can make or break a machine learning system. During the Q&A, I collected more requirements and wrote them on the Miro board for later synthesis.

3. Data volume

During the workshop, we discussed how data flows through the system, including the volume of documents processed, how and where the original data is stored, and who is responsible for managing and maintaining it. We also clarified where the data is accessible, which teams or roles have ownership, and what implications this has for training, monitoring, and maintaining a machine learning solution. These conversations were crucial for identifying integration points, potential bottlenecks, and data governance requirements.

The monthly data volume was substantial, several million new documents, more than enough to fine-tune a robust machine learning model. The data is stored in the client’s cloud environment and is fully accessible, so there are no anticipated issues with exporting it for training, evaluation, or integration purposes.

4. Platform insights

Once we had a feel for the input data, we shifted focus to the platform itself. That’s when we learned something critical: the client had already implemented a quality control mechanism where users review and correct model predictions. The users, accountants, have a strong incentive to ensure the data is accurate, since their work (and liability) depends on it.

From a machine learning perspective, that’s ideal. Clean, labeled training data arrives every month, driven by regular invoice cycles, and domain experts make corrections under legal pressure to be accurate. We could confidently assume the labeled data was of high quality.

5. User challenges

The platform demo also opened up a valuable conversation around what users care about most and what causes frustration. By observing how users work, we gained intuition about their day-to-day workflow. Hearing their stories, we learned what slows them down, what they appreciate, and where automation still falls short.

For these users, the key metric is time spent manually fixing model errors. By improving data extraction, we can reduce time spent on manual fixing and, therefore, improve the platform's UX. That's a clearly defined project objective.

6. Success metric

Every model error adds manual work, a clear and quantifiable pain point. So, instead of hunting for abstract KPIs, we focused on a direct metric: the number of errors per document or field class. It’s simple, measurable, and directly tied to the business outcome.

7. Client involvement

At several points, the discussion expanded into a broader machine learning strategy - what else the client wanted to build and how they envisioned scaling. They were deeply engaged, emphasizing long-term partnership and their plan to grow and maintain the platform cost-efficiently over the years.

They also came prepared with challenging questions, including tool suggestions like LLMs, and expected us to evaluate options critically. I agreed to prepare a comparison of different solution paths, including development effort, operational costs, maintainability, retraining strategy, and scalability. They also wanted our clear recommendation on the best fit for their case.

8. Business process mapping

Next, we mapped out the process: from when a document enters the system to when the extracted fields leave it. In this case, the pipeline was relatively straightforward - an isolated service with no downstream systems relying on its output in real time.

9. Functional requirements

After a short lunch break, we resumed by tackling functional and quality requirements. This part went smoothly. We agreed that the new model should outperform the legacy one on the currently supported language and achieve at least 50% accuracy on multilingual (European languages only) documents.

Speed was another topic: the model should ideally match the current system’s latency (under a few seconds). But we left the door open – if a longer runtime significantly boosts accuracy, it’s worth considering.

10. Question list

We closed the session by reviewing the question list I had prepared in advance. The client had done his homework, so we could finalize and agree on key assumptions. Most of the points had already been touched on during the workshop — this was a final sweep to ensure alignment. The questions covered data volume and format, quality expectations, and ownership of different tasks.

11. Wrap-up

At the end of the session, I summarized our achievements, outlined our next steps, and explained how much time we’d need to finalize the offer and process everything we had learned.

At that point, the client’s job was done, and ours had just begun. With the right inputs in place - requirements, constraints, and a solid understanding of both process and platform - we could start crafting a truly tailored proposal.

Workshop outcomes: more than just a concept

Thanks to the workshop, we quickly gathered a massive amount of critical information - all essential for preparing an accurate proposal and project report.

Multilingual support

We uncovered several essential project requirements that had not been communicated earlier, such as the need for a multilingual solution. Sure, it might have come up via the pre-workshop questionnaire; but what if that question hadn’t made it onto the list? We would’ve faced an unpleasant surprise during implementation.

Feedback mechanism

We also learned that the platform already includes a customer feedback mechanism for reviewing and correcting model predictions. That was one of the milestone components in our preliminary project scope offer. With this new information, we could confidently remove that milestone, significantly reducing implementation effort and total project cost.

Data quality

Moreover, we gained insights into how the system’s data is used in the client’s business context. That reinforced our assumption that user-validated data is of exceptionally high quality. Too much depends on it for users to afford neglect; data accuracy directly impacts their deliverables.

Choosing the right approach to Key Information Extraction

During the workshop, we explored different approaches to automating key information extraction from invoices — including SaaS platforms, commercial LLMs, and training custom models on client data. Since this topic was strategically essential and technically nuanced, I covered it in a separate article. It includes a detailed comparison of each approach regarding cost, risk, requirements, and business fit.

You can read it here, which I highly suggest doing.

Final offer: What the client received?

1. Education

First and foremost, we guided the client through planning an ML project so he could clearly understand how such a project works, what distinguishes it from typical IT projects, and what key elements need special attention.

2. Idea validation

During the workshop, we reviewed the data representing the core problem, understood how the target system should work, how much data is available for training, and what other limitations are in place. With that information, we were able to assess whether the problem could be solved using machine learning and how to approach it to reduce the risk of failure and maximize long-term savings for the client.

3. Cost analysis

The delivered report included a deep analysis of multiple solution variants. One of its key outcomes is clarity on the development effort and the long-term operational costs of different solution options.

With this data, the client could estimate the total cost of ownership (TCO) and compare it against the expected business impact - cost savings, process efficiency, or revenue growth. It gives him a solid basis for calculating the ROI of the investment and projecting when it will break even. Instead of making decisions based on intuition or vendor promises, the client works with concrete numbers tailored to your specific context.

4. Project plan

The final offer included five project phases.

• Phase 1: Data analysis
• Phase 2: Data annotation for training
• Phase 3: Main PoC phase
• Phase 4: Production deployment
• Phase 5: Automation of retraining & monitoring workflows

Each phase was described in detail, with corresponding pricing and duration estimates. It gave the client a holistic visibility into overall costs, estimated delivery timelines, and the possibility of accelerating execution by increasing team involvement.

Acceptance criteria

Each project phase had clearly defined prerequisites - what must be prepared or delivered before a phase can begin. We described each phase's implementation path, key milestones, and acceptance criteria. Responsibilities and point-of-contact persons were outlined on both sides. The documentation also included time estimates and the expected number of engineer-days required for each stage.

Time estimates

The offer provided a detailed technical plan covering the full project lifecycle. The entire plan was mapped out along a timeline and visualized to show the sequencing of tasks and where phases could be run in parallel.

This wasn’t just a proposal. It was a deployment-ready project roadmap.

A workshop is not a cost; it’s an investment

For many companies considering using machine learning, a workshop might seem like an "optional" step they can skip to jump straight into development. However, a well-designed and properly executed workshop is one of the most critical elements directly impacting the project's success.

Instead of starting the project based on gut feeling, the client received a real, informed choice that included operational costs, maintenance complexity, retraining options, and an estimated ROI. This approach removes unnecessary risk and allows business decisions to be driven by data, not guesswork.

From our perspective, the workshop is also where trust begins; not through promises of "AI magic," but through a transparent, pragmatic approach rooted in experience, expertise, and collaborative problem-solving.

If you’re considering whether an artificial intelligence project makes sense for your organization, a workshop is the best way to find out. In the worst-case scenario, you walk away with a clear risk map and insight into what you don’t yet know. In the best case, you launch a well-structured, realistic, and financially sound project from day one.

Summary

What will you get by choosing the workshop?

• Identify real system constraints early, immediately ruling out specific off-the-shelf tools (e.g., lack of multilingual support)
• Recognize the hidden value of existing data and validation mechanisms, which allowed us to establish an effective feedback loop for the ML model - something the client didn’t initially see as a strategic advantage
• Uncover subtle but important requirements, like the need to map model predictions onto document layouts, or the ability to scale to new languages and countries - crucial for UX and future growth
• Build shared understanding of the problem, resulting in client expectations and increased readiness to make well-informed investment decisions
• Accelerate estimation and execution, since within a few days, the client had a detailed project plan, cost breakdown, and solution recommendation in hand
• Verify the idea before you start implementation, which will improve your chances for project success
• Verify that the implementation cost and operating costs are within the budget
• Get cost estimates to calculate ROI

All this occurs before spending a fortune on a project that was not thoroughly considered.

Interested in organizing a workshop like this for your team? Let’s talk.

We’d happily guide you through the discovery process, assess feasibility, and help you make informed, data-driven decisions about your next ML initiative. If you need more information about what will happen after you contact us, here is an explanation.

Reviewed by: Rafał Pytel, Adam Jan Kaczmarek