Youtube has resigned from making their annual 2020 rewind, because 2020 was just too much. Was it just a bad year everyone wishes to forget? Definitely not, let’s dive deep into tech innovations that stood out and will need our closer attention in 2021.

Looking for a dash of engineering approach in your tech and software development trends yearly digest? Here’s our compilation based on just that!

Have a great read and a Happy 2021!

Widespread of AI and MLOps

Artificial Intelligence (AI) and Machine Learning (ML) are reaching new sectors. They can be applied nowadays almost everywhere. Mobile devices are getting enough power to apply ML models on the fly. What is more, specialized hardware and software are being created for small, low powered devices, so that they can also leverage the power of ML. They can be located e.g. in factories and analyze data on the spot without the need to transfer data to a central data center. Thanks to that, decisions based on data can be made quicker, sometimes saving lives. Such a trend is called Edge AI or Edge Analytics.

Crises inspire people to achieve new unthinkable goals. Protein folding remained an unsolved problem for the last 50 years. Breakthroughs in deep learning have changed that. We saw disruptive projects, like the Folding@home project, coming to life. It is a “crowdsourcing” model for people willing to offer their CPU and GPU power in order to create a powerful supercomputer, and use it to help solve and understand complicated problems. Because of COVID-19 we saw a boom in its active number of users wanting to help scientists fight the outbreak. The protein folding problem has been the focus of other initiatives also. Google DeepMind revealed the AlphaFold project, naming it the solution to protein folding grand challenge. Nature named it a “gigantic leap”. AI can now solve protein structures in 30 minutes, while scientists were trying to do that by other means for the whole decade!

Phenomenal breakthroughs are moments to stop and think about the future humanity paths. Topics known from sci-fi movies and literature now appear in standard media. Is AI ethical? Who bears the responsibility for its decisions? Nowadays even a simple image cropping algorithm may bring discrimination and inequality. However, there are even bigger challenges since self-driving cars are allowed in more and more countries.

Philosophical doubts from previous years become reality with extended discussion, who would be potentially guilty in the accident - the driver or the car company? Explainable AI is promoted since in many cases it is even not possible to understand why the algorithm took a specific decision. We all need to be responsible for the things we create and make sure they will make the world a better place.

Check how the predictive power of AI lets us use data-driven models of environmental processes to fight the deforestation problem.

Wider AI & ML adoption requires the data scientist and ML engineers to apply more practices originating in standard software development. Versioning or Continuous Delivery are much more complicated in the ML world. Models can be quite big plus they may be a part of bigger processing pipelines. DVC, MLflow or Kubeflow are quite new tools trying to solve some of the challenges. They add version control or support the lifecycle management of ML projects. Automatization brings a need of a new role in the team - MLOps - person applying mentioned DevOps principles to ML systems.

Key Takeaways

• Edge AI is the next wave of AI.
• Phenomenal breakthroughs in deep learning that will help us tackle unsolved problems will bring ML engineers on the frontier of science.
• We will start seeing the need of Continuous Delivery and automation pipelines in machine learning.

Real-time Big data powered by AI

Big Data is not a new thing, mass data processing has been on CIOs radars for quite a while already. However, what happened recently, is a visible shift toward fast real-time or near-real-time data analysis. Old batch solutions which produce results after a few hours are no longer acceptable. Business requires the results now in order to interact with customers in real time to provide an effective, digitally focused customer experience.

This is where stream processing comes in, a technology that enables rapid and distributed data processing. Stream processing is leading the new big data era and we are seeing a spiking interest in technologies like Apache Kafka or Apache Flink. Big companies often still leverage older solutions like Hadoop, which in many cases is not fully effective, but remains in use because the migration cost is very high. We noticed that more and more products are delivered now in a Service (SaaS - Software as a Service) manner. Confluent offers Confluent Cloud with hosted Kafka, DataStax offers DataStax Astra with hosted Cassandra. Sometimes it is easier, faster and cheaper to buy an out-of-the-box solution than hire additional DevOpses with proper skills and set up their own cluster which needs to be monitored 24/7. It depends, however, on the use case.

Another example where using SaaS introduces cost optimization is the Snowflake. Snowflake is a data warehouse with a quite simple pricing model. You pay for storage and your queries' execution time. This is much simpler than to deploy, e.g. an AWS Redshift cluster, and maintain it 24/7. Such advantages cause Snowflake to soar and become one of the fastest growing products in last months.

The use of big data tools for decision making can be even more sophisticated when injected with AI algorithms that can prepare and clean data, automate data collection and enhance the data analysis. AI-driven solutions will have the advantage of using deep learning algorithms to make predictions not only on historical data, but mostly on the results of complex simulations. But, most importantly, while traditionally data is processed according to fixed computational rules, AI creates computations based on existing data.

The rise of intelligence data platforms enables the whole market to develop more and more into streaming and real time processing. 2021 will be all about Big data in real time and businesses getting actionable data for faster processing and making intelligent business decisions.

Key Takeaways

• Software as a Service usage will accelerate, resulting in additional abstraction layers and more robust software development.
• Intelligent data processing tools will give businesses actionable insights about digital-first customers.
• AI software solutions will have a higher business impact and define previously unknown algorithms to analyse data.

Cybersecurity

60 Percent Of Small Companies Close Within 6 Months Of Being Hacked. Have you heard about Garmin outage? Seemingly a downtime of a fitness-related service also causes problems in the aviation sector. Nowadays everybody is a target. Cybersecurity company FireEye also got recently hacked.

Cyberattacks are a common threat to businesses. Every hacking scandal hurts the brand, and causes a customer exodus from the platform. The bottom line is that we live in a world that’s run by computers, where more and more personal information is processed or stored online. A significant portion of that data can be sensitive information. Cybersecurity and data privacy are therefore interconnected. Current privacy laws that regulate data breaches (European GDPR, Canadian PIPEDA or California Consumer Privacy Act) bring lawyers to the forefront of technology innovation. Companies that do not protect customer data may face big fines or lawsuits.

The ever-evolving nature of security risks causes the growth of the adoption of tools and techniques supporting cybersecurity. There are solutions analyzing external code dependencies, Docker images, or auditing deployed systems. Everything in order to detect vulnerabilities or improve configuration which can become a security issue. O’reilly Radar noticed an increased interest in security training and certifications. Techniques like zero trust promote approaches in which breach impact may be limited, since every service in the infrastructure offers no trust towards others.

Until now many companies treated security as an unpleasant necessity, and implemented only the absolute minimum of requirements. Now it may not be enough, and additional expert insight could be needed to introduce alternative practices and tools. In the current IT infrastructure the entire internal traffic needs to be encrypted and authorized. Therefore, companies tend to employ more specialists in SecOps (Security Operations) or DevSecOps roles.

However, major challenges remain. The “software supply chain” heavily relies on a large number of reusable components (both open- and closed-source). The sheer number of such dependencies makes auditing the whole software package a hard and ongoing task. Malicious code, injected into an otherwise totally unrelated or innocent library, can cause a major data leak.

Key Takeaways

• The cybersecurity space presents a great business opportunity to develop new technologies.
• The ever-evolving nature of security risks causes the growth of the adoption of tools for monitoring network security and scanning web vulnerabilities.
• Even larger numbers of businesses will be seeking cybersecurity expertise in 2021, they will be also more strategic about their tools.
• The 2020 pandemic has brought more businesses to the cloud and they need to educate themselves on security practices. For them, knowledge will be a powerful first step towards protection against various kinds of threats.

DevOps and DevSecOps

Old times with once-a-month, hours-long deployments are gone. Operations must be now quick & simple, and, more importantly, covered by automation. Every infrastructure element needs to be defined as code: Infrastructure as Code, Security as Code and so on. In case of emergency, it is possible to launch a whole new product instance on new hardware or a different cloud provider. Rapidly growing platforms like Kubernetes (k8s) are supporting that. k8s is currently offered as a service by every major cloud provider including AWS, GCP, Azure or Alibaba. Service meshes enhance k8s with additional observability, monitoring and - more importantly - Zero Trust.

Some companies decide to leverage hybrid cloud approach, where part of the infrastructure exists in a private datacenter and some in the cloud. Such solutions add flexibility (it is easier to add more computing resources due to the cloud) and additional reliability. Even if the cloud or network is down (do you remember that multiple Google-related services were down for 45 minutes on Dec 14?), the company can still operate on private infrastructure, without any downtime, just with lowered capabilities. We can say that hybrid product teams will become a pillar of customer value delivery.

However, not everything needs to happen in centralized data centers. In many cases latency and response times are important. Edge computing focuses on that, since computation and storage is located as close to the customer as it is possible.

We’ve started seeing commercial requirements from clients for cloud-provider-agnostic solutions. After cloud adoption has increased significantly since the pandemic started, we expect to see increased demand for technologies that can be swiftly deployed through cloud-based, on-demand platforms. Also, the new approach to automation will become more popular and will strongly link DevOps role to cybersecurity and software risk reduction.

DevOps tools reached such a level of adoption that last year even celebrities were hired to explain the nuances to the public.

Key Takeaways

• The sophistication of the automation will make DevOps rely on more advanced and autonomous techniques.
• Cloud-native technologies powered by Kubernetes are the new tools of choice and will cause broader adoption of containers and microservices.
• DevOps and DevSecOps roles in cybersecurity will grow.

Blockchain is maturing

Quite recently Bitcoin attempted to reach new All Time Highs (ATH). In some countries banks can now hold cryptocurrencies.

However, blockchain is way beyond just cryptocurrencies. This sector is maturing and we see less attempts to leverage just the hype and implement blockchain in every possible project. Instead, blockchain projects are focused on creating trusted business models and leveraging private blockchains, like Hyperledger Fabric.

Interestingly, the pandemic has directly influenced some blockchain-related topics:

• health passports with health information about vaccinations and patients profile
• digitalization and durable media
• there were also ideas to leverage blockchain during fights with fake news and deep fakes.

The Decentralized Identity Foundation, the organisation focused on making Blockchain-based ID systems a reality, is getting more attention from big companies. Decentralized identity uses blockchain as the point-to-point exchange of information about people, organizations or things. The goal of the project is to make decentralized blockchain solutions available to everyone, giving them the ability to control and protect their own personal information.

At the same time, blockchain-based tech is introducing novel business operating models to many industries. Blockchain features like transparency, authenticity management, cost-reduction and security are very attractive for the enterprise sector. For example, private blockchain-based solutions developed based on Hyperledger introduce modular architecture and scalability and are becoming a standard when organisations are looking for a strong data consistency paired with great performance.

Key Takeaways

• Decentralized Identity is a new emerging trend in blockchain technology.
• Blockchain projects are focused on creating trusted business models and leveraging private blockchain, like Hyperledger Fabric.

IoT gets smarter

Who could predict that shops like Ikea or Lidl will be offering Smart Home IoT devices? IoT reached masses and you no longer have to be an expert to add some automation to your home. Automatic lighting, water valves closing when you leave home or voice-enabled home assistants are no longer rocket science. Unfortunately the Smart Home market also gets some fragmentation. Most companies focus on their products and do not allow for easy integration with competitors’ devices. Open Source projects like Home Assistant try to overcome that offering over 1700 integrations.

Fun fact, the technology can backfire. We need to be careful with cloud-based solutions that take control of our homes. Their lost reliability can cause troubles in some cases, so the local control is important.

IoT is more than Smart Home. It has applications in various businesses. At SoftwareMill we had an opportunity to design software for Smart Buildings.The goal of the project was to bring connected technology to the lighting industry. We leveraged sensors and intelligent lightning to adjust light levels according to current weather. Lights could be automatically turned off in empty areas or be adjusted according to a programmed scheme. This has a positive impact both on the budget but also on the environment. Additional sensors (like e.g. temperature, PIR) were used to analyze people behaviour in the building (which areas are most used) but also to improve security, e.g. in case of a fire.

Predictive maintenance can save companies from downtimes and production line issues, because Industry 4.0 is getting richer with more sensors and intelligent devices. Machine parameters are monitored and analyzed almost instantly with the Edge Analytics and Edge AI, allowing to detect potential failures and dangerous situations for the employees. Following 2020, such possibilities can be shifted towards employee social distancing and track-and-trace capabilities to minimize pandemic spread, like vision sensors detecting if a person wears a mask, or RFID tags to ensure employees wash hands.

In the upcoming year 5G can become a factor strengthening IoT and Big Data growth. Bigger network bandwidth, more connected devices for sure will increase the amount of the data and possibilities.

Key Takeaways

• IoT will ensure better safety in the work environment and will take us closer to Edge AI applications
• 5G may become a catalyst of widespread adoption of IoT solutions.

Pandemic IT innovations

A competitive business environment is harsh for the companies which don’t embrace change, adapt and evolve. COVID-19 only stressed that even more. We had to reinvent the way we live and do business. On the good side, the outbreak introduced a surge of technologies that will spawn a digital transformation of society further.

The IT sector will continue to play an important role in coping with business challenges in 2021. Pandemic-driven tech innovations clearly demonstrated that. In just a few months’ time, the crisis has brought about years of change in the digitization of customer and supply-chain interactions, manufacturing and production processes, internal operations and creating new digital products. The need to plan business continuity induced new kinds of investments that will stick and continue.

We faced the new economy with a new model of a composable and intelligent business. And companies that invested in digital technologies early fared better during the pandemic. Which innovations will continue to grow in the post-pandemic world?

With customers being more digitally focused and smart, we will continue to see an increased shift to e-commerce with cashless, contactless and online purchases, self-service solutions and AI-powered chatbots, drone deliveries and fast methods of shipping. There are the right technologies matured enough out there to leverage this opportunity.

Remote movement is here to stay. It requires businesses to rethink the conventional workplace. Ensuring everyone with the right tools to stay connected and to collaborate is crucial. A significant part of this process is the need to establish a correct data strategy with flexible and intelligent infrastructure based on cloud.

We stay at home more. We do almost everything from home, things like home gym equipment, smart devices or even an increased media consumption kickstarted the race to provide an effective, digitally focused customer experience. The use of technologies like AI, IoT and code development will help us feel reconnected.

And, finally, maybe the obvious one: we saw an increase in telehealth diagnostics. While we think we have all of the digital tools that can facilitate access to healthcare, the challenge of an effective application of technology and data to healthcare remains. Nevertheless, in 2021 we will definitely see the expansion of telemedicine and the use of Big Data and AI in the national digital health portals and population diagnostics.

Key Takeaways

• The IT sector will continue to play an important role in coping with business and social challenges, while cutting edge technologies will help us feel reconnected.

Digital ready culture

Everyone switched to digital solutions during the lockdown. Instead of going to work or school, we worked and attended classes remotely, instead of visiting friends and family, we played online and talked on Zoom, Meet or Microsoft Teams. This digital shift will be a driver of change for our experience across all industries and geographies in 2021.

Leveraging technology in new ways is helping us re-engineer the world around us. When it comes to remote work, the good news is that initially it worked and saved many businesses. In 2021 companies will look to build resilience in this new model and, as well, will be seeking tools to improve productivity.

Some companies were already digital darlings and transformed smoothly. Some may need more education and investment in order to transition. Many software companies can be a benchmark of adopting best practices as they were on this journey already. Distributed working scenarios will become the norm after the pandemic ends, if you need a helping hand and advice on transitioning a business to a remote model, let us know, we’ve been a remote-only company since 2009.

Digital-ready culture is happening. Netflix already made a whole series called Social Distance with dark and funny takes on how people strive to stay connected while staying apart. Such reality is not without challenges. Society needs to ensure adequate levels of secure connectivity, come up with automated models of digital control of physical assets and most of all, craft a future that’s centered on the wellbeing of people and a sustainable environment. 2021 will be a year where for IT and CIOs, enabling and increasing technology savviness across lines-of-business remains a key priority in a digitally-enabled business environment.

Key Takeaways

• The shift towards remote will accelerate and organisations will focus on building resilience and implementing tools to improve productivity.

Monthly tech news flash

Looking for quality tech content about Machine Learning, Big Data or Stream Processing?

Join Data Times, a monthly dose of tech news curated by SoftwareMill's engineers.

This post was written together with Maria Wachal, our Marketing Geek & Tech Evangelist.