Author | Thoughtworks
Technology Radar is a technology trends report published semi-annually by Thoughtworks that continuously tracks how interesting technologies are evolving, which we call entries. Technology radars classify them using quadrants and rings, with different quadrants representing different kinds of technologies and rings representing our maturity assessment of them.
After half a year of tracking and precipitation, the Thoughtworks TAB (Thoughtworks Technical Advisory Committee) has produced the 25th technical radar for technicians based on our practical cases in multiple industries. More than 100 technical articles are analyzed, their current maturity is explained, and corresponding technical selection suggestions are provided.
First, the theme of this issue
Bizarre Bazaar: The Changing Economics of Open Source Software
Thoughtworks has long been a fan of open source software. Because of Eric Raymond's famous article "Cathedral and Bazaar", open source software became popular, increased developer agency, and used crowdsourcing to fix errors and innovate. However, attempts at commercialization present enormous economic complexity in the current ecosystem.
For example, Elastic changed its licenses to require feedback from cloud service providers that profited from them, which led AWS to fork Elasticsearch to OpenSearch in September 2021. This shows how difficult it is for commercial open source software to guard the moat of competition (the same problem applies to free closed-source software, as Docker has been struggling to find the right business model, and we've seen some companies exploring alternatives to Docker Desktop).
Sometimes, power dynamics are reversed: thanks to Facebook funding the open source software Presto, Presto's contributors were able to retain ip (intellectual property) and rename it Trino after they left the company, effectively thanks to Facebook's investment. The fact that a large amount of critical infrastructure is not sponsored by businesses can make the situation even more confusing, and it is often only when critical security vulnerabilities are discovered that these businesses notice how dependent they are on unpaid labor (as in the case of the recent Log4J problem).
In some cases, funding amateurs and maintainers through GitHub or Patreon can provide enough motivation to make an impact; but for others, it's on top of their day-to-day work, adding an extra sense of responsibility and leading to burnout. We remain strong proponents of open source software, but also recognize that economics is becoming increasingly bizarre and that there are no simple solutions to find the right balance.
Innovation in the software supply chain
Notoriously significant security incidents — Equifax data breaches, SolarWinds attacks, Log4J remote zero-day exploits, and more — are caused by mismanagement of the software supply chain. The development team now realizes that sound engineering practices also include the validation and management of project dependencies, so there are many related entries on this issue of the Technology Radar. These entries include checklists and standards such as the Software Artifact Supply Chain Hierarchy (SLSA): a Google-led one designed to provide guidance on general threats to the supply chain; and CycloneDX: another set of standards promoted by the OWASP community. We also cover specific tools, such as Syft, which generates a software bill of materials (SBOM) for container images.
Hackers are increasingly exploiting the asymmetry of attack and defense in the security realm and combining them with increasingly sophisticated hacking techniques – meaning they only need to find one vulnerability, and defenders must secure the entire attack surface. Improving supply chain security is a key part of our response as we strive to keep our systems secure.
Why are developers always keen to implement state management in React?
Booming frameworks seem to have become a common pattern in the radar of technology: a basic framework becomes popular, followed by the emergence of a large number of tools to create an ecosystem for common flaws and improvements, and finally ends with the consolidation of several popular tools. However, React's state management seems to be resisting this general trend. Since the release of Redux, we've seen a steady stream of tools and frameworks manage state in slightly different ways, each with different trade-offs.
We don't know why, we can only speculate: is this a natural drain that the JavaScript ecosystem seems to be advocating? Or is it an interesting, seemingly easy-to-fix potential flaw in React that encourages developers to experiment more? Or is there an obstacle between the document reading format (browser) and the interactivity (and state) required by the application hosted on top of it that can never match? We don't know why, but we look forward to the next round of attempts to solve this seemingly eternal problem.
A never-ending pursuit of master data cataloging
The desire to extract more value from enterprise data assets is driving much of the investment we are seeing in digital technologies today. At the heart of this work is often focused on discovering and accessing all relevant data. Almost as long as a business is collecting digital data, it has been working to rationalize and catalog it into a single, top-down enterprise catalog. Yet again and again, this intuitively appealing concept runs counter to the harsh reality of complexity, redundancy, and ambiguity inherent in large organizations. Recently, we've noticed a renewed interest in enterprise data cataloging, as well as a number of radar entry proposals for clever new tools, such as Collibra and DataHub. These tools provide consistent, discoverable access to data lineage and metadata across warehouses, but their ever-expanding set of capabilities also extends to data governance, quality management, publishing, and more.
Contrary to this trend, the trend towards joint governance and data discovery based on data grid architectures is also growing, moving away from top-down centralized data management. This approach addresses the inherent complexity of enterprise data by centralizing expectations and standards, while data management is divided according to business domain lines. Domain-oriented data product teams control and share their metadata, including discoverability, quality, and other information. At this point, cataloging is just a way to present information for searching and browsing. The resulting data cataloging is simpler and easier to maintain, reducing the need for a feature-rich cataloging and governance platform.
Second, some of the quadrant highlights are a sneak peek
SLSA (Assessment)
As software complexity continues to increase, the threat path of software dependencies becomes increasingly difficult to guard. The recent Log4J vulnerability shows how difficult it is to understand these dependencies – many companies that don't use Log4J directly become vulnerable unknowingly because other software in their ecosystems depend on Log4J. The Software Artifact Supply Chain Hierarchy, also known as SLSA (pronounced "salsa"), is a collection of guidelines for organizations to protect against supply chain attacks, curated by alliances. The framework is derived from an internal guide that Google has been using for years. To its credit, the SLSA does not promise to provide a "silver bullet," a methodology that uses only tools to secure the supply chain, but rather provides a list of specific threats and practices based on a maturity model. This threat model is easy to understand, contains examples of real-world attacks and requires guidance in the documentation to help organizations prioritize their actions based on increasing robustness levels to improve the security posture of their supply chains. We believe the SLSA provides applicable advice and expect more organizations to learn from it.
Server-side driver UI (experimental)
When compiling a new phase of the technology radar, we are often overwhelmed by a sense of déjà vu. With the advent of development frameworks, server-side driver UI technology has sparked a hot topic. This technology allows mobile developers to take advantage of faster cycles of changes without violating any of the Store's policies on revalidating mobile apps. We introduced this technology in a previous radar from the perspective of enabling mobile development to scale across teams. The server-side driver UI separates rendering into a common container for the mobile application, while the structure and data for each view is provided by the server. This means that modifications that used to take an app store publishing journey are now achieved by simply changing the response data sent by the server. To be clear, we don't recommend this approach for all UI development. We did get stuck in some dreaded over-provisioning, but backed by giants like AirBnB and Lyft, we suspect it's not just our Thoughtworks that's tired of handing everything over to the client. This area is worth noting.
tfsec (adopted)
For those projects we are using Terraform, tfsec has quickly become the default static analysis tool when it comes to detecting potential security risks. It's easy to integrate into the CI pipeline and has a growing inspection library that can be used to check all major cloud vendors and platforms such as Kunernetes. Given its ease of use, we believe tfsec will be a great addition to any Terraform project.
Carbon Footprint of Cloud Services (Trial)
Cloud Carbon Footprint (CCF) is a visualization tool for viewing carbon emissions on AWS, GCP, and Azure cloud platforms through a cloud API. The Team at Thoughtworks has successfully used the tool to work with multiple organizations, including energy technology companies, retailers, providers of digital services, and companies using ARTIFICIAL intelligence. Cloud platform providers recognize the importance of helping customers understand the impact of carbon emissions when using cloud services. So they started building similar features on their own. Because CCF is cloud-independent, it allows users to view energy use and carbon emissions from multiple different cloud providers in one place, while translating the carbon footprint into real-world impacts, such as how many flights or how many trees are emitted. In recent releases, the CCF has begun to include optimization recommendations for possible energy savings and CO2 reductions on the Google Cloud and aws Clouds, as well as support for more types of cloud instances, such as GPUs. Considering that the tool is now in the spotlight and continues to add new features, we are confident that we will move it into experimental status in the future.
eBPF (Test)
In recent years, the Linux kernel has included an extended Berkeley Packet Filter (eBPF), a virtual machine that provides the ability to attach filters to specific sockets. However, eBPF goes far beyond packet filtering, allowing custom scripts to be triggered at different points in the kernel with very little overhead. While this technology isn't new, eBPF is becoming self-contained as more and more microservices are deployed through container orchestration. Kubernetes and service mesh technologies such as Istio are commonly used, employing "sidecars" for control functions. With new tools such as Bumblebee that make it easier to build, run, and publish eBPF programs, eBPF can be seen as an alternative to traditional sidecars. Cilium's maintainers even announced the demise of sidecars. The eBPF-based approach reduces some of the performance and operational overhead associated with sidecars, but it does not support common features such as local termination of SSL sessions.
Cloudflare Pages (Evaluation)
When Cloudflare Workers was released, we highlighted that it was an early function-as-a-service (FaaS) solution for edge computing that was very interesting to implement. Cloudflare Pages, released last April (2021), wasn't particularly eye-catching, as Pages is just one of many Git-based web hosting solutions. While Cloudflare Pages does have a useful feature that most alternatives don't have – continuous previews. Now, though, Cloudflare has more tightly integrated Workers and Pages, creating a fully integrated JAMstack solution that runs on a CDN. Key-value storage and highly consistent harmonic primitives make the new version of Cloudflare Pages even more appealing.
Testcontainers (adopted)
Based on our long experience with Testcontainers, we consider it to be the default option for creating a reliable environment to run automated tests. Testcontainers is a library in multiple languages, and Docker incorporates common test dependencies — including different kinds of databases, queuing techniques, cloud services, and UI test dependencies (such as web browsers), as well as the ability to run custom Dockerfiles on demand. It is compatible with JUnit-like test frameworks and flexible enough to allow users to manage the lifecycle and advanced network of containers and quickly set up an integrated test environment.
Zig (Evaluation)
Zig is a new language that shares many of its attributes with C, but has stronger typing, easier memory allocation, and support for namespaces and many other features. However, its syntax is more reminiscent of JavaScript than C, which will cause some people to oppose it. Zig's goal is to provide everyone with a very simple language that can be compiled directly to reduce side effects, and program execution that is predictable and easy to trace. Zig also provides a simplified interface for LLVM cross-compilation functionality. Some of our development colleagues found this feature so important that they used it as a cross-compiler, even though they didn't program with Zig. Zig is a novel language that is worth a try for applications that are considering or already using C, as well as low-level system applications that require explicit memory manipulation.