ArchGuard Co-mate: Exploration of Big Language Models and Architecture Governance and Architecture Design

In the past few months, in order to explore the possibility of LLM combined with SDLC and BizDevOps, we (the Thoughtworks open source community) have created a series of open source projects such as ClickPrompt, AutoDev, DevTi, etc. From personal experience, more and more applications will be designed around LLM@Core in the future, so we have dug a new pit, a new experiment: ArchGuard Co-mate: https://github.com/archguard/comate, welcome to explore and research together.

Why is Co-mate ？

We started the Co-mate project because of the development skills in the LLM-first application development era. These related skills include, but are not limited to:

• Explore if there is a better interaction model? Such as Chat-based dynamic UI (DSL) generation, etc.
• For LLM application development under the JVM architecture? Such as JVM version of LangChain, Embedding, etc.
• How to assist the context to understand and build a local small model? That is, local processing of Sentence Transformer + ONNX.
• How do I design a next-generation software API for LLM? That is, how to interact with the ArchGuard API in a deterministic way.
• How can you better build contexts and engineer the context (i.e., prompt) of your AI applications?

For the scenario of architecture governance, we want to explore:

• How do LLM explain software architecture? After all, making people understand that building is a laborious task.
• How to get LLM to explain architecture and business based on code, layering, dependencies, etc.?
• How can LLM better understand the specification and design the software architecture from the documentation?

Each question is interesting enough and has a level of business and technical complexity to give us a grasp of the factors to consider in LLM-based software architectures.

What is Co-mate ?

As an exploratory project, explaining the uses of Co-mate is not an easy task. We haven't been clear enough about how far an architect's assistant can go—because we haven't agreed on the definition of architecture.

But first, let's take a look at Co-mate's current process:

On the surface, there is basically no difference from mainstream AI applications; From the actual code point of view, there is not much difference from mainstream AI applications. Or let's look at two typical Co-mate scenarios.

Example 1: Describe the system

When a user analyzes a co-mate in Co-mate, a result similar to the following is returned:

The Co-mate project is an AI assistant application that uses a range of related libraries such as Jetpack Compose, RxJava, DataStore, and more to build features such as design and governance tools. The application also uses third-party libraries to build user interfaces such as Coil-Compose, Voyager Navigator, etc., as well as deep learning inference tasks such as the Kotlin Deep Learning API, Inference API, and more. This application needs to consider non-functional requirements such as efficient algorithms and model inference capabilities.

For this scenario, the data processing process is as follows:

1. Match locally relevant instructions (e.g. "analysis system")
1. If there is a match, the AG API is called directly to build the context.
2. If there is no match, call LLM to select the command, and then you can call the AG legacy API.
2. Call the AG API to build the context (project information, software dependency information, etc.).
3. Call LLM to summarize and return to the user.

So, we tried to build two new APIs: the local semantic analysis and the (dynamic) context collection API.

Example 2: API prescriptive checks

Based on ArchGuard's capabilities, the second scenario we picked was to check if the API was standardized. When you have an API in a Controller that needs to check for compliance with the API specification, you can execute: Check API Specification.

Let's say your API is: /api/blog/get and has already converted your API specification via spec-partitioner (not implemented yet).

Finally, Co-mate returns:

API '/api/blog/get' does not conform to the URI construction specification, Rule: uri construction regex: \/api\/[a-zA-Z0-9]+\/v[0-9]+\/[a-zA-Z0-9\/-]+, it is recommended that the API be modified to '/api/blog/v1/get'.

(PS: Garbage GPT 3.5 Turbo, actually approved /get)

So, when you have the full architecture specification, then you can move on to the next generation of architecture generation:

This is also something we want to explore further.

How Co-mate works ?

As we all know, GPT is full of uncertainties, and people's ability to understand GPT is also different. Therefore, from the perspective of architectural design, we need to decompose the atomic capabilities of GPT, such as summarizing, classifying, extracting, translating, logical reasoning, eliminating the uncertainties, and then providing dynamic capabilities by our software-wrapped API.

Layered architecture mapping with ArchGuard capabilities

In Example 1, the first thing we do is to decompose the schema and data, analyzing it by different schema elements. Because we lack a uniform definition of architecture, I looked to Global slides for a layered architecture that fits LLM's understanding and also works with ArchGuard expressions. Subsequently, a less successful layered mapping with the required context data is constructed:

So in Example 1, the prompt template we give is:

The {xxx} project is a {channel type} application that uses Jetpack Compose, {xxx}, and a range of related libraries to build {xxx} and other features. The app also uses some third-party libraries to build the user interface {xxx}, as well as tasks such as {xxx}. The app needs to consider non-functional requirements such as {xxx}.

In this prompt, it relies on two main data: the project description and the project's technology stack (dependent information). The technology stack can be obtained directly from ArchGuard SCA, while the project description is parsed from README.md.

Capability mapping of LLM to Co-mate API

In Example 2, the first thing we do is decompose the API documentation, decomposing by the atomic capabilities of different LLMs. Four different atomic capabilities were built:

• Infer regular expressions that apply to URIs.
• Reason out a reasonable example.
• Extract some checklists, such as status codes, HTTP actions, etc.
• Throw together the rest of the uncertainty.

As shown in the following figure:

On the right, we built a Kotlin Typesafe DSL to dynamically load into the system (future), with each function corresponding to a Rule.

rest_api {
    uri_construction {
        rule("/api\\/[a-zA-Z0-9]+\\/v[0-9]+\\/[a-zA-Z0-9\\/\\-]+")
        example("/api/petstore/v1/pets/dogs")
    }

    http_action("GET", "POST", "PUT", "DELETE")
    status_code(200, 201, 202, 204, 400, 401, 403, 404, 500, 502, 503, 504)

    security("""Token Based Authentication (Recommended) Ideally, ...""")

    misc("""....""")
}           

As a demo, this DSL still has a lot of room for improvement. The more interesting part of this is the security and misc parts, and these uncertainties apply precisely to LLM inference. Therefore, when performing the corresponding misc and security rule checks, GPT will be called again to check:

In order to better combine certainty and uncertainty, it makes full use of the capabilities of LLM and ArchGuard and reduces the consumption of GPT.

Welcome join us

The following diagram shows all the modules of the current ArchGuard Co-mate:

A brief introduction is as follows:

• Comate-Core provides the basic capabilities required by the CLI and GUI.
• Meta-Action defines the basic Action
• Architecture defines what a co-mate understands as an architecture
• LLM-Core is a call to LLM.
• Spec Partitioner is the automatic generation of the plan for the extraction and automatic generation of specifications (currently manual prompt)

When we used the JVM technology stack, we encountered several pitfalls KotlinDL and Deep Java Library both called HuggingFace Tokenizers and ONNX APIs through JNI/Rust, resulting in a crash of the application under macOS. An ideal way would be to call it via JSON RPC, so we plan to build a new module using Rust: Comate Agent.

So, if you're interested in using the JVM stack to develop AI applications, and the Rust stack for AI applications, welcome to join us.

summary

This article introduces a series of open source projects created by the Thoughtworks open source community, and explores the possibilities of big language models, architecture governance, and architecture design. Among them, ArchGuard Co-mate is an exploratory project that aims to explore the capabilities of the architect assistant, including local semantic analysis, dynamic context collection API, schema specification checking, and more. The article also covers layered architecture and ArchGuard capability mapping, LLM and Co-mate API capability mapping, and more.