How to create beautiful architecture diagrams using Python?

This article discusses the technical diagram in Data Engineering projects, examining solutions such as Google Slides and draw.io. It highlights alignment challenges and concludes with a reflection on simplifying this process.

Introduction

In a Data Engineering project, a technical diagram summarizing the entire pipeline is necessary to have a complete overview. How can you quickly and easily create a technical diagram that summarizes a project's data pipeline? This is a question that frequently arises when it comes time to represent your pipeline. It is possible to do it directly in Google Slides by inserting shapes and arrows, but this can quickly become tedious when you need to resize different parts, move them together, or align text. There are also tools, like draw.io, that make creating diagrams easier by linking different parts; however, alignment issues persist, and the pipeline will be created using only shapes.

‍

‍

To avoid all these problems and limitations, you can use the Diagrams package, which produces an easily readable technical diagram in just a few lines of code. Furthermore, creating a technical diagram with code allows you to reuse your work, and if several people are collaborating on the same diagram, it makes it easy to use a version control tool.

We will see in detail that Diagrams is a flexible tool that allows you to produce technical diagrams easily while maintaining clarity for readers. We will look step-by-step at how to use this package and its features.

Prerequisites

To use the diagrams package, you must have Python 3.6 or higher installed. Next, you will need to install GraphViz, as it is what allows the graphics to be rendered. You can find GraphViz in the "Getting Started" section of the Diagrams project GitHub.

Then you can install the diagrams library using your package manager, and you will be ready to start creating beautiful diagrams.

For my part, I installed the package using pip

The basics

There are 4 different elements in this package:

• Diagrams (Diagram)
• Groups (Cluster)
• Edges
• Nodes

The first 3 elements are each defined by a class. Regarding nodes, there are numerous classes offered by various providers such as AWS, Azure, or GCP for clouds, as well as Kubernetes. You can find all the classes in the official package documentation.

Finally, these 4 elements are linked: a diagram is made up of nodes that can be grouped together and connected by edges. You will therefore need to import the necessary classes to correctly represent your architecture diagram.

Now, let's try coding a first diagram to understand the basics of the package.

‍‍

This diagram describes a data engineering pipeline, using a table contained in an RDS database, processed via AWS Glue ETL. The processing results are stored in an S3 bucket and the logs in Cloudwatch. Finally, a Quicksight dashboard is connected to the S3 bucket.

Let's look at this first piece of code in detail. First, we import the Diagram class, which is necessary to produce a diagram. Then, we import some node classes from the AWS provider, for example, RDS, Glue, etc.

We then create a new Diagram with the name ‘Pipeline – Global Overview’. Since we provided the filename parameter, the diagram will be saved at the indicated location. Note that the path provided is a relative path (the root will be the same as the location where the code is executed; for example, if the code is run from the desktop, the diagram will be saved on the desktop) and not an absolute path. The show parameter is set to True, so Python will open the diagram immediately after the code executes. The direction parameter indicates the orientation in which the graph will be built; here it will be from left to right (LR), which is the default setting. Other options are right to left (RL), top to bottom (TB), and bottom to top (BT). Inside the diagram, we create several nodes using the classes we imported. To create a link between two nodes, you must add ‘>>’ between the two nodes if you want the arrow to point from left to right, or ‘<<‘ if applicable.

To finish with all the package classes, let's try a slightly more complex diagram that incorporates clusters.

‍

‍This diagram describes the same pipeline as the previous one; the only difference is that here there are two Jobs represented in AWS Glue.

The primary purpose of clusters is to group similar elements into a single subset.

The second use I find for clusters is the ability to more clearly delineate the different parts of the pipeline, as shown in the following example:

‍

The code will once again produce the same diagram, but the appearance will be different; each part of AWS that was used will be even more clearly identified.

Advanced settings

Now that we know how to use diagrams, clusters, edges, and nodes, let's look at customization. There are two customizable objects: nodes and edges.

Customizing Edges

Let's first look at how to customize edges. There are 3 customization parameters: color, style, and label. The default color is gray, but if you want to set a different color, the colors are those used by the matplotlib package, which you can find here</a >.

Next, it is possible to adjust the style, and there are 4 available:

• The default style, which is a solid line
• Bold, a thick solid line
• Dashed, the line is made of dashes
• Dotted, the line is made of dots

It is not possible to combine different styles, i.e., having a bold dashed line.

Finally, it is possible to add a label to an Edge if you want to clarify what that Edge represents.

Let's see what this looks like in code.‍‍

‍

Here, I created a function, arrow(), which by default produces a bold black arrow, which I prefer over the package's default arrow. I then use this function to define the different arrows I want in my chart. When you want to customize an Edge, you must explicitly mark it in the diagram schema between the two nodes concerned. Here, I wanted the pipeline Edges to be red and bold, except for the arrow for the logs, which is pink and dotted. You can see this in the last two lines of the code.

Customizing Nodes

Let’s move on to the second point: customizing Nodes. What does this mean? Customizing nodes means displaying a Node with an image that isn't already pre-saved in the package's image library, effectively creating a Node that doesn't exist by default.

Do you want to represent sending an email in the event of an error? That isn't among the options available in the package, but you can simply upload an image representing an email and use the Custom() Node to integrate this new Node into your diagram. There are countless possibilities for Nodes, and the only limit is your imagination.

‍

Here, I chose to represent the developers who worked on this project by specifying which parts of the pipeline they were involved in. To make it more visual, I created two new Nodes with photos of the developers, making it easy to identify exactly who to contact if there are any issues with the pipeline.

Finally, once you have mastered the various features of the package, you can produce very detailed diagrams. Here is an example:‍

‍

First, you'll notice that the code is more complex than in the previous diagrams. As for the diagram itself, it provides a truly detailed overview of the data processing pipeline, from the database to the dashboard, all within the AWS environment.

Conclusion

Diagrams is a package that allows you to represent pipelines through diagrams with ease and flexibility. If you would like more information or want to use more advanced commands, I recommend checking out the Diagrams project GitHub, specifically the Issues section.

Note: This article was freely inspired by the article "Create Beautiful Architecture Diagrams with Python" written by Dylan Roy and available here