Before going into the details, Tokopedia is one of the largest e-commerce company in Indonesia that serve more than 100 millions users per month. I am the one of the person in charge of business process in my service that handling all of the shipping fee and free shipping (known as Bebas Ongkir in Indonesia) courier allocation in the Tokopedia system. The shipping fee are usable to charge the buyer when they order a package then handing over the shipping fee to the settlement with the respective logistic partners.

Existing Condition

The service we maintained here have been lived for approximately 8 years, written in Go programming language. That is nothing wrong the software age actually, but the problem is coming from the features that have been introduced day-by-day for years that is already rooted inside the code that is very hard to remove. The features is deeply rooted to the current business logic, even though the features has been turned off for a couple of years.

After e-commerce industry evolved, the logistic partners also started to add some features to improve their pricing to keep exists and competitive with the other logistics provider as well. This directly affecting with the shipping fee calculation because we store of the partner pricing in our cache and calculate them on the fly to reduce API and network hops to the logistic partners system. This was nicely handled in the current shipping fee service, but the problem is the cache is getting more complex because so many ifs inside the method whether it is for the migration different scheme, different input, different handling for some region and so many more. Other than that, in logistics we also have multiple package handling that is also depends with the shipping fee, such as insurance or cash on delivery payment method. Let me show you the picture to visualize the shipping fee service responsibility in Tokopedia.

User Interface that served by my service.

Before calculating the shipping fee, we do multiple validations such as checking weight, distance, routes and so many featured things. After that we will check whether the cache for the request is exists or not, if the cache doesn't exists we will process the request to the partner and save the cache after the request is finished. After that we need to manipulate text, formatting, package handling modifiers as the app picture shown before.

Most of all, the current technical solution inside the shipping fee service uses the functional programming. I marked things that needed to be decoupled by refactoring with the star sign like the in the image shown below.

Existing condition inside the service.

Why do all of the function marked as the thing needed to be decoupled? Am I anti functional programming? It is a big no. Functional programming is great for its simplicity, but functional programming also preferring things that are composable and can be simplified by using the common known pattern. It means, either it is an object oriented programming or functional programming, both of them are highly encourages that a function is not only mean to solve one problem, but if possible the function needed to be generic to solve another problem easily.

Why Keep Refactoring?

As I described above it is hard to refactor, but why keep executing even though the code is still working correctly? Yes, correct, but sometimes we saw some issues that we don't know where it is coming from (hard to debug). It can reduce the service reliability. There are also some signs that our service are bit slowed down because of unnecessary validation repetition and turned off features that highly coupled (can also be the culprit of some issues). Some of the leader boards suggest that we can start fresh our service, it will absolutely going faster. It is valid argument, but I think that will be just another story like we have been going through these years. Then I started to think that if we start fresh, it will be happened again in the future. So I proposed to use my architecture design to makes it more maintainable and efficient but still managed to get the service is fast enough. That is the idea behind this initiative.

The Thought Process

The principle I always do when designing a software is to divide the complex problem by three phases, pre-processing (input), processing (the actual core business logic lived here), and post-processing (output). By dividing the phases, we can easily identify what we need to change if there is a feature update. Whether it is the input, the process or the output.

The key principle I always use in any problem solving is to decompose the things as small as possible. The keyword "small" here is not implies that the function should only do one logic, but one responsibility. It can be have more than one logic, but as long as it handles one responsibility, I think it is enough to breaking down the complex logic here. One would say that ~30 lines of code is the sweetest spot the one method should handle. I think that is a common mistakes, code responsibility can't be represented with a total lines of code. As long as the function is clear, then you are good to go. By keep anything small, we can easily replace and remove some features that aren't needed anymore. Also, we don't need to hold the thought or compute process that took so much space in our head while debugging the software.

The principle above is kind of abstract and difficult to understand. That's okay, it is because it have no concrete example. As you are solving more problem, you will get the sense of "smaller is simpler". The gut and your intuition will teach you as the days goes by.

This is the grand design to solve the problem above, it is a bit different with the actual design implemented inside Tokopedia, but that is fine we can still learn.

Preprocessing the Request

Let's start with the pre-processing the request. It is happened when the incoming request are processed. Is it need to be overrode or keep as it is, because there are certain cases that need to override the request payload due to features. In this phase also when the validation request are done, allowing us to do some generic validation such as payload data type validation, weight, distances, features, and so many things that can break the shipping and order fulfillment process. This is a really simple process, but involve so many things to be validated.

System Design of the validation logic

The image shown above is the flow within the validation process. It is really simple, so we only need to use procedural and functional call and define functions to do those simple tasks above. Starting to do some generic validator, and then we can call the validation A, override payload with A, and do the validation B and so on.

Process the Request

Continue to the core business logic lies, it is when we do the shipping fee calculation. There bunch of ways to do that and not all of the region can use the calculation. Based on that situations, I got the conclusion that the requests can be handled with several and various operations based on the buyer who wants to place an order, the seller who will fulfill the order, and the logistic partners, or even the service level that can be used to ship the package from the seller to the buyer. It has some rules that need to be fulfilled before deciding who is going to do what and where. Based on those two conditions, we can narrow it down with the requirements below.

1. System can decide where they should look the shipping fee (the data source);

2. System can handle a different case of calculation and caching mechanism; and

3. System have expected the same exact output even for different calculation and the data sources.

The three requirements above represents things that already exists in payment gateway system. I highly inspired with the agnostic payment gateway that built by the community written in PHP, omnipay. Those framework have the agnostic principle approach to maintain the different payment gateway and let the users decide how to use the multiple payment gateway registered in the system. This concept can help us to solve the requirement number two.

But in the omnipay, they don't have the capabilities to decide whether what is going to where. In the payment gateway context, the user is the one who will decide and select where the payment gateway be going. Then I need to build something that can decide where the gateway should going to. I also inspired on how router works, so I made with the idea of implement the static routing and dynamic routing concept. Within the knowledge, we will be able to solve the requirement number one.

To fulfill the requirement number three, it is the most simplest approach of all of them. All of the gateway should have the exact output by defining an interface and using the polymorphism approach, since Go have the capability to define an interface.

Let's see an image below, the end architecture concept we need to solve this problem.

System Design of the core process of shipping fee calculation

In the image shown above, there is shape with a different three colors. It is defined in the legend, blue for polymorphic approach, light purple for functional, and the light green one is just a separate struct to organize our code structure.

Starting from the top, we have the routers, the one that have capability to use the routing rules either it is static or dynamic. The static rules means that if the request is A, then we should go into the gateway A. In the dynamic rules one, it handle all the rules having more complex routing, such as migration, experiment, feature rollout, or region whitelists as long as the rule implementing the DynamicRuleInterface interface. Both of the rules are preloaded in the app starts to reduce overhead in the request. The router will have the end information of the payload should go into the Gateway A / Gateway B / Gateway C and simply forward the request to its destination gateway, concurrently.

In the image, gateway is represented with the blue color, means it needed to comply with the GatewayInterface. Just like the payment gateway has, the client should define all of the GatewayInterface method and implement them separately. It is the polymorphism design that we don't really change as the existing condition. This time, the gateway should also have to know where to look at and how to calculate the shipping fee. Since we don't want to add more complexity to the gateway directly, I decided to breaking them down into the smaller pieces by separating the calculation logic and the cache layer handling on a different struct. Remember the philosophy, keep everything small. In these structs we can throw the cache saving asynchronously, to boost the software more faster, because it is only the data read service that doesn't needed a consistency level of transaction. It is different case with the payment, we have to wait until it is successfully wrote the data. Concurrently, we can calculate the shipping fee results to give the response to the router and combine them as a whole.

After all of the gateway successfully return the results, the router will receive and combine them as a one slice of data, then its job is done.

The approach I choose above is significantly reduce the complexity, because we are able to remove the dynamic routing rules and move into the static one if we don't need rollout or whitelists kind of feature anymore. We can separate the gateway if there is something more complex are coming ahead since we aren't capable of predicting future, but there is a room we spare for. That is the point of decoupling and breaking down the software into several components. Again, keep everything small.

Present the Output

After we receive the results, the structs and format is different than the legacy system. First, we need to introduce a simple layer to translate the structs into the desired outcome, in this example is the legacy response. Simple, easy.

Second, we need to separate the component handling as it is mentioned in the first UI I show. So defining a Decorator interface is a helpful to modify the response is all we need. We can apply some discounts, price aggregation, labeling, adding some wording if the conditions apply. Let's see the diagram below.

Presenter system design as an adapter or decorator.

In the presenter, we can have so many structs we need to modify, manipulate and map the data as needed. In this case, two structs that have responsibility to maintain the legacy flow, and the newer one that implements a sophisticated decorator. As I explained above, the decorator pattern. This can easily make the program runs faster in several cases. Let's say we have multiple APIs that serve different purpose.

1. API to render the UI to user

2. API to calculate something and get the results to be used in another service

3. API to serve the original shipping fee without any component handling

By leveraging the decorator pattern, we can have different strategies to implement the three different API above in a resource efficient manner.

1. In the first API, since we need to render the available information to the real user, we need to use many components such as:

   1. Labeling;

   2. Text manipulation (price aggregation);

   3. Applying discounts;

   4. Rendering UI components (e.g. set disabled and show errors); and

   5. Additional package handling information.

2. In the second API, calculating things can involve so many things such as additional package handling information that can be used in another service such as:

   1. Applying discounts; and

   2. Calculating insurances for package handling

Of course we don't need other things such as labeling, text manipulation, and rendering components that is not needed in this API. Boom, compute resource saving.

3. The third API is the most simplest one, just do some shipping fee calculation and it's done. We can create some adapter to the desired response.

Some Caveats from the Internet Seniors.

There are some caveats from people that preferred to use concrete type and avoid doing abstraction. It is okay because the decision is in theirs, because the pitfalls it can be if they doing it wrong. I think both of the decision can lead to some mistakes if it is not deliberately choose the options. Either they will use abstraction and not using abstraction. Decision is a decision, if it is a mistakes, then be it. The decision is temporary, we still can change the decision and learn from that. Do the things that matters. Besides of that, there are some arguments why I keep do the breaking down and uses polymorphism approach till this day.

Using Interface (Dynamic Dispatch) is Slower than Concrete (Static Dispatch)

There are plenty of input and proof that interface type calls are much slower than the concrete type. It is because of using interface means that the compiler cannot decide and check the memory (if it implements some interface) and will decide in the runtime. Because of interface is known only in the runtime, then it is expected to have the concrete type one that is preloaded when the compile is happened.

https://medium.com/@sanjayshiradwade/understanding-dynamic-vs-static-dispatch-in-go-a5319fcdddec

To me personally it is doesn't matter if the software decoupling value is much higher than the performance drawbacks. The performance drawbacks is not that high actually, the code is still run fast enough. I will show you in the end of the article. I am the one who prefer to organize code by using interfaces and structs rather than separating app into a microservice. That will add much more performance drawbacks, network failure, unknown error, and the other uncertainties. It is really not worth to do the service separation because we will add more unneeded complexity. If the engineer and traffic is not that on the hyperscale then to me it is not worth to do.

Abstraction can make the code worse

There so much contents I saw on youtube, stackoverflow, twitter and so many more people are hate the usage of abstraction. I also hate the abstraction when it comes to inheritance because these things can make our code make much more ugly just to avoid a repetition. This is well explained by this youtube video.

https://www.youtube.com/watch?v=rQlMtztiAoA

TLDR;

Abstraction using inheritance is awful if not well managed. Instead, use polymorphism approach (defining Interface behavior) that can simplify, isolate, and decouple things more clearer. It is okay to have redundant and repeatable code.

Summary

That is the strategy I used to redesign and refactor the system I in charge on. Thankfully I implement the design with my colleagues that are incredible, clever and help me with giving advice along the way. The development is done in 2023 and will be finished rolling out to all endpoints in 2024. This initiative has shown a great metrics and the efforts given is paid off. In the middle of rollout, we can see that our service can receive almost +290% more throughput and -30% on average latency, and -500% max latency after implementing the system design changes.

Simplifying things is not only makes our code is more organized. Because everything is more clear and concise we can manage the code and remove unused modules so it won't overburden the system performance. That's all. Keep everything small to manage simplicity.

Keep in mind that the official guides that can be accessed here. In a Mac with an Apple Silicon CPU, it is not work as intended from the official guide and the article will show the workaround of TimescaleDB extension installation.

Installing Homebrew

If you have installed the homebrew app, then you can skip this step. To install the latest homebrew, we can run the existing script that exist in their homepage.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

After installing homebrew, you can go through the installation of the Timescale DB directly.

Installing TimescaleDB

Installing TimescaleDB need to use the custom third party repositories. To add the TimecaleDB custom third party repository (Tap) to the homebrew, we can use the command below.

brew tap timescale/tap

After adding the repository, we can continue to install the TimescaleDB directly using the brew install command.

brew install timescaledb

Installing TimescaleDB will takes some time, since it will two pakages that consists of PostgreSQL and TimescaleDB.

Setting Up TimescaleDB

By default, TimescaleDB will ship its binaries that can be used to configure the PostgreSQL extension installation. This is when the issue happened in the configuration stage. To configure a TimescaleDB extensions to a PostgreSQL app, we need to use the timescale-db tune command.

timescaledb-tune --quiet --yes

Running command above will resulting an error on Apple Silicon mac devices. This is expected because of the shipped binary from the TimescaleDB tap is compiled using the amd64 instruction sets, while the Apple Silicon mac expect of arm64 instruction.

To fix the error above, we need to recompile the binary into arm64 architecture. Since the TimescaleDB uses Golang and Bash script to setting up the database app instance, we can recompile the installation script to make them work in the arm64 machine. Both of the language are compatible to the arm64 architecture.

Installing Golang

Golang can be installed using this bash command.

brew install go

After go is installed, we can proceed to installing the timescaledb-tune application with the arm64 compatible binary. Thanks to golang that have simple binary compilation on the go using the based on the repository. To install it, the command should be as follow.

$ go install github.com/timescale/timescaledb-tune/cmd/timescaledb-tune@main

The command above will install the timescaledb-tune app into your $GOPATH/bin directory. In our case, because it is default so the expected app binary directory is in the ~/go directory. We will use the recent installation from source of timescaledb repository.

Configuring TimescaleDB

In this time, we will see what the timescaledb-tune bring into our existing PostgreSQL and TimescaleDB installation. Before running our installed timescaledb-tune we need to go into the GOPATH binary directory. In our case, it would be in the ~/go/bin directory. To get into the directory we can use this command below.

cd ~/go/bin
./timescaledb-tune --quiet --yes

The complete installation output can be seen in the picture below.

The result above indicates that our TimescaleDB and PostgreSQL are both installed and successfully configured.

Copying TimescaleDB lib into the PostgreSQL Installation

PostgreSQL and TimescaleDB are now successfully installed, but the extension is not enabled yet in the postgres. To load that, we need to execute the command below. Remember that the <VERSION> tag is replaceable based on the TimescaleDB extension you have installed.

cd /opt/homebrew/Cellar/timescaledb/<VERSION>/bin/

In our case, the <VERSION> tag would be 2.14.2, and the command should be like this.

cd /opt/homebrew/Cellar/timescaledb/2.14.2/bin/

After changing directory, we can procced to move the extensions and library file from TimescaleDB into the main PostgreSQL DB app by using the shipped bash script named timescaledb_move.sh that exist in the directory.

./timescaledb_move.sh

Let's run the script above.

The result shown on the picture above indicates that the script has not going through any failure again. At this point, we can create a PostgreSQL database that enable the TimescaleDB extension.

Standard Streams

Starting with the Standard Streams, it is the most basic fundamentals in the Unix and Unix-Like Operating system like Linux, BSD and Mac OS. By default, it consists of three main file process such as.

1. stdin that means standard input,

2. stdout that means standard output, and

3. stderr that means standard error.

Every application must use of those three standard I/O above to process input and output. As we know, every application have many processes. In each process, they use three standard file descriptors in the unix-like operating system, stdin as the input, stdout as their basic output, and stderr as the error output. To visualize what are those our computer, it can be demonstrated using the picture below.

File Descriptor in standard streams

In the picture above, there is a process will interact with a three numbers, which 0 that belong to stdin, 1 to stdout, and 2 to stderr. The number represented is the file descriptor opened in each process. Since unix system follow the philosophy of "everything as a file", then the standard streams would be accessible from /dev/stdin as the input stream file, /dev/stdout as the standard output stream, and /dev/stderr as the standard error output stream. Digging deeper in the golang standard library, the Stdin, Stdout, and Stderr files are predefined with path explained before.

Let's write a simple demo program using go that use stdin, stdout, and stderr.

package main

import (
    "fmt"
    "io"
    "os"
    "strings"
)

func main() {
    stdin, err := io.ReadAll(os.Stdin)

    if err != nil {
        panic(err)
    }
    str := string(stdin)

    lowercased := strings.ToLower(str)

    fmt.Fprintf(os.Stdout, "Hi %s", strings.ReplaceAll(lowercased, "\n", ", "), )
    fmt.Fprintf(os.Stdout, "\n")
    fmt.Fprintf(os.Stderr, "There is no error, this is just an example\n")
}

The simple go code above would do the following actions:

1. Read all input given in the standard input.

2. Convert the input string to be all lowercase and concatenate them if the inputs is more than one line.

3. Print the lowercased string into standard output.

4. Print something into standard error just for example.

While running those program, we will get the input from the os.Stdin (in golang, it is the standard input stream file. The input then will be buffered into the stdin variable that turned into lowercased string later.

To print an output to the terminal, we prefer using the fmt.Fprintf() function instead of fmt.Println() because under the hood, the fmt.Println method also write bytes to the os.Stdout (golang standard output file). It is the same though, it just to simplify our learning points about the unix basic standard streams. To print out to the standard error output, we also use fmt.Fprintf() function to print the error message to os.Stderr that will be explained later in this post.

In this post, to make sure that you will be learning something, the post will prefer using screenshot only for the command used in the example. This is intended to aim you to try and type the command by yourself.

Standard Input (stdin)

First thing first, let's build the program above using the following command.

go build main.go

The output binary file should be named main in the same directory. Before execution, we will explain several standard input operators such as pipe operator that can be denoted with the | and the < less than operator.

Input Redirection < Operator

Input redirection means that we can replace the input using the files or custom input we decided to use. In this example we use a separated file named names.txt that contains three names in the file.

To redirect the standard input, we can use < (less than) operator. The convention of its full command would be the binary < input . It can be shown in the example below.

The picture above show that we can redirect the input using the < operator to replace the input data streams from /dev/stdin to names.txt.

Pipe | Operator

Pipe will be useful to combine two commands, it will take the output of the first command as the inputs of the second command. Let's take the example of the pipe operator usage in the terminal history below.

The commands used in the picture above is echo that print something to the standard output, and the main command (the previous simple application) will take the echo command as the input of the program.

Hi hai,

There is no error, this is just an example

In the first run, the output "Hai" will be used in the main command as the standard input. Our main command do the lowercasing of the "Hai" input and printed out as "hai" with the following additional output in the terminal.

In the second run, the output from echo command is "Rendy" that will be lowercased by executing the main command as it gave the output below.

Hi rendy,

There is no error, this is just an example

Take another look with another command, ls to show our files in the current directory. Combining with pipe operator and main command gave us similar output.

The ls command giving output of the files inside the directory and piped with the main command. All of the files would nicely concatenated using comma.

Standard Output (stdout)

Standard output has two operators such, that is the output redirection with replacing file and output redirection with append file. The replace operator use the > (greater than), and append operator use >> (double greater than) symbols.

Redirect Output > Operator

First example, we will use the standard unix command like echo to redirect the output into a file.

In the example above, we can understand that the string "Hello" is written through the file named hello.txt. If we do the the same thing with a different string, the hello.txt file will be replaced with the new string. It can be shown by the example below.

As the example above, we don't intentionally delete the files but the old hello.txt file that contains "Hello" got replaced by a new string "Rendy". If we don't want to replace the string, then we should use the append output redirection operator >> .

Append Output >> Operator

In the second attempt, we will using the same file hello.txt and write another string to the this file but using the append >> operator.

In the example above, we can see that the >> operator did not change the file contents of the hello.txt. Instead it will append the "George" string below the existing file contents.

The standard output and standard input redirection operator can be used together with the main program we wrote before.

With the command above, you can see that the Hi rendy string is not printed in the terminal, instead it written in the main.log file. Hence the example error line "There is no error, this is just an example" that is printed in the standard error would still shown in the terminal output because we only redirect the standard output.

Standard Error (stderr)

Basically, redirecting the standard error would still the same intuition as the standard output. Remember that the standard error using 2 as the file descriptor number in this picture? We only need to redirect them using 2 prefix in each operator. For instance, to redirect the standard error we can use 2>, and to redirect append output of a standard error we can use 2>>. Let's try that using the main command.

The picture above shown that we already successfully redirect the standard error to the error.log file. While using the append operator, we will got the similar result as the picture below.

That is the basic usage of the standard input, standard output, and standard error operator in bash system.

Advanced Use Cases

After understanding its function and available operators in the standard streams, there is a bunch another use case combination of its that may help improving our productivity. Utilizing the main program we wrote before, we can redirect the standard output to a file, and the standard error to another file like the picture below.

In some cases, there is needed a bigger input, usually from a file. It is possible by leveraging the < operator as the example below.

There another use case that need to combine all of the output of standard error and standard output in a one file.

Usually we have a bunch of files in the same directory (usually logs). To querying some string, we can use the cat command to list all of the files that match the pattern, and find the relevant information using grep command.

That's all. Understanding the basic standard input and output streams of unix system is worth it. We can improve the productivity because we don't need to open the IDE that may heavy, or even impossible in the production server.

Network is a substantial component and key support in the virtualization environment. In the modern application deployment, we use virtualization to manage resources as well as scaling them. However, there are certain cases that we might need a nested virtualization and containerization to manage more complex app and service deployment like doing a containerization inside the virtual machine that runs on a single host or more.

In this article, we will covering the issues in nested networking problem and how to make LXD network management in a nested virtualization. This approach is still applicable in a non LXD Cluster, but in this article we will be covering specifically in the LXD cluster environment.

LXD Virtualization architecture in the picture above shown that we have a single host, that spawning 4 virtual machines that consists of

1. lxd-1 as the LXD cluster node

2. lxd-2 as the LXD cluster node

3. lxd-3 as the LXD cluster node

4. load-balancer as the load balancer that need to be accessible through the internet.

They were made to manage the resource that can be used in each workloads. In those setup, we will be using three nodes of LXD cluster in the lxd-1, lxd-2 , and lxd-3 VM. LXD cluster can contains several containers to run app and service inside, to make it visible in the host's neighbor network and the internet through the edge router. This is the basic of the data center architecture looks like.

Let's remind the objective of this article, to simplify the networking inter containers and virtual machine communications. However, the load balancer traffic forwarding to the internet through the edge router will not be covered in this post. Based on the objective, this is the deliverables that needed to be done.

1. Each container inside the LXD VM should be able to communicate with each others.

2. Each container inside the LXD VM should be able to communicate with another container from another LXD VM. e.g. container-3 can reach the container 1.

3. Each running containers should be able to be reached from the load balancer VM.

4. Each running containers should be able to be discovered from the host machine.

If you have already LXD cluster installed, then you can skip this step and going to the Problem.

Preparation

In this post, we are using Ubuntu 22.04 distribution as it is a common operating system used in the server. The test laptop used is Lenovo Yoga Slim 7 Pro with the detailed specifications are shown here.

Spinning Up Virtual Machines

Ubuntu multipass is used to create lxd-1 , lxd-2 , lxd-3 , and load-balancer VM instances. Each of the instance will have tiny specification, 1 CPU core and 1GB of RAM and 10GB of storage. Of course another VM tools in the market like virtual box, VMWare, Proxmox, Vagrant, or cloud VPS, or even LXC container can be used as well, it doesn't matter.

multipass launch --cpus 1 --memory 1G --disk 10G --name lxd-1
multipass launch --cpus 1 --memory 1G --disk 10G --name lxd-2
multipass launch --cpus 1 --memory 1G --disk 10G --name lxd-3
multipass launch --cpus 1 --memory 1G --disk 10G --name load-balancer

The command above will spawn three virtual machines with 1 CPU core, 1 Gig of memory, and 10 Gigs of storage. To ensure that all of that machine already live, running multipass list command will show all the instances on the host machine.

The terminal above showing that all of the instances already live and ready to be used in the next step.

Forming a LXD Cluster

Starting from lxd-1, we can start to configure the LXD using this command using interactive shell.

lxd-init

When running the command above, it will shows several questions like the images shown here.

Take a careful look we only need this question below to be answered with yes.

Would you like to use LXD clustering? (yes/no) [default=no]:

Afterwards, we can leave any other questions with the LXD defaults configuration. In the default configuration of LXD cluster, they will automatically create a new Fan Overlay Network. This fan network will help us to solve the problem later on.

Before initializing the lxd-2 and lxd-3, we need to generate the join token that will be used when configuring the rest of nodes. Command to generate token would be as follow.

lxc cluster add lxd-2
lxc cluster add lxd-3

Those tokens are used as an authentication that lxd-1 (as the LXD Cluster leader) are permitting another new cluster member joining the cluster.

Joining member will use the same command as initialize the LXD Cluster leader with mandate the sudo command because it will modify system.

While joining cluster, there is interactive shell questions that to be answered before leaving it with the default configurations.

Would you like to use LXD clustering? (yes/no) [default=no]: yes

What IP address or DNS name should be used to reach this server? [default=10.79.89.132]: (leave it blank)

Are you joining an existing cluster? (yes/no) [default=no]: yes

Do you have a join token? (yes/no/[token]) [default=no]: yes

Please provide join token: (enter the acquired token from lxd-1)

All existing data is lost when joining a cluster, continue? (yes/no) [default=no]: yes

Repeat the command in lxd-3 host. Finally, the LXD cluster are successfully set up. Check whether the LXD Cluster are correctly formed using this command:

lxc cluster list

The command can be called anywhere, either in lxd-1 or lxd-2, or lxd-3 , the result should be showing that all of those host are in the same cluster like the picture below.

The Problem

The problem that we are going to solve is in the networking part, so we may need to revisit the current networking topology after installing the cluster.

The image above shows that the created VM has two IP addresses except for the load-balancer. That is because of the LXD cluster node assign a new interface for its fan overlay network.

TLDR of fan networking;

Fan overlay networking will mask the underlay network (the containers) to the overlay network. As long as we are connected to the overlay network, then we can reach any network available as its underlay network.

The IP address assigned in the lxd-1 instance is coming from lxdfan0 interface that use 240.0.0.0/8 netmask as well as the lxd-2 and lxd-3 has. While initializing the clusters, LXD has managed the networking inside the LXD cluster to be able to communicate and discover the other containers within the cluster. It has been the default network settings for LXD Cluster. Hence, if you are not using the LXD cluster setup, the LXD will use default bridge network.

Let's try to spawn a new container in the any node in the cluster using the command below.

lxc launch ubuntu:22.04 container-1
lxc launch ubuntu:22.04 container-2
lxc launch ubuntu:22.04 container-3

LXD is smart enough to separate the workloads, so the created container will be balanced accordingly based on the free node.

The three containers are up and running and evenly distributed to the LXD cluster node. Now, our network topology with its IP address would be as follow.

With the given conditions, we are having the difference network within the container and the load balancer. Here is the proof that the load balancer are not able to discover the container inside the LXD cluster.

Ping test above showed that the load balancer is not know where the IP 240.228.0.217 are located. In order to make the container available to the load-balancer we need to route the load balancer with the containers. The common way to do this is to append the routing table in the load-balancer instance to one of the LXD cluster because at least one of the instance know where to route to the respected container. The problem using a defined routing is we need to configure the load-balancer and the lxd-* instance to be able to communicate in the both directions.

Solve Using the LXD Cluster Overlay Network

Ubuntu fan networking can easily solve the issues , because adding a new node to join the network is fairly simple and straightforward. The fanctl can be installed by using the ubuntu-fan package in the load-balancer instance.

sudo apt install ubuntu-fan

After the package has been installed, joining a fan network can be done using fanctl as shown in the picture below.

The fanctl up command will register the underlay 10.79.89.225/24 network to the overlay network of 240.0.0.0/8 network. The command before will create two new interfaces that will be used to register the overlay network (fan-240) as a bridge interface and its tunnel interface (ftun0) to route the packet.

Let's try to ping the container again.

It works! Let's try to reverse the other way around.

Voila~ it works 🪄️!!

Apparently, the command above is not persisting the fan-240 config interface. If the system is still using the legacy /etc/network/interfaces config model, then we can use the ifupdown command method as it is explained in the documentation example. However, the Ubuntu 22.04 uses systemd-networkd to configure its host interfaces. Upping interface fan-* requiring executing fanctl command as it is mentioned in the docs, that by default is not supported by systemd-networkd. Another tool that can handle this is the networkd-dispatcher . The complete docs can be accessed in the owner repository.

First, install the networkd-dispatcher package using the command below.

sudo apt install networkd-dispatcher

After completing installation, by default there will be pre-created directories in the /etc/networkd-dispatcher . If the directories doesn't exist, we can create them manually using this command.

sudo mkdir -p /etc/networkd-dispatcher/{routable,dormant,no-carrier,off,carrier,degraded,configuring,configured}.d

Creating a hook script that will be executed when the ens3 interface is up in the load-balancer instance. Create a hook script in the routable.d directory and mark it as an executable. The command would go as follow.

sudo touch /etc/networkd-dispatcher/routable.d/fan.sh
sudo chmod +x /etc/networkd-dispatcher/routable.d/fan.sh

Let's modify the content of the fan.sh using your favorite text editor. In this command, we will be using vim.

sudo vim /etc/networkd-dispatcher/routable.d/fan.sh

The bash script contents can be seen here. Obviously, you need to modify the interface name as my machine use ens3, that can be different on your machine. The fanctl command should be customized as well using the value of your underlying network and overlay network.

#!/bin/bash

if [ "$IFACE" != "ens3" ];
then
    echo "$0: not the interface target, ignoring"
    exit 0
fi

case "$STATE" in
    routable)
        echo "$0: configuring fan network for interface: $IFACE"
        fanctl up -u 10.79.89.225/24 -o 240.0.0.0/8
        ;;
    *)
        echo "$0: nothing to do with $IFACE for \`$STATE'"
        ;;
esac

Restart the instance and the fan-240 interface should be created as well as the ens3 has been configured. To make the host also available in the overlay network, then we just need to do the same things in the load-balancer instance. Restart the instance and see if the fan interface is up with the ens3 interface.

This is a checkpoint on what we have been doing. After setting up the overlay network, we can compare on the complexity problem that exists before.

The picture above is showing that the both the host and the load balancer does not know where to find the specific container inside the LXD Cluster. Registering a new routing table seems to solve the problem, but the directions is only one way not even a roundtrip. For instance, registering a route in the load-balancer to container inside the LXD Cluster through the lxd-3 or lxd-2 or lxd-1, the container-1 still won't be able to reach the load balancer. It will becoming more complex and hard to maintain if we want to add more cluster nodes in the LXD Cluster, or introducing a new workload model like docker container.

In the picture above, it shows that the network topology are becoming flat. It shows that that we can significantly reduce the network complexity. Each instance can communicate each other easily. Even the host can reach the LXD containers and vice-versa. Now, using the current network topology, we will try to add the docker engine to the overlay network.

Connecting Docker To The Overlay Network

From the previous solution we know that fan networking create a new bridge interface. Thus, by the fan network interface attached, means that we can utilize them even further like adding the docker to the overlay network.

We will try to install docker to the load balancer. We won't cover the installation process, but you can follow the official docker installation instruction here. After the installation succeeded, try to launch a new nginx container with docker-container-1 name.

sudo docker container run -d --name docker-container-1 nginx

By default, the docker-container-1 will use the default bridge network and any instances, even the host (baremetal host) cannot reach it except the load-balancer VM as the docker host. To utilize the fan networking, we can create a new bridge network using the fan-240 interface. In this command, we need to add the subnet as the fan network subnet, the IP range using the host network but sliced in the /24 subnet. Then the gateway address using the load-balancer fan IP address. Last, we need to exclude the unwanted IP range to be used, such as 240.225.0.0.

sudo docker network create --driver bridge \
                      -o "com.docker.network.bridge.name=fan-240" \
                      -o "com.docker.network.driver.mtu=1450" \
                      --subnet=240.0.0.0/8 \
                      --ip-range=240.225.0.0/24 \
                      --gateway=240.225.0.1 \
                      --aux-address="net=240.225.0.0" \
                      fanbr0

After running the command above, we can directly attach the new bridge network to the docker container using the command below.

sudo docker network connect fanbr0 docker-container-1

After running those command, all of the member of the network overlay can communicate to the new nginx container within the load-balancer VM. The IP address of the docker-container-1 can be queried using the docker inspect command and filtering the result using jq (installation required).

sudo docker inspect docker-container-1 | jq '.[0] | .NetworkSettings | .Networks | .fanbr0 | .IPAddress'

As the image above, the IP adderss of the nginx container is 240.225.0.2 . Here are the test results of ping and curl in the container-2 that is inside the LXD Cluster to the docker-container-1 inside the docker host load-balancer.

That's all. Nice and clean results. Now, let's draw the final network topology again after the overlay network applications and the docker network connection.

As the picture above new docker-container-1 is just appending the flat structure we already built above. Anyone can connect to anywhere as long as they know the IP address inside the overlay network.

Wrap Up

Network management is a one of the key pillar of cloud, virtualization, and containerization environment that plays a big role. The misconfiguration and mismanagement could add more complexity in the long run. The tested environment are forming three nodes virtual machines that spawn a LXD Cluster inside it plus one virtual machine with another containerization workload model such as docker. The test results demonstrate that usage of the overlay network could significantly redact the configuration management complexity in a nested and complex virtualization and containerization environment.

package main

import (
    "log"
    "net/http"
)

func main() {
    handler := http.NewServeMux()

    handler.HandleFunc("/", HandleHealthCheck)

    log.Printf("listening app in localhost:80")
    if err := http.ListenAndServe("localhost:80", handler); err != nil {
        log.Panic(err)
    }
}

func HandleHealthCheck(rw http.ResponseWriter, r *http.Request) {
    rw.Write([]byte("service is healthy"))
}

Potongan kode di atas adalah aplikasi web sederhana yang jalan di port 80. Port 80 ini kan termasuk di port yang restricted, jadi butuh privilege untuk dapat menjalankannya. Beberapa hal yang bisa dilakukan untuk jalankan aplikasi ini adalah dengan menjalankannya dengan menjadi root terlebih dahulu.

Untuk Menjalankannya, aku juga bikin bash script untuk build dan menjalankan binary yang telah dibuat.

#!/bin/bash
go build -o main app.go

if [[ $1 == '--run' ]]; then
  ./main
fi

Untuk menjalankannya tinggal panggil script-nya aja seperti ini

➜  example-linux-cap$ sh build.sh --run

Maka hasil keluarannya akan kurang lebih seperti di bawah.

2023/07/29 22:10:23 listening app in localhost:80
2023/07/29 22:10:23 listen tcp 127.0.0.1:80: bind: permission denied
panic: listen tcp 127.0.0.1:80: bind: permission denied

goroutine 1 [running]:
log.Panic({0xc0000bff50?, 0x67e1bb?, 0x0?})
        /usr/lib/go/src/log/log.go:384 +0x65
main.main()
        /home/rendy/Workspace/private/example-linux-cap/app.go:15 +0x105

Menjadi Root 🥚

Cara yang paling mudah adalah dengan menjadi root. Menjalankannya hanya cukup dengan prefix sudo. Untuk awalan mari gunakan cara bodoh untuk menjalankan aplikasi tersebut. Kenapa cara bodoh, karena dapat mengakibatkan peretas mendapatkan bug yang ada di aplikasi dan mengeksploitasinya. Tapi tidak apa-apa karena ini bagian dari belajar. Nanti kita juga akan belajar cara yang lebih baik.

Pertama-tama ganti user ke root terlebih dahulu.

➜  example-linux-cap$ sudo su
➜  example-linux-cap sudo su
[sudo] password for rendy: 
[root@canvas-mobile example-linux-cap]# whoami
root

Setelah menjadi root, mari dicoba kembali untuk menjalankan aplikasi.

[root@canvas-mobile example-linux-cap]# sh build.sh --run
2023/07/29 22:21:38 listening app in localhost:80

Aplikasi sukses berjalan. Untuk memastikan, bisa menggunakan perintah curl ke localhost:80. Hasilnya akan seperti di bawah.

➜  example-linux-cap curl localhost:80
service is healthy%

Cara ini juga bisa diraih dengan menggunakan sudo, tanpa mengganti user ke root. Caranya adalah seperti ini.

[root@canvas-mobile example-linux-cap]$ sudo sh build.sh --run
2023/07/29 22:21:38 listening app in localhost:80

Untuk memastikan, dapat menggunakan perintah curl seperti yang sebelumnya.

➜  example-linux-cap curl localhost:80
service is healthy%

Cara ini sukses, tapi sangat tidak dianjurkan demi keamanan server, karena ketika sekali saja terkena serangan exploit, seluruh akses di server akan juga dapat diambil alih oleh penyerang (hacker). Fatal banget pengaruhnya.

Menggunakan Linux Cap 🎩

Sekarang, menuju ke hidangan utama yaitu ke Linux capabilities. Sebenarnya capabilities ini sudah lama dirilis, sejak kernel versi 2.2, tapi dokumentasinya cukup minim. Kegunaannya juga cukup low level, jadi ini jarang digunakan oleh end-user. Tapi sebenarnya fitur ini banyak digunakan untuk menjalankan aplikasi yang rootless, seperti podman yang merupakan tool untuk memanajemen container yang dapat berjalan secara rootless.

Kembali ke linux capabilities, dilansir dari laman linux man-pages (Halaman manual linux), capabilities ini dipecah menjadi lebih kecil-kecil sesuai dengan aksi yang ingin dilakukan. Untuk lengkapnya, bisa dilihat langsung di halaman dokumentasinya, namun sebagai contoh, ini aku lampirkan sedikit di bawah.

CAP_NET_BIND_SERVICE
    Bind a socket to Internet domain privileged ports (port
    numbers less than 1024).

CAP_NET_BROADCAST
    (Unused)  Make socket broadcasts, and listen to
    multicasts.

CAP_NET_RAW
    •  Use RAW and PACKET sockets;
    •  bind to any address for transparent proxying.

Pada kasus ini, yang diperlukan untuk listen di restricted port, berarti memerlukan capabilityCAP_NET_BIND_SERVICE. Untuk memberikan capabilities pada sebuah file atau binary, diperlukan pengetahuan juga terkait tipe capability yang akan dilampirkan. Dikutip dari laman linux man, ada 3 tipe capability yang tersedia.

Permitted (formerly known as forced):
    These capabilities are automatically permitted to the
    thread, regardless of the thread's inheritable
    capabilities.

Inheritable (formerly known as allowed):
    This set is ANDed with the thread's inheritable set to
    determine which inheritable capabilities are enabled in
    the permitted set of the thread after the execve(2).

Effective:
    This is not a set, but rather just a single bit.  If this
    bit is set, then during an execve(2) all of the new
    permitted capabilities for the thread are also raised in
    the effective set.  If this bit is not set, then after an
    execve(2), none of the new permitted capabilities is in
    the new effective set.

Berdasarkan penjelasan tersebut, kita tidak bisa menggunakan tipe Inheritable karena dia tipenya diwariskan, jadi belum parent process thread yang akan dijalankan akan memiliki capabilityCAP_NET_BIND_SERVICE. Sehingga, kita perlu menambahkan capability CAP_NET_BIND_SERVICE dengan tipe Permitted (untuk memastikan) dan Effective. Setelah mengetahui, berarti build script yang telah dibuat tadi perlu dilakukan penambahan. Untuk memberikan capability pada sebuah file, terdapaat perintah program setcap.

Untuk menggunakannya, diperlukan minimal 2 argumen yang pertama adalah capability nya dalam bentuk string dan target file nya

# setcap <capabilities> <target-file>

Berdasarkan dokumentasi, format capabilities string berbentuk <capability>=type. Untuk capability yang dibutuhkan yaitu CAP_NET_BIND_SERVICE dan tipenya disingkat, e untuk effective dan p untuk permitted. Sehingga formatnya menjadi seperti ini cap_net_bind_service=ep. Setelah itu, build script yang tadi diubah menjadi seperti ini.

#!/bin/bash
go build -o main app.go
sudo setcap 'cap_net_bind_service=ep' main

if [[ $1 == '--run' ]]; then
  ./main
fi

The Moment of truth 🥁

Sekarang waktunya membuktikan apakah berhasil menggunakan linux capabilities.

➜  example-linux-cap sh build.sh 
➜  example-linux-cap ./main 
2023/07/29 22:55:19 listening app in localhost:80

Hasil keluaran terminal di atas menandakan bahwa tanpa user root, menjalankan aplikasi web di restricted port tetap bisa dicapai menggunakan linux capabilities.

Jadi apa yang bisa disimpulkan terkait percobaan ini? Ya tidak semuanya harus menjadi root. Linux capabilities memberikan kenyamanan untuk mengatur dan mengerucutkan permission dengan lebih detail.

Agar aman dalam menggunakan linux cap adalah, ekspektasinya memberikan capabilities pada sebuah binary ketika installation. Karena di saat itu-lah membutuhkan akses root untuk menambahkan capability. Di luar itu, linux kernel yang akan melakukan pengecekan capability pada sebuah binary. Sehingga scopenya benar-benar kecil, yaitu di level capability pada sebuah thread process, tidak di level user. Dengan terbatasnya permission yang diset pada sebuah binary, dapat memungkinkan kita memperkecil kemungkinan untuk diretas. Istilah kerennya sih Hardening -- 🚧.

Laptop ini yang rusak layar dan keyboardnya. Jikalaupun dijual juga sayang, karena banyak kenangan dan jasanya 😚, salah satunya laptop ini bikin aku seneng ngoprek-ngoprek waktu masih SMP.

Fitur Server Impian

Berbagi Koneksi Internet

• Berbagi koneksi internet, karena biasanya aku share internet pake hotspot dari smartphone. Hotspot itu bikin smartphone aku panas dan baterainya cepat habis, mungkin ini bisa jadi solusi yang efektif.

Web Server

• Pas ada proyek IT atau yang berhubungan dengan hal itu, server ini bisa buat tempat untuk hosting website sendiri.

DNS Server

• Buar bisa buka reddit sama netflix yang diblock sama ISP. hehe.

• Setelah host website di local network , website tersebut dapat diatur menggunakan domain tertentu agar tidak perlu menghafalkan alamat IPnya pas ingin akses.

• Mengembangkan aplikasi android lebih gampang untuk dijadikan host API untuk development. Karena nggak perlu mengubah file /etc/hosts di device android untuk resolve domain lokal.

Cloud Server

• Google Drive x Google Photos / iCloud x iCloud Photos Replacement. Jadi data-data pribadi menjadi lebih aman dan gratis.

Print Server

• Misal ingin ngeprint, dapat dilakukan di mana saja asalkan terhubung dengan jaringan rumah. Nggak perlu colok-colok USB printer lagi.

Deployment Sandbox

• Akan berguna kalo nanti diinstal kubernetes cluster dan docker di servernya, untuk belajar DevOps.

Banyak kan kegunaannya! Kembali ke topik awal, jadi laptop ini akan digunakan untuk home server yang fiturnya sudah kusebutkan tadi.

Perlengkapan dan Bahan

PC sebagai server

Untuk PC aku pake laptopku yang rusak tadi. Speknya biasa saja, tapi sudah lumayan juga jaman dulu. Ini speknya :

Core i3 Sandy Bridge 2.2 Ghz dual core, RAM 4GB, HDD 1TB.

Server ini bisa diganti pakai Raspberry PI, atau Desktop PC. Terserah ya.

Sebenernya ingin menggunakan Raspberry PI, karena bentuknya kecil dan hemat daya karena pas dihitung-hitung, konsumsi listriknya jika menggunakan Raspberry PI akan 4 kali lebih efisien (15watt) daripada menggunakan laptop (65watt). Tapi kalau menggunakan Raspberry PI, jangan diinstall kubernetes cluster ya, masih belum tau apakah dia mampu atau tidak. 😆

Router

Untuk router, aku menggunakan router yang terinstall routerOS yang paling murah yaitu MikroTik HAP Lite 2 karena kebutuhannya menurutku yang sangat ringan, jadi tidak perlu yang banyak fitur.

Koneksi Internet

Cukup menggunakan modem 3g lama yang menganggur dari sewaktu SMP. Kenapa pake 3g ?? Ya, karena 3g dari ISP ini sudah lumayan kencang. Tapi nanti akan diupgrade menggunakan modem 4g atau menggunakan internet kabel yaa. Yang penting tujuan-tujuan itu bisa tercapai terlebih dahulu. ISP yang digunakan bebas ya jika menggunakan modem, tapi aku menggunakan Telkomsel By.U.

Kabel LAN

Yang terpenting juga, kabel LAN nya jangan lupa! Pakai Cat 5 / Cat 5e terserah. Tapi jika kecepatan internet yang dimiliki di atas 100Mbps, maka membutuhkan kabel LAN bertipe Cat 5e atau 6. Untuk gawai yang tidak memiliki port ethernet, beli USB to Ethernet converter juga ya.

Printer

Ini tidak esensial, jika tidak ingin menggunakan Printer Server. Disini aku mempunyai printer, tapi kondisinya rusak, belum tau apakah akan diperbaiki atau beli baru :weary:. Fitur ini akan ditunda dulu entah sampapi kapan.

Total Pengeluaran

Totalnya cuman Rp. 342.000 kalau mau memanfaatkan barang bekas sepertiku haha.

Topologi

Sebelum eksekusi, baiknya direncanakan dulu apakah idenya feasible dan memungkinkan untuk direalisasikan. Topologi yang akan aku pakai adalah seperti ini.

Jadi akan dibuat seperti ini, karena koneksi menggunakan modem, maka server akan digunakan sebagai gateway karena dial-up modem dilakukan di server.

Kemudian, koneksi internet dari server akan diteruskan ke router dan dibagikan ke host-host yang lain menggunakan wireless ataupun ethernet

Perangkat Lunak

1. OS Server menggunakan Ubuntu Server 18.04 LTS

2. Perlindungan firewall menggunakan ufw

3. Dial-up Modem menggunakan wvdial

4. Masquerading untuk berbagi koneksi internet ke port ethernet menggunakan iptables

5. DNS Server menggunakan dnscrypt-proxy

6. Web Server menggunakan nginx

7. Cloud Server menggunakan Nextcloud

8. Containerization menggunakan Docker

9. OS Router menggunakan MikroTIK

10. Winbox untuk konfigurasi MikroTIK

11. Untuk melakukan dial-up modem di background dan juga secara otomatis dapat menggunakan Supervisor

Ilustrasi

Sejauh pemakaian 2 minggu, beberapa permasalahan yang sering aku alami:

1. Koneksi akan tidak bisa digunakan jika salah satu client melakukan download, terlebih jika server sedang update. Jadi client yang lain nggak kebagian bandwidth dan malah terkesan tidak mendapat koneksi internet. Masih belum dapat disimpulkan apakah karena modemnya atau ada konfigurasi yang masih belum pas.

UPDATE

1. Jadi, sekarang aku pake internet provider, karena habis banyak di kuota hehe.

2. Sekarang pakai raspberrypi, karena laptop itu benar-benar sudah mati total karena iseng pakai DVD ROMnya (tidak tahu hubungan antara DVD ROM dan mati-totalnya). Sekarang jadi benar-benar hemat listrik.

3. Karena mikrotiknya ada issue, jadi sekarang ganti pakai router 5Ghz punya xiaomi.

4. Karena pake internet provider, aku cuman bisa akses reddit aja, karena netflixnya diblok biar nggak bisa akses hostnya.

Intinya, mulai seadanya dulu tidak ada masalah, terus di-upgrade pelan-pelan. Tidak semuanya langsung bagus, semuanya perlu proses. Cukup di sini ceritaku. Jadi tunggu ceritaku selanjutnya ya~~ 🙈