Local development environment requirements

I’m assuming that your environment is based or on Linux Desktop distribution or macOS. We will be trying to recreate locally infrastructure which will be looking as near to the real production infrastructure as this is possible. Our main plan is to introduce many ways we can achieve nice and clean work with our microservices.

If you are using Windows and you will be kind enough to make others live better just prepare merge request with instructions for Windows.

Our workshops are introducing a lot of tools which can be used for the local development. Some of them are mandatory and some just optional. In the workshop guide you will be attending you should find which of those tools you need. In case of the stationary/online workshops we will be sending information about what is required in the meeting invitation.

Managers

Brew

Brew[1] is a very good package manager which will help in our work. As many next steps using is as a tool to install other tools we need to be sure you have installed on the target machine. You can find installation instructions here: https://brew.sh/

We can use different ways of installing all tools we plan to use during this workshop but I strongly advice use it as most of the examples here use it as tool for installing them.

Services

Docker Engine

Docker Engine[2] is an open source containerization technology for building and containerizing your applications and will be required in our work.

We will be using Docker to run part of the infrastructure required by our services to be developed and tested.

Linux

For Linux most common way to get docker is by installing it with steps available here: https://docs.docker.com/engine/install/

If you are using Linux standard procedure of installing Docker Engine please also remember about post install procedure: https://docs.docker.com/engine/install/linux-postinstall/ to allow your user to use docker engine.

It is also possible to just install Docker Desktop available here: https://docs.docker.com/get-docker/ which have Docker Engine included. The main problem is that it is paid for companies.

Windows

For Windows preferred way is to use Brew. Steps for installing it looks like this:

  1. (IF Docker desktop installed - Remove/Uninstall it and reboot )

  2. Make sure you are using WSL v2 :

    wsl -v
Example output:
WSL version: 2.1.5.0

    • If WSL Version is any of 2 or higher , continue to step 5

  3. Start CMD. For update Running CMD as Administrator might be required. ( PowerShell as Administrator in Windows 11 should work, but some misbehavior and problems with it where already reported ).

  4. Make sure WSL is in version 2

    wsl -v

  5. Install WSL of your choice, to list available type :

    wsl --list --online

    • To install :

      wsl --install -d DISTRO-NAME

    • (for example Ubuntu) type:

      wsl --install -d Ubuntu

    • Provide username and password for WSL system - those are different and independent credentials to main Windows accounts.

  6. Login to WSL ( by default already logged in to WSL directly after starting it for the first time)

    wsl -d Ubuntu

  7. In WSL required curl, git and build-essential might be not available by default and may need to be installed:

    # sudo apt update && sudo apt install -y curl git build-essential

  8. In WSL install brew tool to simplify and unify tools management and installation ( https://brew.sh/ )

    # /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

    • Remember after installation to read and perform steps listed in ⇒ Next Steps

  9. In WSL, it is easier to install docker as system component to setup all required services:

    # sudo apt install docker.io

  10. Install docker and docker-compose and all other required components in wsl:

    # brew install docker
# brew install docker-compose
# brew install docker-buildx

  11. Add to the WSL config file /etc/wsl.conf the following lines:

    [boot]
systemd=true
[network]
generateResolvConf=false

  12. Allow our user to work with docker by adding it to docker user group:

    # sudo usermod -aG docker $USER

  13. Restart WSL (from Windows CMD/PowerShell

    • shut down all WSL instances

      wsl --shutdown

    • start again our WSL

      wsl -d Ubuntu

  14. Verify docker works properly:

    # docker ps
# docker -v
# docker-compose -v

    • and finally sample container:

      # docker run hello-world

  15. If hello-world run properly - than everything is set and ready to work.

In case of issues with connectivity to work around issues with DNS in WSL it might be easiest to add another DNS server to WSL config.

# sudo bash -c 'echo "nameserver 1.1.1.1" >> /etc/resolv.conf'

It is possible to install number of WSL Linux distributions at the same time, but keep in mind, that services for docker should be served only by one at the same time. Be aware which one is currently running to avoid conflicts and issues:

wsl -l -v

Reference: WSL2 with Docker without Docker Desktop : https://nickjanetakis.com/blog/install-docker-in-wsl-2-without-docker-desktop

MacOs

We need to install docker cli with brew.

brew install docker

Please note that currently docker engine is not available without Docker Desktop, and we need to install a replacement like Rancher Desktop or Colima. In our case we will use Colima

brew install colima

For trivy we need to force to use default docker engine sock location

sudo ln -sf $HOME/.colima/default/docker.sock /var/run/docker.sock

After installing it we need to remember about starting it whenever we need it:

colima start --dns 8.8.8.8 --dns 1.1.1.1 --arch aarch64 --vm-type=vz --vz-rosetta --cpu 4 --memory 8

It is also possible to just install Docker Desktop available here: https://docs.docker.com/get-docker/ which have Docker Engine included. The main problem is that it is paid for companies.

In ~/.docker/config.json change credsStore to credStore.

Tools

Ansible

Ansible[3] is a radically simple IT automation engine we will be using here for a configuration management.

Most common ways to install Ansible is by using steps available here: https://docs.ansible.com/ansible/latest/installation_guide/index.html

Other method which can be used to install ansible is to use brew.

brew install ansible

Sometimes you also need to configure locales if you do not have them configured

export LC_ALL="en_US.UTF-8"
export LC_CTYPE="en_US.UTF-8"
sudo dpkg-reconfigure locales

We will be using Ansible to generate from templates our service’s configuration and special values files for our helm charts used for a deployment process.

Docker Compose

Docker Compose[4] is a tool for defining and running multi-container Docker applications.

We will use it in the console for two ways of work. First is when we will be actively working with our application, and we will be using docker-compose-local.yml file with only our dependencies like by example databases which our microservice requires and other microservices or their mockups which our microservice needs. Second is when we will plan to test also our multi-stage.Dockerfile or single-stage.Dockerfile. We will run emulated environment with docker container prepared from it with docker-compose-test.yml.

For Linux most common way to get docker is by installing it with steps available here: https://docs.docker.com/compose/install/compose-plugin/ - it is also possible to just install Docker Desktop available here: https://docs.docker.com/get-docker/ which have Docker Compose included. Other method available for use is to use brew.

brew install docker-compose

If you have Docker Desktop you should not install it as you will have it already installed

In case of macOS compose is a Docker plugin. For Docker to find the plugin, add "cliPluginsExtraDirs" to ~/.docker/config.json:

"cliPluginsExtraDirs": [
    "/opt/homebrew/lib/docker/cli-plugins"
]

Additionally, if for some unknown reason system will have issues finding docker-compose you can do these operations:

mkdir -p ~/.docker/cli-plugins
ln -sfn /opt/homebrew/opt/docker-compose/bin/docker-compose ~/.docker/cli-plugins/docker-compose

Docker BuildX

buildx is a Docker CLI plugin for extended build capabilities with BuildKit.`

brew install docker-buildx

In case of macOS for Docker to find the plugin, add "cliPluginsExtraDirs" to ~/.docker/config.json:

"cliPluginsExtraDirs": [
    "/opt/homebrew/lib/docker/cli-plugins"
]

Create a new Buildx builder instance with the name "multiplatform-builder"

docker buildx create --name multiplatform-builder

Use the new builder instance by running:

docker buildx use multiplatform-builder

Verify that the builder instance is configured for multi-platform builds:

docker buildx inspect --bootstrap

kubectl

Command line tool kubectl[5] is commonly used to manage K8s[6] and K3s[7] clusters and we will need it for our daily work.

There is many methods of installing it and the most common one is available in this link: https://kubernetes.io/docs/tasks/tools/

Other method of installing it and prefered by Me is brew.

brew install kubectl

kubectx

Command line tools kubectx and kubens[8] are very helpful which switching clusters and namespaces.

Those tools are optional and can be installed with brew.

brew install kubectx

K3d

For the local work we will be using k3d[9] which just wraps K3s clusters we will be using.

We can install it with a help of brew.

brew install k3d

Commands connected to the K3s clusters management can be found just by using --help by example: k3d cluster --help will show available commends for the cluster command.

Manage cluster(s)

Usage:
  k3d cluster [flags]
  k3d cluster [command]

Available Commands:
  create      Create a new cluster
  delete      Delete cluster(s).
  edit        [EXPERIMENTAL] Edit cluster(s).
  list        List cluster(s)
  start       Start existing k3d cluster(s)
  stop        Stop existing k3d cluster(s)

Flags:
  -h, --help   help for cluster

Global Flags:
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)

Use "k3d cluster [command] --help" for more information about a command.

You will be most probably using often start and stop commands to be sure your machine resources are in the optimum use and create when creating new clusters.

K9s

K9s[10] is a terminal based UI to interact with your Kubernetes clusters. The aim of this project is to make it easier to navigate, observe and manage your deployed applications in the wild. K9s continually watches Kubernetes for changes and offers subsequent commands to interact with your observed resources.

We can install it with the help of brew.

brew install k9s

kube-capacity

kube-capacity[11] is a simple CLI that provides an overview of the resource requests, limits, and utilization in a Kubernetes cluster. It attempts to combine the best parts of the output from kubectl top and kubectl describe into an easy-to-use CLI focused on cluster resources.

brew tap robscott/tap
brew install robscott/tap/kube-capacity

Pipenv

Python Dev Workflow for Humans PipEnv[12] is required to work on mockups.

brew install pipenv

Helm

The package manager for Kubernetes helm[13] will help us with managing our deployments with a use of the helm charts.

We can install it with the help of brew.

brew install helm

This tool will be used to install helm charts in our K8s and K3s clusters.

We will be using this tool to check prepared for our microservice helm chart - which will be used to deploy our service to the production environment. We will be testing it by using our local K3s cluster.

Terraform

Terraform[14] is an infrastructure as code tool that enables you to safely and predictably provision and manage infrastructure in any cloud.

We can install it with the help of brew in two method.

Method 1

brew install warrensbox/tap/tfswitch

This tool will be used to choose current terraform version based on the terraform file/files in the folder we are currently in. This is very useful when more than one versions of the Terraform is used in the project. You will need to use tfswitch prior to use terraform command to switch to the required version.

Method 2

brew install terraform

This method now is preferred as most of the new projects requires just 1.0 compatibility and latest versions of terraform 1.x is the best choosing.

Gitlab Runner

Gitlab runner can be installed in a many ways https://docs.gitlab.com/runner/install/, but we plan to use Brew for this.

brew install gitlab-runner

Go to Runners section which in my case is: https://gitlab.com/groups/devops-training-info/-/runners

Click Create group runner.

In my case I’m configuring shell runner and in Tags I’m putting shell. This runner also will not be used for untagged jobs, please be sure that you will NOT check the untagged job checkbox.

After clicking Create runner you will have instruction what to do next.

After creating this runner in our case we will need to use

gitlab-runner run

During your workshop you will be using Kubernetes based runner. Guide how to create it can be found here: https://gitlab-runner-k3d-guide-1aca27.gitlab.io.

Bruno

CLI of the open-source IDE For exploring and testing APIs https://www.usebruno.com/

We can install it with a help of brew.

brew install bruno-cli

Security Scanners

Trivy

Container security scanner trivy[15] will be used to scan locally our prepared docker containers

We can install it with the help of brew.

brew install trivy
This free tool is very useful for the containers security scanning. It is known as have much less false positives than other tools but also detect fewer vulnerabilities than other tools.

Grype

Container security scanner grype[16] will be used to scan locally our prepared docker containers

We can install it with the help of brew.

brew tap anchore/grype
brew install grype
This free tool is also very useful for the containers security scanning. It is known to have more false positives than other tools but in plus it detects more vulnerabilities than other free tools.

Languages

GO

Go[17] language compiler is required when our example is created in this language.

brew install go

Now in your ~/.profile if you are using Linux or in ~/.zshrc add those 2 lines:

export GOPATH=$HOME/go
export PATH=$GOPATH/bin:$PATH

JavaScript

Node.js[18] which is an open-source, cross-platform JavaScript runtime environment is required to work with our frontend examples.

brew install node

we also need YARN to be installed:

brew install yarn

Python

Python[19] language is required to serve projects documentation which is generated Asciidoctor, and to work on mockups.

brew install python

Ruby

Ruby[20] language is required to generate projects documentation which is based on the Asciidoctor.

brew install ruby

Creative Commons Licence This work is licensed under a Creative Commons Attribution 4.0 International License

1. The Missing Package Manager for macOS (or Linux) https://brew.sh/
2. Docker Engine https://docs.docker.com/engine/
3. How Ansible Works https://www.ansible.com/overview/how-ansible-works
4. Docker Compose https://docs.docker.com/compose/
5. Command line tool (kubectl) https://kubernetes.io/docs/reference/kubectl/
6. Kubernetes https://kubernetes.io/
7. Lightweight Kubernetes https://k3s.io/
8. kubectx + kubens: Power tools for kubectl https://github.com/ahmetb/kubectx
9. Lightweight wrapper to run k3s https://k3d.io/
10. The terminal based UI for Kubernetes K9s https://k9scli.io/
11. kube-capacity https://github.com/robscott/kube-capacity
12. Python Dev Workflow for Humans https://pipenv.pypa.io/en/latest/
13. The package manager for Kubernetes https://helm.sh/
14. The infrastructure as code tool https://www.terraform.io/
15. Trivy https://aquasecurity.github.io/trivy/
16. Grype https://github.com/anchore/grype
17. Go language https://go.dev/
18. Node JS https://nodejs.org/en
19. Python language https://www.python.org/
20. Ruby language https://www.ruby-lang.org/en/