Welcome to DevOps With Brian - Linode Terraform’s documentation!

This documentation goes over the initial setup of everything required to setup a Linode LKE cluster as code via Terraform. This currently sets up the following after complete:

Linode LKE - Kubernetes Cluster with 3 shared nodes
Linode node_balancer - Using nginx ingress it will use this to allow for public resources if wanted from kubernetes.
Linode Domain - Sets up your own domain in linode to be used for dns resolution in kubernetes as well as for a githubpages if desired.

This also sets up cert-manager in kubernetes and allows for auto cert generation using Let’s Encrypt using annocations on our deployments.

Initial Setup

This documentation section goes over the initial setup of everything required to setup a Linode LKE cluster as code via Terraform. This currently sets up the following after complete:

Linode LKE - Kubernetes Cluster with 3 shared nodes
Linode node_balancer - Using nginx ingress it will use this to allow for public resources if wanted from kubernetes.
Linode Domain - Custom domain in linode to be used for dns resolution in kubernetes as well as for a githubpages if desired.

This also sets up cert-manager in kubernetes and allows for auto cert generation using Let’s Encrypt using annocations on our deployments.

There are a few prerequisites if you decide to use this whole project. The following sections go over some of those and how to set them up.

Commitizen & Conventional Commits

This repo uses Conventional Commits along with Commitizen to allow for auto versioning and such.

Ensure you are using conventional commits for your commit messages and ensure you have installed Commitizen as well from the link provided above.

Custom Domain

For the entire project the way I am using it you will need your own domain, otherwise you can skip the ingress ssl and domain parts and only use the LKE Terraform.

Once you setup your own domain you are going to want to point it to the Linode nameservers:

ns1.linode.com
ns2.linode.com
ns3.linode.com
ns4.linode.com
ns5.linode.com

Terraform

You will also want to ensure you have Terraform Downloads installed.

Also this repo uses remote state located in Terraform Cloud, more information can be found at https://www.hashicorp.com/products/terraform/pricing. Currently using the Free tier and setup an account for free. After setting up a free account you will want to generate an API token as discussed in https://developer.hashicorp.com/terraform/tutorials/cloud-get-started/cloud-login.

Linode Account

You will also want to sign up for a Linode account if you don’t already have one.

This setup if used completely will setup as previously stated above the LKE, Node_Balancer, and Domain setup in Linode.

For more information on Terraform on Linode for LKE please see Deploy LKE Cluster Using Terraform

Kubernetes Setup

This section outlines how to setup the Linode LKE cluster via Terraform code. All of the steps below will be ran from the lke directory.

Linode API Token

We will need a Linode API token in order for this to work and be able to setup the resources. You can find a how to located at https://www.linode.com/docs/guides/getting-started-with-the-linode-api/

The scopes you need to give it access to are:

Domains - Read/Write
Kubernetes - Read/Write
IPs - Read/Write
Linodes - Read/Write

This API token will become the TF_VAR_token mentioned in the next section.

Terraform Variables

The first thing we need to do is setup some Terraform variables that we are going to be using.

Sensitive Variables

There are some variables throughout this setup that are sensitive and you don’t want to store in your terraform.tfvars file, so for these you will do a export command to set the variables on your own shell.

As mentioned already in the previous step you will want to from your shell run export TF_VAR_token=XXX which is the Linode API token you already setup in the previous section.

This will be the only required variable that is a secret for the lke setup, when we setup the domain section next it will require more variables.

The other variables can be found in the lke/terraform.tfvars file. These variables below control the cluster label and size of cluster, etc.

Other Variables

There are a few variables in the lke/terraform.tfvars file that need to be set in order to ensure your cluster is setup how you want. This section will outline those variables.

Modify these for your needs.

The label controls what the cluster label will be named.
The k8s_version tells it what version of kubernetes to use.
The region tells what location to build the cluster in.
The pools is a list variable that tells what type of nodes to use and how many. Currently it is setup to use shared nodes, All node types can be found Here

Note

Currently the pool setting defined configures the following:

A shared 2GB Linode with 1 vcpu and costs about $10 per node per month for a total cost of $30, plus $10 a month for the load balancer.

Terraform Init & Plan

Now that we have our variables all setup and should have Terraform installed now, we can initialize our project and run a plan to verify what it will do.

Make sure you are in the lke folder and run the following:

terraform init - This will initialize everything needed for the project to run and install modules.

Once this is complete you can now run the plan command to validate you have all your vars setup and it can generate everything properly before applying it:

terraform plan -var-file="terraform.tfvars" which will give a output of the information that it will deploy, validate this looks right before continuing.

Deploy LKE Terraform

As long as we didn’t have any issues with the previous Init & Plan step we can now deploy our cluster.

Ensuring we still have our TF_VAR_token exported on our shell then we can run:

terraform apply -var-file="terraform.tfvars" which should ask for a yes prompt and then will deploy the cluster and will generate your kubeconfig to connect to it.

Note

This step can take a few to complete since it has to spin up nodes and set it all up, so be patient.

Connecting To New Kubernetes Cluster

After running the terraform apply command to deploy the cluster we should now have a kube-config file in our directory, so from this dir we can now run kubectl commands to interact with the cluster, export this as seen below.

Add this to the $KUBECONFIG env var:

export KUBECONFIG=$(pwd)/kube-config

If for some reason you ever delete your kube-config or lose it, you can regenerate it via:

export KUBE_VAR=`terraform output kubeconfig` && echo $KUBE_VAR | base64 -di > kube-config
export KUBECONFIG=$(pwd)/kube-config

This will recreate it and set the path of KUBECONFIG to it to be used.

Now we should be able to run kubectl cluster-info to get the info from the cluster which confirms we can access it:

Kubernetes control plane is running at https://XXXX.us-east-2.linodelke.net:443
KubeDNS is running at https://XXXX.us-east-2.linodelke.net:443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Go ahead and run the following command to get the node_balancer external-ip address which you will need for the next dns steps:

kubectl -n default get services -o wide ingress-ingress-nginx-controller

This should give us something like:

NAME                          TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE     SELECTOR
my-ingress-nginx-controller   LoadBalancer   10.128.169.60   192.0.2.0   80:32401/TCP,443:30830/TCP   7h51m   app.kubernetes.io/instance=cingress-nginx,app.kubernetes.io/name=ingress-nginx

Let’s move on to the dns folder and steps.

Linode DNS/Domain Setup

This section is used to setup the Linode DNS options if you want to connect a custom domain and be able to do SSL for public host Kubernetes apps.

There is also configuration to setup the required A records for my domain which is a var in the TF code to the Github Pages addresses. By doing this I can host my other github pages repo on my custom domain.

Ensure you are in the dns folder for these steps.

Linode API Token

Just as we setup before for lk3, we will need a Linode API token in order for this to work and be able to setup the resources. You can find a how to located at https://www.linode.com/docs/guides/getting-started-with-the-linode-api/

The scopes you need to give it access to are:

Domains - Read/Write
Kubernetes - Read/Write
IPs - Read/Write
Linodes - Read/Write

This API token will become the TF_VAR_token mentioned in the next section.

Terraform Variables

The first thing we need to do is setup some Terraform variables that we are going to be using.

Sensitive Variables

There are some variables throughout this setup that are sensitive and you don’t want to store in your terraform.tfvars file, so for these you will do a export command to set the variables on your own shell.

As mentioned already in the previous step you will want to from your shell run export TF_VAR_token=XXX which is the Linode API token you already setup in the previous section.
The export TF_VAR_soa_email=xxx@xxx.com needs to be ran to export the email that is associated with the domain when registered.
The export TF_VAR_nodebalancer_ip=X.X.X.X will be the IP of the nodebalancer that was setup in the previous kubernetes step.

Other Variables

There are a few variables in the dns/terraform.tfvars file that need to be set in order to ensure your dns/domain is setup how you want. This section will outline those variables.

Modify these for your needs.

The domain_name variable is the domain name you will be setting up in linode.
The github_pages_alias is used for github pages custom domain and sets up the required A records for that.

Terraform Init & Plan

Now that we have our variables all setup and should have Terraform installed now, we can initialize our project and run a plan to verify what it will do.

Make sure you are in the dns folder and run the following:

terraform init - This will initialize everything needed for the project to run and install modules.

Once this is complete you can now run the plan command to validate you have all your vars setup and it can generate everything properly before applying it:

terraform plan -var-file="terraform.tfvars" which will give a output of the information that it will deploy, validate this looks right before continuing.

Deploy DNS/Domain Terraform

As long as we didn’t have any issues with the previous Init & Plan step we can now deploy our dns & domain changes..

Ensuring we still have our TF_VAR_token, TF_VAR_soa_email, and TF_VAR_nodebalancer_ip exported on our shell then we can run:

terraform apply -var-file="terraform.tfvars" which should ask for a yes prompt and then will deploy the cluster and will generate your kubeconfig to connect to it.

Now you should see your changes reflected in the Linode UI under domains.

Let us move on to to the cert-manager folder and steps for auto SSL certs.

Cert Manager Setup

This section will go over how to get cert manager setup on your kubernetes cluster to allow for automated SSL certificates from Let’s Encrypt.

More information can be found at Linode TLS Encryption Guide Kubernetes or at Cert Manager Install

Ensure you are in the cert-manager directory for all of these steps.

Note

Be sure before starting any of the below steps you already have your dns nameservers from your custom domain pointing to the linode servers and resolving, this should have been done in the previous dns terraform steps.

Install Cert Manager CRDs

Ensure you have your kube-config file from the previous kubernetes step exported or you can copy to this folder if needed, and then you will want to run the following to install the cert manager CRDs:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.crds.yaml

We should see something like this:

customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created

Install Cert Manager

Now we can add our cert-manager helm repo and update it then install it:

helm repo add cert-manager https://charts.jetstack.io

helm repo update

helm install my-cert-manager cert-manager/cert-manager --namespace cert-manager --version v1.8.0

If successful we should see something like this:

NAME: my-cert-manager
LAST DEPLOYED: Mon Nov 21 06:39:07 2022
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
cert-manager v1.8.0 has been deployed successfully!

In order to begin issuing certificates, you will need to set up a ClusterIssuer
or Issuer resource (for example, by creating a 'letsencrypt-staging' issuer).

More information on the different types of issuers and how to configure them
can be found in our documentation:

https://cert-manager.io/docs/configuration/

For information on how to configure cert-manager to automatically provision
Certificates for Ingress resources, take a look at the `ingress-shim`
documentation:

https://cert-manager.io/docs/usage/ingress/

Now verify you see the corresponding pods coming up and running:

kubectl get pods --namespace cert-manager

You should see something like:

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-579d48dff8-84nw9              1/1     Running   3          1m
cert-manager-cainjector-789955d9b7-jfskr   1/1     Running   3          1m
cert-manager-webhook-64869c4997-hnx6n      1/1     Running   0          1m

Note

Before continuing to the next steps ensure these cert-manager pods are running and ready.

Setting Up ClusterIssuer Resource

Next we will be creating a cluster issuer resource that will be in charge of helping with the automated ssl certs.

This manifest we are about to install will register an account on an ACME server used by Let’s Encrypt for the certificates.

To secure the email we have set this up as a export instead of an actual terraform tfvar. So we need to export our email address we want to use with Let’s Encrypt to issue the certificates automatically.

Once you know what this email should be run the following:

export EMAIL=xxx@xxx.com
envsubst < acme-issuer-prod.yaml | kubectl apply -f -

This will update the email section of that file for you automatically and apply it.

We should now have everything we need setup in order to deploy our test application. In this setup we are using a rasa chatbot atm for a demo example. Please proceed to the next Rasa Demo Setup section to see how this works.

Rasa Demo Setup

Now in order to test everything we need a demo app to deploy, you can use whatever you like but for our setup we are using a Rasa chatbot on a github pages setup.

Everything being performed in this step will be done in the rasa directory.

Deploy Rasa On Kubernetes

So now we want to deploy our previous chatbot model from our last video we made, so in order to do that we setup a helm chart values file to use.

First thing we need to do is add our helm repos:

helm repo add rasa https://helm.rasa.com
helm repo update

Now we can actually install our Rasa chatbot using the helm install with our values file rasa.values.yaml.

There are a few custom things that need to be set in this file however:

hostname - This needs to be set to whatever your a record you setup in dns was with your custom domain.
hosts - The hosts section under secret needs to be set to the same name as the hostname.
initialModel - This should be pointing to a non authenticated location where your rasa model is,
               we are using the model from a previous video we made with a chatbot.

Now that we have set our values we can install this into kubernetes:

helm install -f rasa-values.yaml rasa rasa/rasa

This might take a few mins to come up, but once the pod shows ready you can see the status via going to https://subdomain.yourdomain.com/status

You can check the pod status by running:

kubectl get pods

And you should see these:

NAME                                                READY   STATUS              RESTARTS   AGE
rasa-6fb894b7c-vr85l                                0/1     PodInitializing     0          36s
rasa-postgresql-0                                   0/1     ContainerCreating   0          35s

Once these show running you should be able to hit the resource at the https://subdomain.yourdomain.com/status route.

Changelog

0.8.2 (2022-11-22)

Fix

k8s-namespace: updating namespace creation to be done along with… (#9)

0.8.1 (2022-11-20)

Fix

cert-manager: reverting cert-manager to not use terraform and just use helm commands

0.8.0 (2022-11-20)

Feat

cert-manager: starting to update cert-manager to use terraform (#6)

0.7.0 (2022-11-20)

Feat

ci: triggering version bump

Fix

version-bump: testing out the version bump

0.6.0 (2022-11-20)

Feat

terraform-state: updating terraform remote state (#3)

0.5.0 (2022-11-19)

Feat

docs: adding in docs

Fix

.cz.toml: updating it to not use the .cz.toml for the docs version

0.4.8 (2022-11-19)

Fix

auto-readme: updating the auto readme workflow

0.4.7 (2022-11-19)

Fix

auto-versioning: updating the auto versioning badge readme workflow

0.4.6 (2022-11-19)

Fix

auto-version: updating workflow for auto version readme badge

0.4.5 (2022-11-19)

Fix

version-badge: trying to get the auto version readme working

0.4.4 (2022-11-18)

Fix

github-actions: testing out auto updating readme with version badge

0.4.3 (2022-11-18)

Fix

github-actions: testing a change to fix readme

0.4.2 (2022-11-18)

Fix

github-actions: updating workflow for auto version badge in readme

0.4.1 (2022-11-18)

Fix

github-actions: updating to try and get auto version badge working

0.4.0 (2022-11-18)

Feat

commitizen: testing a version bump

0.3.0 (2022-11-18)

Feat

version-badge: working on ci for auto version badge in readme

Fix

version-badge: trying to get the version badge in readme update working

0.2.0 (2022-11-18)

Feat

auto-version-badge: trying to get the readme push back with badge

0.1.0 (2022-11-18)

Feat

badge-version: working on automated readme modification
version-badge: updating the ci to auto update the version badge in the readme and push it back to the repo

Fix

github-actions: fixing some ci issues

0.0.3 (2022-11-18)

Fix

cleanup: updating the code and some workflows (#1)

0.0.2 (2022-11-18)

Refactor

lke: updating lke outputs to be their own file

0.0.1 (2022-11-18)

Feat

README: updating readme and dns settings
linode-dns: updating to match the stuff in my new configuration
rasa: adding in rasa with ssl setup using cert-manager
dns: adding in dns stuff for linode for ingress dns routing
ingress: adding in ingress

Fix

dns: updating dns vars

Welcome to DevOps With Brian - Linode Terraform’s documentation!

Initial Setup

Commitizen & Conventional Commits

Custom Domain

Terraform

Linode Account

Kubernetes Setup

Linode API Token

Terraform Variables

Sensitive Variables

Other Variables

Terraform Cloud Login For Remote State

Terraform Init & Plan

Deploy LKE Terraform

Connecting To New Kubernetes Cluster

Linode DNS/Domain Setup

Linode API Token

Terraform Variables

Sensitive Variables

Other Variables

Terraform Cloud Login For Remote State

Terraform Init & Plan

Deploy DNS/Domain Terraform

Cert Manager Setup

Install Cert Manager CRDs

Install Cert Manager

Setting Up ClusterIssuer Resource

Rasa Demo Setup

Deploy Rasa On Kubernetes

Changelog

0.8.2 (2022-11-22)

Fix

0.8.1 (2022-11-20)

Fix

0.8.0 (2022-11-20)

Feat

0.7.0 (2022-11-20)

Feat

Fix

0.6.0 (2022-11-20)

Feat

0.5.0 (2022-11-19)

Feat

Fix

0.4.8 (2022-11-19)

Fix

0.4.7 (2022-11-19)

Fix

0.4.6 (2022-11-19)

Fix

0.4.5 (2022-11-19)

Fix

0.4.4 (2022-11-18)

Fix

0.4.3 (2022-11-18)

Fix

0.4.2 (2022-11-18)

Fix

0.4.1 (2022-11-18)

Fix

0.4.0 (2022-11-18)

Feat

0.3.0 (2022-11-18)

Feat

Fix

0.2.0 (2022-11-18)

Feat

0.1.0 (2022-11-18)

Feat

Fix

0.0.3 (2022-11-18)

Fix

0.0.2 (2022-11-18)

Refactor

0.0.1 (2022-11-18)

Feat

Fix

Indices and tables