Volunteering allowed me to be at the heart of the conference. I helped with attendee check-ins, session logistics, and general coordination throughout the venue. Being behind the scenes gave me a new perspective on how much work and dedication goes into running an international academic event - and it was exciting to contribute to it.

Between tasks, I had the chance to attend keynotes, speak with participants, and discover ongoing work in cloud-edge computing, distributed systems, and large-scale data processing. Discussions were rich, the atmosphere was collaborative, and I left every day with something new to think about.

Highlights:

• Meeting researchers and professionals from various countries
• Discovering ongoing work in cloud, edge, AI, and big data
• Helping the organization team ensure everything ran smoothly
• Attending inspiring presentations and demos

These four days were both intense and deeply rewarding. The experience strengthened my interest in distributed systems and data technologies, and it motivated me to stay involved in future academic communities and events.

"We grow by learning from others, by contributing where we can, and by stepping into communities larger than ourselves."

A memorable experience, and definitely not the last.

powercap-utils is a suite of user-space command-line tools for Linux that allow you to interact with the kernel's Power Capping Framework. This framework provides a standardized way to monitor and limit the power consumption of hardware devices.

Its most common use is to control Intel's Running Average Power Limit (RAPL) and AMD's RAPL-like features on modern CPUs. These features let you directly set power budgets for the CPU package, its cores, and DRAM.

2. Prerequisites and Installation​

Prerequisites​

1. A Linux System: Any modern Linux distribution.
2. Supported Hardware:
   • Intel Sandy Bridge and newer processors.
   • Modern AMD Zen-based processors.
3. Kernel Support: The intel_rapl (for Intel) or amd_energy (for AMD) kernel module must be loaded. This is typically automatic.

Installation​

The package name is usually powercap-utils.

• On Debian / Ubuntu:

  sudo apt update
  sudo apt install powercap-utils
  

• On Fedora / CentOS / RHEL:

  sudo dnf install powercap-utils
  

• On Arch Linux:

  sudo pacman -S powercap-utils
  

3. Core Concepts of the Power Capping Framework​

The framework is organized as a hierarchy, which you navigate using the tools.

Figure - RAPL domains: package, core, uncore, and DRAM.

• Control Type: The top-level driver. This will be intel-rapl on Intel systems.
• Zone: A device or a part of a device that can be power-managed. Common zones are:
  • package-0 (or 0): The entire CPU socket. This is the most important zone.
  • core (sub-zone): A group of CPU cores.
  • uncore (sub-zone): Components like the L3 cache and memory controller.
  • dram (sub-zone): The DRAM interface.
• Constraint: A specific power limiting rule within a zone.
  • Constraint 0: The long_term limit. This is the sustained power limit, similar to the CPU's TDP (Thermal Design Power).
  • Constraint 1: The short_term limit. This is a higher power limit that the CPU is allowed to use for a short burst of time.
• Parameters: Each constraint has settings you can read and write:
  • power_limit_uw: The power limit in microwatts (µW).
  • time_window_us: The time period in microseconds (µs) over which the power limit is enforced.

⚠️ Critical Note on Units: The tools operate in micro-units. This is the most common source of error.

• To set a limit of 50 Watts, you must use 50000000 microwatts.
• To set a time window of 1 second, you must use 1000000 microseconds.

4. The Tools: powercap-info and powercap-set​

Part 1: Exploring with powercap-info (Read-Only)​

powercap-info is your tool for discovery. It's safe to use as it only reads information.

1. List all power capping control types and their full hierarchy: This is the most common command to get an overview of your system.

powercap-info

Output (example from an Intel CPU):

intel-rapl
└── package-0
    ├── core
    ├── uncore
    └── dram

This shows the control type is intel-rapl and the primary zone is package-0.

2. Get detailed information for a specific zone: Based on the --help output, the correct syntax is powercap-info <control-type-name> -z <zone-id>.

Let's inspect package-0, which corresponds to zone 0.

powercap-info intel-rapl -z 0

Output (abbreviated):

package-0:
  enabled: 1
  energy_uj: 1234567890
  constraints:
    0: long_term
      power_limit_uw: 65000000
      time_window_us: 28000000
      name: long_term
    1: short_term
      power_limit_uw: 85000000
      time_window_us: 2441
      name: short_term

Explanation:

• This package has a long_term limit (constraint 0) of 65,000,000 µW (65 W).
• It has a short_term limit (constraint 1) of 85,000,000 µW (85 W).
• energy_uj is the total energy consumed (in microjoules) since boot.

3. Get a single value: The --help shows many flags to query specific values. For example, to get only the power limit for the long_term constraint:

# Syntax: powercap-info <control-type> -z <zone> -c <constraint> --c-power-limit
powercap-info intel-rapl -z 0 -c 0 -l

Output:

65000000

This is excellent for scripting, as it gives you a clean number to work with.

Part 2: Setting Limits with powercap-set (Write/Modify)​

powercap-set is used to change the power settings. You will need sudo for this.

Syntax: sudo powercap-set <control-type-name> [options]

Key Options:

• -z, --zone=<zone-id>: Specifies the zone to modify (e.g., -z 0).
• -c, --constraint=<constraint-id>: Specifies the constraint (0 for long_term, 1 for short_term).
• -l, --c-power-limit=<uw>: Sets the power limit in microwatts.
• -s, --c-time-window=<us>: Sets the time window in microseconds.
• -e, --z-enabled=1|0: Enables (1) or disables (0) an entire zone.

Example Workflow: Limiting a CPU to 35 Watts

Let's cap the entire CPU package (package-0, which is zone 0) to a sustained 35 Watts.

Step 1: Identify the current limits.

powercap-info intel-rapl -z 0 -c 0

Let's assume the current limit is 65000000 (65W).

Step 2: Set the new power limit. We want to set a 35W limit for the long_term constraint (ID 0). 35 Watts = 35000000 microwatts.

sudo powercap-set intel-rapl -z 0 -c 0 -l 35000000

Step 3: Verify the change.

powercap-info intel-rapl -z 0 -c 0 -l

New Output:

35000000

Success! The CPU's sustained power limit is now 35W.

5. Practical Example: Setting a Custom Boost and Sustained Limit​

Goal: Configure a CPU to run at a sustained 60W but allow it to boost to 90W for up to 15 seconds.

1. Set the long_term (sustained) limit:

• Constraint: 0
• Power: 60W = 60000000 µW

sudo powercap-set intel-rapl -z 0 -c 0 -l 60000000

(The default time window for long_term is usually fine, so we don't need to set it.)

2. Set the short_term (boost) limit:

• Constraint: 1
• Power: 90W = 90000000 µW
• Time Window: 15 seconds = 15000000 µs

sudo powercap-set intel-rapl -z 0 -c 1 -l 90000000 -s 15000000

3. Verify all settings:

powercap-info intel-rapl -z 0

Check the output to confirm that both constraint 0 and constraint 1 now reflect your new values. The CPU will now adhere to this custom power profile.

6. Scripting and Automation​

Because the tools can output clean numbers, they are perfect for scripts.

Example: A script to toggle a "low power" mode (25W) and a "high power" mode (80W).

power-mode.sh

#!/bin/bash

# Check for root privileges
if [[ $EUID -ne 0 ]]; then
   echo "This script must be run as root" 
   exit 1
fi

MODE=$1
CONTROL_TYPE="intel-rapl"
ZONE="0"
CONSTRAINT="0" # We will only change the long_term limit

if [[ "$MODE" == "low" ]]; then
    POWER_LIMIT_UW=25000000 # 25 Watts
    echo "Setting low power mode (25W)..."
elif [[ "$MODE" == "high" ]]; then
    POWER_LIMIT_UW=80000000 # 80 Watts
    echo "Setting high power mode (80W)..."
else
    echo "Usage: sudo $0 <low|high>"
    exit 1
fi

powercap-set "$CONTROL_TYPE" -z "$ZONE" -c "$CONSTRAINT" -l "$POWER_LIMIT_UW"

# Verify
CURRENT_LIMIT=$(powercap-info "$CONTROL_TYPE" -z "$ZONE" -c "$CONSTRAINT" -l)
CURRENT_LIMIT_W=$((CURRENT_LIMIT / 1000000))
echo "Successfully set power limit to ${CURRENT_LIMIT_W}W."

How to use the script:

chmod +x power-mode.sh

# Set low power mode
sudo ./power-mode.sh low

# Set high power mode
sudo ./power-mode.sh high

7. Conclusion​

powercap-utils gives you precise, low-level control over your CPU's power consumption. By understanding the hierarchy of control types, zones, and constraints, you can effectively use these tools to manage thermals, optimize performance, and control energy usage.

Key Takeaways:

• Use powercap-info to explore the available zones and constraints on your system.
• Use sudo powercap-set to apply limits, always remembering that units are in microwatts and microseconds.
• The primary targets are usually zone 0 (package-0), with constraint 0 (long_term) for sustained limits and constraint 1 (short_term) for boost behavior.
• The single-value flags (like -l for powercap-info) are incredibly useful for scripting and verification.

This is a powerful pattern that allows you to extend the Kubernetes API to manage your own applications declaratively.

What We Are Building​

We will define a Website custom resource that looks like this:

# config/samples/website_v1alpha1_website.yaml
apiVersion: website.my.domain/v1alpha1
kind: Website
metadata:
  name: my-first-website
spec:
  # The git repository URL containing the website's static files
  gitRepo: "https://github.com/someuser/my-static-site.git"
  # The number of replicas for the web server
  replicas: 3

Our controller will see this resource and automatically create:

1. A Deployment to run an Nginx server with the specified number of replicas. We'll imagine it serves content from the gitRepo.
2. A Service of type LoadBalancer to expose the Nginx pods to the internet.

Prerequisites​

Before you begin, make sure you have the following tools installed:

1. Go (version 1.19+): https://golang.org/dl/
2. Docker: To build container images.
3. kubectl: The Kubernetes command-line tool.
4. A Kubernetes Cluster: A local one like Minikube or Kind is perfect for development.
5. Kubebuilder: The scaffolding tool.

   # Install Kubebuilder
   os=$(go env GOOS)
   arch=$(go env GOARCH)
   curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/${os}/${arch}"
   sudo mv kubebuilder /usr/local/bin/
   

Step 1: Initialize Your Project​

First, create a new directory for your project and initialize it with Kubebuilder.

# Create and enter the project directory
mkdir website-operator
cd website-operator

# Initialize the project
# --domain is used for the API group of your CRDs
# --repo is your Go module path (e.g., github.com/your-user/website-operator)
kubebuilder init --domain my.domain --repo github.com/your-user/website-operator

This command scaffolds a complete project structure, including a Makefile, a main.go file, and a Dockerfile.

Step 2: Create the API (CRD and Controller)​

Now, let's create the API for our Website resource. This single command will generate the CRD definition, the Go struct for our resource, and a skeleton for our controller.

# Create the API. We'll create the resource and the controller.
kubebuilder create api --group website --version v1alpha1 --kind Website

When prompted, answer y to create both the Resource and the Controller.

This command does several important things:

• api/v1alpha1/website_types.go: This file defines the Go structs that represent your Website resource. This is where you define your Spec (the desired state) and Status (the observed state).
• controllers/website_controller.go: This is the skeleton for your controller's logic.
• config/crd/bases/website.my.domain_websites.yaml: The YAML definition for your CRD.

Defining the Spec and Status​

Open api/v1alpha1/website_types.go. We need to define the fields for our resource.

Find the WebsiteSpec and WebsiteStatus structs and modify them as follows.

// api/v1alpha1/website_types.go

// ... (other imports)

// WebsiteSpec defines the desired state of Website
type WebsiteSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file

	// GitRepo is the git repository URL for the website's content.
	// +kubebuilder:validation:Required
	GitRepo string `json:"gitRepo"`

	// Replicas is the number of desired pods. Defaults to 1.
	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:default=1
	// +optional
	Replicas *int32 `json:"replicas"`
}

// WebsiteStatus defines the observed state of Website
type WebsiteStatus struct {
	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
	// Important: Run "make" to regenerate code after modifying this file

	// DeploymentStatus reflects the status of the created Deployment.
	// +optional
	DeploymentStatus appsv1.DeploymentStatus `json:"deploymentStatus,omitempty"`

	// ServiceStatus reflects the status of the created Service.
	// +optional
	ServiceStatus v1.ServiceStatus `json:"serviceStatus,omitempty"`
}
// ... (rest of the file)

• WebsiteSpec: We added GitRepo (a required string) and Replicas (an optional integer with a minimum value of 1). The //+kubebuilder comments are "markers" that generate validation rules in the CRD.
• WebsiteStatus: We will use this to report the status of the Deployment and Service we create. You'll need to add the appsv1 and v1 imports from k8s.io/api.

After modifying the _types.go file, always run make to regenerate the deepcopy functions and CRD manifests.

make

Step 3: Implement the Controller Logic​

This is the most important part. The controller's job is to read the state of a Website resource and take action to make the real world (the cluster state) match the desired state in the Spec.

Open internal/controllers/website_controller.go. The core logic goes inside the Reconcile function.

The Reconciliation Loop​

The Reconcile function is called every time a Website resource is created, updated, or deleted. Its goal is to reach a steady state. It's like a thermostat: it checks the temperature (current state) and turns the heating on or off (takes action) to reach the desired temperature (desired state).

Let's implement the logic.

1. Get the Website resource: The request contains the name and namespace of the resource that triggered the reconciliation.
2. Check if Deployment exists: If not, create it.
3. If Deployment exists: Ensure its spec (e.g., number of replicas) matches the Website spec. If not, update it.
4. Check if Service exists: If not, create it.
5. Update Status: Report the current status back to the Website resource.

Updating the Controller Code​

Replace the contents of internal/controllers/website_controller.go with the following code. Read the comments carefully to understand each step.

{% raw %}

// internal/controllers/website_controller.go

package controllers

import (
	"context"
	"reflect"

	appsv1 "k8s.ioio/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/types"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/log"

	websitev1alpha1 "github.com/your-user/website-operator/api/v1alpha1"
)

// WebsiteReconciler reconciles a Website object
type WebsiteReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=website.my.domain,resources=websites,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=website.my.domain,resources=websites/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=website.my.domain,resources=websites/finalizers,verbs=update
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete

func (r *WebsiteReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	logger := log.FromContext(ctx)

	// 1. Fetch the Website instance
	website := &websitev1alpha1.Website{}
	err := r.Get(ctx, req.NamespacedName, website)
	if err != nil {
		if errors.IsNotFound(err) {
			logger.Info("Website resource not found. Ignoring since object must be deleted.")
			return ctrl.Result{}, nil
		}
		logger.Error(err, "Failed to get Website")
		return ctrl.Result{}, err
	}

	// 2. Reconcile the Deployment
	deployment := &appsv1.Deployment{}
	err = r.Get(ctx, types.NamespacedName{Name: website.Name, Namespace: website.Namespace}, deployment)

	// If the deployment does not exist, create it. Otherwise, update it.
	if err != nil && errors.IsNotFound(err) {
		// Define a new deployment
		dep := r.deploymentForWebsite(website)
		logger.Info("Creating a new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
		err = r.Create(ctx, dep)
		if err != nil {
			logger.Error(err, "Failed to create new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
			return ctrl.Result{}, err
		}
		// Deployment created successfully - return and requeue
		return ctrl.Result{Requeue: true}, nil
	} else if err != nil {
		logger.Error(err, "Failed to get Deployment")
		return ctrl.Result{}, err
	}

	// Ensure the deployment size is the same as the spec
	size := website.Spec.Replicas
	if *deployment.Spec.Replicas != *size {
		deployment.Spec.Replicas = size
		err = r.Update(ctx, deployment)
		if err != nil {
			logger.Error(err, "Failed to update Deployment", "Deployment.Namespace", deployment.Namespace, "Deployment.Name", deployment.Name)
			return ctrl.Result{}, err
		}
		// Ask to requeue after 1 minute in order to give time for the
		// pods be created
		return ctrl.Result{RequeueAfter: time.Minute}, nil
	}

	// 3. Reconcile the Service
	service := &corev1.Service{}
	err = r.Get(ctx, types.NamespacedName{Name: website.Name, Namespace: website.Namespace}, service)
	if err != nil && errors.IsNotFound(err) {
		// Define a new service
		svc := r.serviceForWebsite(website)
		logger.Info("Creating a new Service", "Service.Namespace", svc.Namespace, "Service.Name", svc.Name)
		err = r.Create(ctx, svc)
		if err != nil {
			logger.Error(err, "Failed to create new Service", "Service.Namespace", svc.Namespace, "Service.Name", svc.Name)
			return ctrl.Result{}, err
		}
		return ctrl.Result{Requeue: true}, nil
	} else if err != nil {
		logger.Error(err, "Failed to get Service")
		return ctrl.Result{}, err
	}
	
	// 4. Update the Website status
	// We'll just reflect the deployment status for simplicity
	if !reflect.DeepEqual(deployment.Status, website.Status.DeploymentStatus) {
		website.Status.DeploymentStatus = deployment.Status
		err := r.Status().Update(ctx, website)
		if err != nil {
			logger.Error(err, "Failed to update Website status")
			return ctrl.Result{}, err
		}
	}

	return ctrl.Result{}, nil
}

// deploymentForWebsite returns a Website Deployment object
func (r *WebsiteReconciler) deploymentForWebsite(w *websitev1alpha1.Website) *appsv1.Deployment {
	labels := map[string]string{"app": w.Name}
	replicas := w.Spec.Replicas

	// For a real-world controller, you would likely use an initContainer
	// to clone the gitRepo into a volume. For this tutorial, we'll use
	// a simple nginx image.
	dep := &appsv1.Deployment{
		ObjectMeta: metav1.ObjectMeta{
			Name:      w.Name,
			Namespace: w.Namespace,
		},
		Spec: appsv1.DeploymentSpec{
			Replicas: replicas,
			Selector: &metav1.LabelSelector{
				MatchLabels: labels,
			},
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{
					Labels: labels,
				},
				Spec: corev1.PodSpec{
					Containers: []corev1.Container{{
						Image: "nginx:1.14.2",
						Name:  "web-server",
						Ports: []corev1.ContainerPort{{
							ContainerPort: 80,
							Name:          "http",
						}},
					}},
				},
			},
		},
	}

	// Set Website instance as the owner and controller
	ctrl.SetControllerReference(w, dep, r.Scheme)
	return dep
}

// serviceForWebsite returns a Website Service object
func (r *WebsiteReconciler) serviceForWebsite(w *websitev1alpha1.Website) *corev1.Service {
	labels := map[string]string{"app": w.Name}

	svc := &corev1.Service{
		ObjectMeta: metav1.ObjectMeta{
			Name:      w.Name,
			Namespace: w.Namespace,
		},
		Spec: corev1.ServiceSpec{
			Selector: labels,
			Ports: []corev1.ServicePort{{
				Protocol:   corev1.ProtocolTCP,
				Port:       80,
				TargetPort: intstr.FromString("http"),
			}},
			Type: corev1.ServiceTypeLoadBalancer,
		},
	}
	// Set Website instance as the owner and controller
	ctrl.SetControllerReference(w, svc, r.Scheme)
	return svc
}

// SetupWithManager sets up the controller with the Manager.
func (r *WebsiteReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&websitev1alpha1.Website{}).
		Owns(&appsv1.Deployment{}).
		Owns(&corev1.Service{}).
		Complete(r)
}

{% endraw %}

Key changes and explanations:

• RBAC Permissions (//+kubebuilder:rbac): We added permissions for the controller to manage Deployments and Services. After adding these, run make manifests to update the RBAC roles in config/rbac/role.yaml.
• deploymentForWebsite & serviceForWebsite: These helper functions create the desired Deployment and Service objects.
• ctrl.SetControllerReference: This is crucial. It sets the Website resource as the "owner" of the Deployment and Service. This means when you delete the Website resource, Kubernetes garbage collection will automatically delete the owned Deployment and Service.
• SetupWithManager: We configured the controller to Own Deployments and Services. This tells the controller-runtime framework to trigger a reconciliation for a Website resource whenever its owned Deployment or Service is changed or deleted.

Step 4: Run and Test the Controller​

Now for the exciting part! Let's run our controller and see it in action.

1. Install the CRD into the Cluster​

The cluster needs to know about the Website resource type.

make install

Verify that the CRD is installed:

kubectl get crd websites.website.my.domain
# NAME                          CREATED AT
# websites.website.my.domain    2023-10-27T10:30:00Z

2. Run the Controller Locally​

Running the controller on your local machine is fantastic for development and debugging. It will use your local kubeconfig to talk to the cluster.

make run

You will see logs from the controller manager starting up. It is now waiting for Website resources to be created.

3. Create a Website Resource​

In a new terminal, apply the sample Website resource. Kubebuilder has already created a sample for you in config/samples/website_v1alpha1_website.yaml. Let's modify it to match our Spec.

# config/samples/website_v1alpha1_website.yaml
apiVersion: website.my.domain/v1alpha1
kind: Website
metadata:
  labels:
    app.kubernetes.io/name: website
    app.kubernetes.io/instance: website-sample
    app.kubernetes.io/part-of: website-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: website-operator
  name: website-sample
spec:
  gitRepo: "https://github.com/my-user/my-repo.git"
  replicas: 2

Apply it:

kubectl apply -f config/samples/website_v1alpha1_website.yaml

4. Observe the Results​

Look at the logs from your make run terminal. You should see the controller creating the Deployment and Service.

Now, check the resources in your cluster:

# Check our custom resource
kubectl get website

# NAME             AGE
# website-sample   15s

# Check the Deployment created by our controller
kubectl get deployment website-sample

# NAME             READY   UP-TO-DATE   AVAILABLE   AGE
# website-sample   2/2     2            2           12s

# Check the Service created by our controller
kubectl get service website-sample

# NAME             TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
# website-sample   LoadBalancer   10.10.10.10    <pending>     80:31234/TCP   10s

Success! Our controller saw the Website resource and created the corresponding Deployment and Service.

Now, try changing the number of replicas in the YAML file to 3 and kubectl apply it again. You will see the controller update the Deployment to match.

5. Cleanup​

Delete the Website resource. Because we set the Owner Reference, the Deployment and Service will be automatically deleted too.

kubectl delete -f config/samples/website_v1alpha1_website.yaml

# Verify they are gone
kubectl get deployment,service

Step 5: Build and Deploy to the Cluster​

Running locally is great for development, but for production, you need to run the controller as a Deployment inside the cluster.

1. Build and Push the Docker Image: (Replace your-dockerhub-username with your actual username)

   make docker-build IMG=your-dockerhub-username/website-operator:v0.0.1
   make docker-push IMG=your-dockerhub-username/website-operator:v0.0.1
   

2. Deploy the Controller to the Cluster: This command will use the manifests in the config directory to create the Deployment, RBAC roles, and everything else needed to run the controller.

   make deploy IMG=your-dockerhub-username/website-operator:v0.0.1
   

Your controller is now running inside the cluster, just like any other application. It will continue to watch for Website resources and reconcile them.

Conclusion and Next Steps​

Congratulations! You have successfully built a fully functional Kubernetes controller. You learned how to:

• Scaffold a project with Kubebuilder.
• Define a Custom Resource Definition (CRD).
• Implement the reconciliation logic in a controller.
• Set owner references for garbage collection.
• Run, test, and deploy your controller.

From here, you can explore more advanced topics:

• Finalizers: To perform cleanup actions before a resource is deleted.
• Webhooks: To validate or mutate your custom resources on creation/update.
• More Complex Status Updates: Provide richer status information about the resources your controller manages.
• Testing: Write unit and integration tests for your controller logic.

What We Will Build:

• A Filter plugin: It will only allow pods that request high-speed networking to be scheduled on nodes that are labeled as network-topology=high-speed.
• A Score plugin: It will give a higher score to nodes with the high-speed label, making them more desirable.
• We will then package this plugin into a new scheduler binary, containerize it, and deploy it to a local Kubernetes cluster.

Part 1: Understanding the Foundations - The Scheduling Framework​

Before writing any code, it's crucial to understand the "why" and "how".

What is the Kubernetes Scheduler?​

The default Kubernetes scheduler (kube-scheduler) is a control plane component that watches for newly created Pods that have no nodeName assigned. For every Pod it discovers, the scheduler becomes responsible for finding the best Node for that Pod to run on. This process involves two main phases:

1. Filtering (Predicates): The scheduler finds the set of feasible Nodes where the Pod can run. For example, if a Pod requests 2 CPUs, all nodes with less than 2 available CPUs are filtered out.
2. Scoring (Priorities): From the list of feasible nodes, the scheduler ranks each node by assigning it a score. It then picks the node with the highest score. For example, it might prefer a node that has fewer running pods.

The Scheduling Framework: Your Entry Point​

The Scheduling Framework is a pluggable architecture for the kube-scheduler that makes it easy to add custom logic without forking the entire Kubernetes codebase. It exposes a set of Extension Points in the scheduling lifecycle where your custom code (a Plugin) can be executed.

Visualizing the Extension Points:

Here is the flow of the scheduling process and where the main extension points fit in.

                  (A Pod is created without a node)
                                  |
                                  v
+-------------------------------------------------------------------------+
|                        SCHEDULING CYCLE (for one Pod)                     |
+-------------------------------------------------------------------------+
                                  |
                                  v
[ QueueSort ] <--- Your plugin can sort pods in the scheduling queue.
                                  |
                                  v
+-------------------------------------------------------------------------+
|                      SCHEDULING ATTEMPT (for one Pod)                     |
+-------------------------------------------------------------------------+
|                                 |                                       |
|                                 v                                       |
| [ PreFilter ] <--- Pre-process info about the Pod before filtering.     |
|                                 |                                       |
|          --------------------- FOR EACH NODE ----------------------     |
|          |                                                        |     |
|          |    [ Filter ] <--- Can this Pod run on this Node? (Yes/No) |   |
|          |          (If No, Node is discarded for this Pod)        |     |
|          |                                                        |     |
|          ----------------------------------------------------------     |
|                                 |                                       |
|                                 v                                       |
| [ PostFilter ] <- If no nodes are viable, your plugin can take action.  |
|                                 |                                       |
|                                 v                                       |
| [ PreScore ] <--- Pre-process info before scoring viable nodes.         |
|                                 |                                       |
|          --------------------- FOR EACH VIABLE NODE ---------------     |
|          |                                                        |     |
|          |    [ Score ] <--- Give this Node a score (e.g., 0-100).  |   |
|          |                                                        |     |
|          ----------------------------------------------------------     |
|                                 |                                       |
|                                 v                                       |
| [ NormalizeScore ] <- Modify all scores to fit a common scale.          |
|                                 |                                       |
|                                 v                                       |
|                          (Scheduler picks Node with highest score)        |
|                                 |                                       |
|                                 v                                       |
+-------------------------------------------------------------------------+
|                           BINDING CYCLE                                 |
+-------------------------------------------------------------------------+
|                                 |                                       |
|                                 v                                       |
| [ Reserve ] <--- Mark resources as "reserved" on the chosen node.       |
|                                 |                                       |
|                                 v                                       |
| [ Permit ] <--- Approve or deny the binding. Can delay binding.         |
|                                 |                                       |
|                                 v                                       |
| [ PreBind ] <--- Actions to take right before the Pod is bound.         |
|                                 |                                       |
|                                 v                                       |
| [ Bind ] <--- The actual binding of the Pod to the Node.                |
|                                 |                                       |
|                                 v                                       |
| [ PostBind ] <--- Actions to take after the binding is successful.      |
|                                 |                                       |
+-------------------------------------------------------------------------+

Our plugin will focus on the Filter and Score extension points.

Part 2: Setting Up the Professional Development Environment​

This is where we align with the project's standards.

Prerequisites:​

1. Go: Version 1.19+ (the scheduler-plugins project specifies its version in the go.mod file).
2. Docker: To build container images. Make sure the Docker daemon is running.
3. kubectl: The Kubernetes command-line tool.
4. kind: Our local Kubernetes cluster.

Step 1: Clone the scheduler-plugins Repository​

We will use the official repository not just as a template, but as our direct development environment. This is the standard practice.

git clone https://github.com/kubernetes-sigs/scheduler-plugins.git
cd scheduler-plugins

Step 2: Explore the Makefile​

The Makefile in this repository is the heart of the development workflow. Instead of running raw go or docker commands, we will use make targets. This ensures consistency and correctness.

Key targets we will use:

• make verify: Runs formatters and linters to ensure code quality. Always run this before committing.
• make unit-test: Runs all unit tests in the repository.
• make local-image: This is crucial. It builds a scheduler container image for your local architecture, gives it a standard local name (localhost:5000/...), and loads it directly into your local Docker daemon. Perfect for testing with kind.
• make build: Compiles the Go binaries (kube-scheduler and controller) into the /bin directory.

Part 3: Developing Our Custom Plugin​

The core logic remains the same, but the process around it will be more rigorous.

Step 1: Create the Plugin Directory and File​

mkdir -p pkg/networkspeed
touch pkg/networkspeed/networkspeed.go

Step 2: Write the Plugin Code​

Place the same Go code from the previous guide into pkg/networkspeed/networkspeed.go. This code implements the Filter and Score extension points.

// pkg/networkspeed/networkspeed.go

package networkspeed

import (
	"context"
	"fmt"

	v1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/klog/v2"
	"k8s.io/kubernetes/pkg/scheduler/framework"
)

// Define plugin name
const Name = "Networkspeed"

const (
	// Annotation key on a Pod to specify network preference.
	PodNetworkAnnotation = "my-network-preference"
	// Label key on a Node to specify the type of network it has.
	NodeNetworkLabel = "network-topology"
	// Value for high-speed network preference.
	HighSpeedNetwork = "high-speed"
)

// Networkspeed is a plugin that filters and scores nodes based on network labels.
type Networkspeed struct {
	handle framework.Handle
}

// Ensure our plugin implements the necessary interfaces.
var _ framework.FilterPlugin = &Networkspeed{}
var _ framework.ScorePlugin = &Networkspeed{}

// Name returns the name of the plugin.
func (na *Networkspeed) Name() string {
	return Name
}

// New is the factory function that creates a new instance of the plugin.
// This is called by the scheduler framework when it initializes.
func  New(_ context.Context, obj runtime.Object, handle framework.Handle) (framework.Plugin, error) {
	klog.V(3).Infof("Creating new Networkspeed plugin")
	return &Networkspeed{
		handle: handle,
	}, nil
}

// ----------------- FILTERING LOGIC -----------------

// Filter is called by the framework for each node to see if the pod can be scheduled on it.
func (na *Networkspeed) Filter(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeInfo *framework.NodeInfo) *framework.Status {
	klog.V(3).Infof("Filtering pod %v for node %v", pod.Name, nodeInfo.Node().Name)

	// 1. Get the pod's network preference from its annotations.
	podNetworkPreference, ok := pod.Annotations[PodNetworkAnnotation]
	if !ok {
		// If the pod doesn't have the annotation, it has no special network needs.
		// So, it can run on any node. We allow it.
		return framework.NewStatus(framework.Success, "")
	}

	// 2. If the pod requests a high-speed network...
	if podNetworkPreference == HighSpeedNetwork {
		// ...check if the node has the required label.
		nodeLabels := nodeInfo.Node().GetLabels()
		nodeNetworkType, labelExists := nodeLabels[NodeNetworkLabel]

		if labelExists && nodeNetworkType == HighSpeedNetwork {
			// The node has the required label. The pod can run here.
			klog.V(3).Infof("Node %v has the required '%s' label. Allowing pod %v.", nodeInfo.Node().Name, HighSpeedNetwork, pod.Name)
			return framework.NewStatus(framework.Success, "")
		} else {
			// The node does not have the label. The pod cannot run here.
			klog.V(3).Infof("Node %v does not have the required '%s' label. Filtering out.", nodeInfo.Node().Name, HighSpeedNetwork)
			return framework.NewStatus(framework.Unschedulable, fmt.Sprintf("Node %s lacks required network label", nodeInfo.Node().Name))
		}
	}

	// For any other annotation value, we don't apply any special filtering.
	return framework.NewStatus(framework.Success, "")
}


// ----------------- SCORING LOGIC -----------------

// Score is called for each node that passed the Filter phase.
// It returns a score for the node, with higher scores being better.
func (na *Networkspeed) Score(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeName string) (int64, *framework.Status) {
	klog.V(3).Infof("Scoring node %v for pod %v", nodeName, pod.Name)

	nodeInfo, err := na.handle.SnapshotSharedLister().NodeInfos().Get(nodeName)
	if err != nil {
		return 0, framework.AsStatus(fmt.Errorf("getting node %s from snapshot: %w", nodeName, err))
	}

	nodeLabels := nodeInfo.Node().GetLabels()
	podNetworkPreference, ok := pod.Annotations[PodNetworkAnnotation]

	// If the pod wants a high-speed network and the node has it, give it the highest score.
	if ok && podNetworkPreference == HighSpeedNetwork {
		if nodeNetworkType, labelExists := nodeLabels[NodeNetworkLabel]; labelExists && nodeNetworkType == HighSpeedNetwork {
			klog.V(3).Infof("Node %v matches high-speed preference. Giving max score (100).", nodeName)
			return 100, framework.NewStatus(framework.Success, "")
		}
	}

	// For all other cases, give a minimal score. The scheduler will still use other plugins
	// (like least-allocated) to make a final decision.
	klog.V(3).Infof("Node %v does not match high-speed preference. Giving min score (10).", nodeName)
	return 10, framework.NewStatus(framework.Success, "")
}

// ScoreExtensions returns a ScoreExtensions interface if the plugin implements one.
func (na *Networkspeed) ScoreExtensions() framework.ScoreExtensions {
	// We don't need to normalize scores, so we return nil.
	return nil
}

Step 3: Register the Plugin​

This step is also the same. We must tell the main binary about our new plugin.

Edit cmd/scheduler/main.go and add our plugin to the registry.

// cmd/scheduler/main.go

import (
    // ... other imports
	"k8s.io/component-base/cli"
	"k8s.io/kubernetes/cmd/kube-scheduler/app"

    // IMPORT YOUR PLUGIN'S PACKAGE
	"sigs.k8s.io/scheduler-plugins/pkg/networkspeed"
    
	// ... other plugin imports
)

func main() {
	// Register all plugins.
	command := app.NewSchedulerCommand(
		app.WithPlugin(networkspeed.Name, networkspeed.New), // <-- ADD THIS LINE
		app.WithPlugin(coscheduling.Name, coscheduling.New),
		// ... other plugins
	)
    // ...
}

Part 4: Verifying, Building, and Packaging​

This section is completely revamped to use the official Makefile.

Step 1: Verify Your Code Quality​

Before you even think about building, run the verification scripts. This catches formatting errors, dependency issues, and other common problems.

make verify

If this command fails, fix the issues it reports (e.g., run go fmt ./...) and try again. This is a standard prerequisite for contributing to open-source projects.

Step 2: Build the Local Container Image​

This is the most significant improvement. Forget manually setting registry and image names. The local-image target is designed specifically for this workflow.

make local-image

What does this command do?

1. It compiles your Go code, including the new networkspeed plugin, into a kube-scheduler binary.
2. It builds a container image using Docker.
3. It automatically sets the platform to your local machine's architecture (e.g., linux/amd64 or linux/arm64).
4. It tags the image with a convenient name for local development: localhost:5000/scheduler-plugins/kube-scheduler:v0.0.0.
5. It uses the --load flag with docker buildx to ensure the final image is available in your local Docker daemon, ready for kind to use.

After it finishes, you can confirm the image exists:

docker images | grep scheduler-plugins

You should see localhost:5000/scheduler-plugins/kube-scheduler.

Part 5 : Production-Grade Deployment on VMs​

We will provision two Virtual Machines (one control-plane, one worker) and bootstrap a Kubernetes cluster using the kubeadm tool.

Prerequisites for this Section:​

• VirtualBox and Vagrant: For easily provisioning and managing local VMs.
• Sufficient System Resources: At least 4 CPUs and 8GB of RAM available for the VMs.

Step 1: Provision the Virtual Machines​

We'll use a Vagrantfile to define our two-node cluster. Create a file named Vagrantfile with the following content:

# Vagrantfile
Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/jammy64"
  config.vm.provider "virtualbox" do |v|
    v.memory = 4096
    v.cpus = 2
  end

  config.vm.define "k8s-control-plane" do |control|
    control.vm.hostname = "k8s-control-plane"
    control.vm.network "private_network", ip: "192.168.56.10"
  end

  config.vm.define "k8s-worker-01" do |worker|
    worker.vm.hostname = "k8s-worker-01"
    worker.vm.network "private_network", ip: "192.168.56.11"
  end
end

Now, bring up the machines:

vagrant up

Step 2: Install Container Runtime and Kubernetes Tools on Both VMs​

Execute these commands on both the k8s-control-plane and k8s-worker-01 VMs. You can connect using vagrant ssh k8s-control-plane and vagrant ssh k8s-worker-01.

# SSH into each VM and run the following:

# 1. Install prerequisites and containerd
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y containerd.io

# 2. Configure containerd and enable required kernel modules
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system
sudo systemctl restart containerd

# 3. Disable swap
sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

# 4. Install kubeadm, kubelet, and kubectl
sudo apt-get update
curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Step 3: Initialize the Control-Plane Node​

Run this command only on the k8s-control-plane VM:

# On k8s-control-plane
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.56.10

This command will take a few minutes. At the end, it will output two important things:

1. Commands to set up kubectl for the vagrant user.
2. A kubeadm join command with a token. Copy this join command and save it.

Run the kubectl setup commands on the control-plane node:

# On k8s-control-plane
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Step 4: Install a CNI Network Plugin​

The cluster nodes will not be Ready until a CNI is installed. From the control-plane node, install Calico:

# On k8s-control-plane
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml

Step 5: Join the Worker Node​

Now, SSH into the k8s-worker-01 VM and run the kubeadm join command that you saved from Step 3. It will look something like this (your token and hash will be different):

# On k8s-worker-01
sudo kubeadm join 192.168.56.10:6443 --token abcdef.1234567890abcdef \
	--discovery-token-ca-cert-hash sha256:1234...cdef

After a minute, verify that both nodes are Ready from the control-plane node:

# On k8s-control-plane
kubectl get nodes

Step 6: Provision the Static Configuration on the Control-Plane​

This is the critical step. We must place our custom scheduler's configuration file at the exact path /etc/kubernetes/config/scheduler-config.yaml on the k8s-control-plane VM.

# Run this from your local machine (not inside a VM)
vagrant ssh k8s-control-plane -- -t '
  sudo mkdir -p /etc/kubernetes/config && \
  sudo bash -c "cat > /etc/kubernetes/config/scheduler-config.yaml" <<EOF
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
leaderElection:
  leaderElect: false
clientConnection:
  kubeconfig: "/etc/kubernetes/scheduler.conf"
profiles:
  - schedulerName: my-custom-scheduler
    plugins:
      multiPoint:
        enabled:
          - name: "Networkspeed"
EOF'

# Verify the file was created correctly
vagrant ssh k8s-control-plane -- "sudo cat /etc/kubernetes/config/scheduler-config.yaml"

Step 7: Load the Custom Scheduler Image onto the Cluster Nodes​

The container image we built exists only on our local machine. We need to transfer it to our VMs.

# 1. On your local machine, save the Docker image to a tar file
docker save localhost:5000/scheduler-plugins/kube-scheduler:v0.0.0 > scheduler.tar

# 2. Copy the tar file to both VMs
vagrant scp scheduler.tar k8s-control-plane:/home/vagrant/scheduler.tar
vagrant scp scheduler.tar k8s-worker-01:/home/vagrant/scheduler.tar

# 3. SSH into EACH VM and load the image into containerd
# On k8s-control-plane:
vagrant ssh k8s-control-plane -- "sudo ctr -n=k8s.io images import /home/vagrant/scheduler.tar"
# On k8s-worker-01:
vagrant ssh k8s-worker-01 -- "sudo ctr -n=k8s.io images import /home/vagrant/scheduler.tar"

Step 8: Deploy the Custom Scheduler​

Create the final deploy-kubeadm.yaml manifest on your local machine. This file is identical to the previous "correct" version, as it's designed to run in a real cluster environment.

deploy-kubeadm.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-custom-scheduler
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-custom-scheduler-as-kube-scheduler
subjects:
- kind: ServiceAccount
  name: my-custom-scheduler
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: system:kube-scheduler
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-custom-scheduler
  namespace: kube-system
  labels:
    app: my-custom-scheduler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-custom-scheduler
  template:
    metadata:
      labels:
        app: my-custom-scheduler
    spec:
      serviceAccountName: my-custom-scheduler
      nodeSelector:
        node-role.kubernetes.io/control-plane: "true"
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/control-plane
      containers:
        - name: my-custom-scheduler-ctr
          image: localhost:5000/scheduler-plugins/kube-scheduler:v0.0.0
          imagePullPolicy: IfNotPresent
          args:
            - /bin/kube-scheduler
            - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
            - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
            - --config=/etc/kubernetes/config/scheduler-config.yaml
            - -v=3
          volumeMounts:
          - mountPath: /etc/kubernetes
            name: etckubernetes
            readOnly: true
      volumes:
      - name: etckubernetes
        hostPath:
          path: /etc/kubernetes
          type: Directory

Now, apply this manifest to your new kubeadm cluster. You can copy this file to the control-plane node or configure kubectl on your host to point to the new cluster. The simplest is to run it from the control-plane:

# Copy the YAML to the control plane
vagrant scp deploy-kubeadm.yaml k8s-control-plane:/home/vagrant/

# Apply it from within the control plane VM
vagrant ssh k8s-control-plane -- "kubectl apply -f deploy-kubeadm.yaml"

Verify the scheduler pod is running on the control-plane node:

vagrant ssh k8s-control-plane -- "kubectl get pods -n kube-system -l app=my-custom-scheduler -o wide"

Part 6: Testing the Custom Scheduler​

The testing process is identical, but you will run the commands from your k8s-control-plane VM where kubectl is configured.

1. Label the worker node:

       # On k8s-control-plane VM
       kubectl label node k8s-worker-01 network-topology=high-speed
   

2. Create and apply test-pod.yaml:

       # test-pod.yaml
       apiVersion: v1
       kind: Pod
       metadata:
       name: high-speed-pod
       annotations:
           my-network-preference: "high-speed"
       spec:
       schedulerName: my-custom-scheduler
       containers:
       - name: nginx
           image: nginx
   

   # On k8s-control-plane VM, apply the pod
   kubectl apply -f test-pod.yaml
   

3. Verify the pod placement:

   # On k8s-control-plane VM
   kubectl get pod high-speed-pod -o wide
   

   The pod should be running on the k8s-worker-01 node.

You have now successfully built a custom scheduler and deployed it to a realistic, multi-node kubeadm cluster.

Your Professional Next Steps:

1. Write Unit Tests: Create a pkg/networkspeed/networkspeed_test.go file. The repository has many examples. Once written, you can run them with make unit-test.
2. Run Integration Tests: The project has a framework for integration tests that spin up real API servers. You can add one for your plugin and run it via make integration-test.
3. Pushing to a Real Registry: When you are ready to share your scheduler, you can use the push-images target. This requires setting environment variables:

   # Example for pushing to Docker Hub
   export REGISTRY=docker.io/your-username
   export RELEASE_VERSION=v0.1.0
   make push-images 
   

   This will build and push a properly tagged image (e.g., docker.io/your-username/kube-scheduler:v0.1.0) to a remote registry.
4. Contribute Back: If you've built a generally useful plugin, you can contribute it back to the scheduler-plugins project! Following this workflow (make verify, adding tests) is the first step to a successful Pull Request.

Before setting up a Kubernetes cluster using Kubeadm and CRI-O, ensure that you meet the following requirements:

Hardware Requirements​

• At least 2 GB of RAM per node (4 GB recommended for the master node)
• At least 2 CPUs per node
• 20 GB of free disk space
• A stable internet connection
• All nodes should be able to communicate with each other

Software Requirements​

• A supported Linux distribution (Ubuntu 20.04+ or Debian 10+ recommended)
• User access with sudo privileges
• Basic understanding of Linux command-line interface (CLI)
• Knowledge of networking concepts

Step 1: Configure Kernel Parameters​

Kubernetes requires some kernel parameters to be set to allow IPtables to see bridged traffic.

Load Required Kernel Modules​

Run the following commands on all nodes:

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

Set Kernel Parameters for Kubernetes​

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

Apply the changes:

sudo sysctl --system

Step 2: Disable Swap​

Kubernetes does not work well with swap enabled. Disable it using:

swapoff -a

To make this change persistent, remove any swap entries from /etc/fstab:

sudo nano /etc/fstab

Step 3: Install CRI-O (Container Runtime Interface - Open)​

Add CRI-O Repository and Install​

sudo apt-get update -y
sudo apt-get install -y software-properties-common gpg curl apt-transport-https ca-certificates

curl -fsSL https://pkgs.k8s.io/addons:/cri-o:/prerelease:/main/deb/Release.key | \
    gpg --dearmor | sudo tee /etc/apt/keyrings/cri-o-apt-keyring.gpg >/dev/null
echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://pkgs.k8s.io/addons:/cri-o:/prerelease:/main/deb/ /" | \
    sudo tee /etc/apt/sources.list.d/cri-o.list

sudo apt-get update -y
sudo apt-get install -y cri-o

Enable and Start CRI-O Service​

sudo systemctl daemon-reload
sudo systemctl enable crio --now
sudo systemctl start crio.service

Step 4: Install Kubernetes Components​

Add Kubernetes Repository​

KUBERNETES_VERSION=1.30
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v$KUBERNETES_VERSION/deb/Release.key | \
    gpg --dearmor | sudo tee /etc/apt/keyrings/kubernetes-apt-keyring.gpg > /dev/null

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v$KUBERNETES_VERSION/deb/ /" | \
    sudo tee /etc/apt/sources.list.d/kubernetes.list

Install Kubeadm, Kubelet, and Kubectl​

sudo apt-get update -y
sudo apt-get install -y kubelet=1.30.0-1.1 kubectl=1.30.0-1.1 kubeadm=1.30.0-1.1

Configure Kubelet Node IP​

sudo apt-get install -y jq
local_ip="$(ip --json addr show eno1 | jq -r '.[0].addr_info[] | select(.family == "inet") | .local')"

sudo tee /etc/default/kubelet > /dev/null << EOF
KUBELET_EXTRA_ARGS=--node-ip=$local_ip
EOF

Note: eno1 and enp85s0 are network interface names that may vary depending on the system. You can check your network interfaces using:

ifconfig -a

or

ip a

Adjust the interface name accordingly based on the output of these commands.

Step 5: Initialize Kubernetes on the Master Node​

Set the master node IP and hostname:

IPADDR="$(ip --json addr show enp85s0 | jq -r '.[0].addr_info[] | select(.family == "inet") | .local')"
NODENAME=$(hostname -s)
POD_CIDR="192.168.0.0/16"

Initialize Kubernetes:

sudo kubeadm init --apiserver-advertise-address=$IPADDR --apiserver-cert-extra-sans=$IPADDR --pod-network-cidr=$POD_CIDR --node-name $NODENAME --ignore-preflight-errors Swap --cri-socket unix:///var/run/crio/crio.sock 

Set up kubeconfig for the root user:

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

Step 6: Deploy Network and Monitoring Components​

Install Calico Network Plugin​

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

Install Metrics Server​

kubectl apply -f https://raw.githubusercontent.com/techiescamp/kubeadm-scripts/main/manifests/metrics-server.yaml

Install Ingress Controller​

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.49.0/deploy/static/provider/baremetal/deploy.yaml

Step 7: Join Worker Nodes to the Cluster​

Generate a join command on the master node:

kubeadm token create --print-join-command

Run the output command on worker nodes to join them to the cluster.

Step 8: Verify Cluster Status​

Check Nodes​

kubectl get nodes

Check All Pods in a Specific Node​

kubectl get pods --all-namespaces --field-selector spec.nodeName=<NODE_NAME>

Troubleshooting Tips​

• Always check system logs: journalctl -u kubelet
• Verify node status: kubectl get nodes
• Check pod status: kubectl get pods --all-namespaces

Security Considerations​

• Regularly update Kubernetes and CRI-O
• Implement RBAC (Role-Based Access Control)
• Use network policies
• Regularly scan containers for vulnerabilities

Note: Always refer to the official Kubernetes and CRI-O documentation for the most up-to-date information and best practices.

Prerequisites​

• Three Linux machines (or VMs) with access to each other.
• Basic understanding of Linux commands and Kubernetes concepts.

Step 1: Installing K3s​

Install K3s on the Master Node​

1. SSH into the Master Node:

   ssh user@master-node-ip
   

2. Install K3s:

   curl -sfL https://get.k3s.io |  INSTALL_K3S_VERSION=v1.27.12+k3s1 sh -s - --disable traefik --write-kubeconfig-mode 644 --node-name <NODE_NAME>
   

The command installs K3s version v1.27.12+k3s1 on a node named <NODE_NAME>, disables the default Traefik ingress controller, and sets the permissions for the kubeconfig file to 644.

3. Verify Installation:

   kubectl get nodes
   

   You should see the master node listed.

Install K3s on the Worker Nodes​

1. SSH into each Worker Node:

   ssh user@worker-node-ip
   

2. Get the token from the master node

   cat /var/lib/rancher/k3s/server/node-token
   

3. Install K3s on Worker Nodes:

   On each worker node, run:

   curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.27.12+k3s1  K3S_NODE_NAME=YOUR_NODE_NAME K3S_URL=https://master-node-ip:6443 K3S_TOKEN=YOUR_NODE_TOKEN sh -
   

   You can find the YOUR_NODE_TOKEN value on the master node at /var/lib/rancher/k3s/server/node-token.

3. Verify Worker Nodes Join the Cluster:

   On the master node, run:

   kubectl get nodes
   

   You should see all nodes (master and workers) listed.

Step 2: Verifying Node Status​

To check the status of the nodes:

kubectl get nodes

You should see the master and worker nodes with the status Ready.

Step 3: Creating a Pod​

Create a simple pod to ensure everything is working correctly.

1. Create a YAML file named hello-world-pod.yaml:

   apiVersion: v1
   kind: Pod
   metadata:
   name: hello-world
   spec:
   containers:
       - name: hello
       image: busybox  # Replace with your desired Docker image
   
   

2. Apply the YAML file to create the pod:

   kubectl apply -f hello-world-pod.yaml
   

3. Verify the Pod is Running:

   kubectl get pods
   

4. Describe the Pod:

   kubectl describe pod hello-world
   

Step 4: Deleting the Pod​

To delete the pod:

kubectl delete pod hello-world

Step 5: Deploying the Application​

Creating a namespace​

A namespace in Kubernetes is used to organize and manage resources within a cluster, providing isolation and a way to divide cluster resources among multiple users or teams.

The commands used in this process are:

1. View Existing Namespaces:

   kubectl get namespaces
   

   or

   kubectl get ns
   

2. Create a New Namespace:

   kubectl create namespace demo
   

3. Verify the Creation of the Namespace:

   kubectl get namespaces
   

This process ensures you have successfully created the demo namespace and confirmed its presence in your Kubernetes cluster.

Deployment YAML File​

A deployment in Kubernetes is a controller that manages the creation, scaling, and updating of a set of replicated Pods, ensuring the desired state of an application is maintained.

Create a file named hello-world-deployment.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
  namespace: demo
spec:
  replicas: 5
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
        - name: hello-world
          image: bashofmann/rancher-demo:1.0.0
          imagePullPolicy: Always
          resources:
            requests:
              memory: "12Mi"
              cpu: "2m"
          ports:
            - containerPort: 8080
              name: web
              protocol: TCP
          env:
            - name: COW_COLOR
              value: purple
          readinessProbe:
            httpGet:
              path: /
              port: web
          livenessProbe:
            httpGet:
              path: /
              port: web

Explanation of the YAML File​

• apiVersion: Specifies the version of the Kubernetes API to use.
• kind: Specifies the type of resource being created (Deployment).
• metadata: Contains data that helps uniquely identify the object, including a name and namespace.
• spec: Defines the desired state of the Deployment.
  • replicas: Number of pod replicas to run.
  • selector: Defines how the Deployment finds which Pods to manage.
  • template: Defines the Pods that will be created.
    • metadata: Labels for the Pods.
    • spec: Specification of the containers within the Pods.
      • containers: List of containers in the Pod.
        • name: Name of the container.
        • image: Docker image to use for the container.
        • imagePullPolicy: Policy to use for image pulls.
        • ports: Ports to expose from the container.
        • env: Environment variables for the container.
        • readinessProbe: Defines a probe to check if the container is ready.
        • livenessProbe: Defines a probe to check if the container is alive.

This is a Go demo application used for demonstrating Kubernetes and load balancing.

It will create a colored icon for each replica within a ReplicaSet and indicate which one most recently served a response. Replicas which haven't been heard from will begin to fade out, until after a configurable number of seconds they will disappear.

For more information on using the container image, please click here.

Apply the Deployment​

1. Apply the YAML file to create the Deployment:

   kubectl apply -f hello-world-deployment.yaml
   

2. Verify the Deployment and Pods:

   kubectl get deployments -n demo
   kubectl get pods -n demo
   

3. Check the ReplicaSets:

   kubectl get replicaset -n demo
   

4. For more information on replicas:

   kubectl get pods -n demo -o wide
   

   -o wide: Provides additional details about each pod, such as the node it's running on, the IP address, the container images, and more.

5. See the pods metrics

   kubectl top pods -n demo
   

   For nodes

   kubectl top nodes
   

Step 6: Exposing the Application​

Create a service to expose the Deployment.

A service in Kubernetes provides a stable network endpoint to access a set of pods, ensuring consistent access despite pod restarts or IP changes.

Types of Services:​

• ClusterIP: Exposes the service on an internal IP within the cluster, making it accessible only within the cluster.
• NodePort: Exposes the service on each node's IP at a static port, making it accessible from outside the cluster using <NodeIP>:<NodePort>.
• LoadBalancer: Exposes the service externally using a cloud provider's load balancer.
• ExternalName: Maps the service to the contents of the externalName field (e.g., foo.bar.example.com), creating a DNS alias.

1. Create a ClusterIP Service:

   kubectl expose deployment hello-world --type=ClusterIP --name=hello-world-service --port=8080 --target-port=8080 -n demo
   

   • expose deployment hello-world: Creates a service to expose the deployment named hello-world.
   • --type=ClusterIP: Specifies the type of service to create. ClusterIP means the service will be accessible only within the cluster.
   • --name=hello-world-service: Assigns the name hello-world-service to the newly created service.
   • --port=8080: Defines the port on which the service will be exposed.
   • --target-port=8080: Specifies the port on the pod that the service should forward traffic to.
   • -n demo: Limits the scope of the command to the demo namespace.

2. Verify the Service:

   kubectl get svc -n demo
   

Step 7: Accessing the Application​

To access the application from your local machine, use kubectl port-forward:

kubectl port-forward svc/hello-world-service 8080:8080 -n demo --address node_address

Open your browser and go to http://node_address:8080 to see the application in action.

1. Create a service of type LoadBalancer

kubectl expose deployment hello-world --type=LoadBalancer --name=hello-world-service-loadbalancing --port=8080 --target-port=8080 -n demo

Open your browser and go to http://any_node_address:PORT to see the application in action. PORT here is generated by default, see the image below.

Application on the browser

Step 8: Inspecting Resources​

Describing a Pod​

To get detailed information about a specific pod:

kubectl describe pod <pod-name> -n demo

Describing a Deployment​

To get detailed information about the deployment:

kubectl describe deployment hello-world -n demo

Viewing Pod Logs​

To view the logs of a pod:

kubectl logs -f <pod-name> -n demo

Step 9: Cleaning Up​