Categories on Luigi Zhou's Blog on Linux, Kubernetes, Observability and more...

Understanding Kubernetes Resource Quotas and Limits: Best Practices for Optimal Cluster Management

Sun, 30 Jul 2023 16:56:56 +0000

Introduction

Kubernetes has revolutionized container orchestration and brought scalability and flexibility to modern applications. As your Kubernetes cluster grows and accommodates an increasing number of workloads, managing resource allocation becomes critical. In this blog post, we will delve into Kubernetes resource quotas and limits, exploring their significance in cluster management and sharing best practices to optimize resource utilization for a stable and efficient environment.

The Importance of Resource Quotas and Limits

Resource quotas and limits play a vital role in Kubernetes by ensuring a well-functioning and fair cluster. They prevent resource-hungry workloads from affecting others and help maintain overall stability.

Understanding Resource Quotas

Kubernetes resource quotas control the amount of resources that a namespace or user can consume. They are essential for preventing individual applications from using excessive resources and ensure that all applications get their fair share.

Types of resource quotas include:

Compute resource quotas (CPU, memory, etc.).
Object count quotas (number of Pods, Services, etc.).

Configuring resource quotas is a straightforward process and can be done through YAML manifests or Kubernetes API.

Working with Resource Limits

Resource limits set caps on the maximum amount of resources that a container or Pod can consume. By setting resource limits, you prevent applications from monopolizing resources and causing cluster instability.

Types of resource limits include:

CPU limits
Memory limits

Setting resource limits can be done alongside resource requests in Pod specifications.

Best Practices for Resource Allocation

Optimizing resource allocation is crucial for efficient cluster management. Follow these best practices to ensure optimal resource utilization:

Analyze Application Requirements: Understand the resource needs of your applications to set accurate quotas and limits. This prevents overallocation or underallocation of resources.
Resource Requests vs. Limits: Distinguish between resource requests and limits. Resource requests are what a container needs to start running, while limits define the maximum resources it can use.
Leveraging Horizontal Pod Autoscaler (HPA): Utilize HPA to automatically adjust resource limits based on application demands. HPA ensures that your applications have sufficient resources during peak usage.

Monitoring and Troubleshooting Resource Quotas and Limits

Monitoring resource utilization is crucial for identifying potential issues and bottlenecks. Use monitoring tools and techniques to track resource consumption in real-time. In case of problems, troubleshoot common issues related to resource constraints.

Additionally, set up alerts and automate remediation for resource-related problems to maintain cluster health.

Conclusion

Resource quotas and limits are indispensable tools for effective Kubernetes cluster management. By implementing best practices for resource allocation and monitoring, you can optimize resource utilization and ensure a stable and high-performing environment for your applications.

Follow the guidelines shared in this blog post to make the most of Kubernetes resource quotas and limits, ensuring the seamless operation of your containerized workloads.

Happy container orchestration!

CKA Tips for Kubernetes Veterans

Sun, 04 Sep 2022 14:16:09 +0000

I’ve recently taken the CKA certification exam online after working for a few years with Kubernetes and just wanted to share a few tips for the veterans!

Tip 1 - Don’t be scared!

Do not be afraid. If you’ve been working with Kubernetes for a relatively long time, chances are that you’ll be able to breeze through the exam quite easily. I have been in the space for a few years now and I was confidently able to complete the exam with just a couple of days of preparation

Tip 2 - You should still prepare yourself

While the test for a veteran Kubernetes engineer won’t be too hard, you still want to make sure you work through your hands-on skill. The CKA test requires a lot of hands-on skill, so roughly knowing the Pod/Deployment/Statefulset/Daemnset spec will come in handy. I always have a hard time remembering the whole syntax for pod/node affinity!

Tip 3 - Get familiar with navigating the Kubernetes official documentation

This is probably the most crucial tip I can give you. The Kubernetes documentation is very powerful and it’s one of the sources you can freely use during the exam. That is how, even without remembering by heart how to write the pod affinity, I was able to still write it correctly when I was asked to.

Of course, I was very familiar with the concept, so the documentation is just a way to not lose time trying to remember what fields go where.

Tip 4 - kubectl explain

You might not know this, but kubectl has an explain command! This is very handy when you’re not entirely sure what values are accepted in certain fields and it can also be used to go through the tree of fields that are available in a Kubernetes object spec!

And while not inherent to the CKA exam, the kubectl explain command also works with Custom Resources and therefore is very useful when you don’t have the documentation page for that CRD readily available!

Tip 5 - killer.sh

As ominous as that name sounds, killer.sh is a CKA exam simulator that is a MUST to take if you want to increase your chances of passing the exam! It’s so good now that when you buy the CKA exam, 2 tries at the killer.sh simulation exams are included in the price and can be redeemed if you’re using your Linuxfoundation account to log in!

The exam is much more difficult and has more questions, so don’t feel down if you’re not able to complete it in the allocated time. If you can get 50-60% right, then you’re good to go!

Tip 6 - Relax

You’re bound to have an answer or two that you’re not sure about but you shouldn’t worry too much and just skip to the next one! You can always go back and review your answer before you submit them and remember that you can attempt the exam twice!

Tip 7 - Location, Location, Location

It is now much more common to take the exam at home now that we’re in a post-covid society. Be sure to follow all the guidelines as the proctor of the exam will ask you to turn your webcam around to check if there are any irregularities.

Make sure you’re in a quiet room and that no one will disturb you for the next 2-3 hours!

Tip 8 - Put aside enough time for the exam

While the time of the exam is roughly around 2 hours, you want to make sure you’re going to be disturbance-free for at least 3 hours in a room you know you won’t be interrupted. When I took the exam, I ended up starting 30 minutes later due to making sure my local setup for the exam was working fine and doing a tour of my room via my webcam

After the exam?

If you’re not a Kubernetes veteran already, now that you have the CKA you’re ready to work with Kubernetes! Otherwise, as a Kubernetes veteran, there are a couple of paths I would suggest to continue:

Learn to create Kubernetes Controller and Admission/Mutating Webhook. They’ll greatly enhance your knowledge of how Kubernetes works internally. It’s a topic that is not covered during the CKA but I consider it to be a must to learn if you’re managing a Kubernetes platform.
Start looking into the CKS certification. I haven’t done that yet myself, but I’ve been working in a very secure Kubernetes environment and one thing I can tell you for certain, you can go to great lengths in ensuring that Kubernetes is secure and it’s worth looking into it

I hope this article will be useful to you all, if so let me know!

About Me

Sun, 04 Sep 2022 12:49:45 +0000

Hey All, I’m Luigi nice to meet ya! I work as Platform Engineer and here’s a few things about me:

I’m passionate about IT and tecnology, love Kubernetes and write golang/python whenever I have the chance.

Discovered I love cycling so been doing that for a while now (you can find me on Strava, but I’m a very private person eheh).

When I am not working or exploring new tech/hobbies, I enjoy a good movie, videogames and spending time with my family ;)

Helm Charts V3

Wed, 12 Feb 2020 00:00:00 +0000

When I started my journey into Kubernetes back in 2018, I remember people talking about Helm as the easiest way to deploy applications into Kubernetes. I remember the tool was a favourite for developers and I got curious about it and started to look more into it.

It was with a heavy heart that I decided to drop Helm altogether at the time.

The main concerns I had were about:

Security around the Tiller component
A non-purist approach towards Kubernetes declarative manifests

The first reason is pretty self-explanatory since Tiller was also described as one giant sudo server. The second one was around the argument that “cloud-native” application should be able to gracefully handle dependencies that takes more time to be deployed either by implementing a retry logic or other means.

What changed my mind now? Well, the advent of helm 3

Helm 3 - A Tillerless world

To my surprise, Helm 3 was released without the need of having Tiller deployed in Kubernetes.

You can read more about all the changes here and in case you didn’t know, you can also read more in detail about the security concerns around Tiller.

This was incredibly liberating since it was my biggest concern with the tool so far and combined with the fact that I was getting a little bit tired of having to write Kubernetes manifests with very repeatable fields, it made adopting helm all the more compelling.

Helm templating was anyway a great feature you could use if you didn’t want tiller in your cluster and many people were using it that way, but because my work on Kubernetes was more project focused I never had a big need for it until quite recently.

The bad things

Finally getting started with Helm proved to be more difficult than I hoped. Although plenty of charts were out there ready to be consumed, developing one was not as easy as I originally believed.

The problem with documentation

I found it quite hard to navigate the documentation. Although very complete and descriptive about the features available to Helm, I found it hard to read it through since sometimes it feels more like a collection of blog posts.

The way it is also formatted makes it hard to find the information you’re seeking and the side menu that keeps on closing every time you click an element is a little irritating.

A great deal of improvement could be made by just adjusting the website font size and menu, which hopefully is something other people can agree with.

The problem with secrets

Another big issue I found with Helm is the lack of management of the secrets or more specifically generated secrets. There is a whole thread opened in github if you want to read about it, but in essence, at the moment of my writing, you shouldn’t use Helm to define your secrets.

I stumbled upon a nice way of generating random password in a helm way, but I soon found out that successive releases were breaking because a new randomly generated password would be pushed into my Kubernetes secrets and would break functionality.

This is because Helm doesn’t handle secrets. It was very confusing for me since many charts had randomly generated password for defaults and I didn’t notice until I started to play around with it.

For my charts, secrets are at the moment statically defined by the values yaml file. I believe it’s the best way to have a chart user being more aware of what’s going on. Because Helm doesn’t do secret management, the best way I believe to move forward is to define secrets externally and load them at runtime during deployment in a pipeline.

The problem with ready to use charts

The charts that can be found under the stable repository of Helm are very nice, definitely really good to get started quickly, but a bad idea to use “AS IS” for production.

The biggest problem I have with them is the over-generalization, which is a must if you want to share your work with other people since what works for you may not work with everyone else depending on different configurations and other factors, but it’s daunting when you want to adopt it in your company.

It becomes a big vetting process, where you try to understand what the chart is doing and why is it doing it in that way. This becomes even more difficult given the fact that Charts are incredibly hard to read especially with if/else logic and obscure template function you may not know about.

The lack of an out of the box diff feature

This is not really on Helm, as I believe this is not available out of the box in Kubernetes as well, but I believe it could be a cool feature to implement out of the box in the tool. I know there is a plugin that does this and I haven’t tested it yet, but since it is possible I believe it would be nice to have it in Helm.

This opinion comes from a heavy Terraform user, which is an incredible tool that gives you the ability to see what is going to happen before you push your infrastructure as code into production (or any other environment). It let me know what is going to change, which is a great way to double-check if I’m doing something wrong.

If Helm had a feature like that out of the box, I believe it could be the killer feature that would push the whole community to adopt helm as a standard for deployment in Kubernetes.

The good things

Helm is an incredible tool that in my opinion fits quite well with the Kubernetes world.

It enables you to create reusable deployments, package it and share it within the company or the community. I also believe it is the only tool so far that makes this possible without having to write additional scripts.

I previously mentioned how Helm approach was “non-purist”, but I learned over time that in real-world scenario it can be quite difficult to have the time to implement the necessary logic to make the application smarter. Sometimes it’s just difficult to convince management that it is worth the time, and sometimes it’s just not that trivial. That’s why the dependency management of Helm comes out as a winner since the outcome is faster implementation and delivery to production.

Conclusion

Although I wrote more about the bad instead of the good, Helm still comes out on top as the tool to use for deployment in Kubernetes. There is no valid alternative at the moment and so far it’s the best way to properly package and standardise your deployment.

It could be further polished, which is what all of the “bad things” I described are all about, but overall it’s pretty good. This is, of course, coming out of a 2-week experience with Helm so I could change my mind in the future (which I will, of course, write about in this blog) but for what I see, the tool can only improvement so I don’t think I will.