Building More Resilient Infrastructure with Kubernetes and AWS

Over the past several months, I have continued expanding my hands-on experience with Kubernetes and AWS. My recent work has focused on building reliable application platforms, improving deployment automation, managing cloud resources, and strengthening identity and access controls.

The most valuable part of this work has been connecting the different pieces into complete systems rather than treating Kubernetes, AWS, storage, networking, monitoring, and IAM as isolated technologies.

Expanding My Kubernetes Environment

My Kubernetes work has centered around a multi-node K3s cluster that hosts a growing collection of applications and infrastructure services.

I have continued developing a GitOps deployment model using Forgejo and Argo CD. Application and infrastructure configurations are stored in Git, giving me a version-controlled source of truth for the cluster. Argo CD monitors those repositories and automatically applies changes while also detecting and correcting configuration drift.

I created an Argo CD ApplicationSet that generates applications from production overlays. This makes it easier to add and manage services without manually creating a separate Argo CD application for every workload. Automated synchronization, pruning, and self-healing help ensure that the cluster continues to match the configuration stored in Git.

Applications currently managed through this environment include but are not limited to:

• Forgejo
• Grafana and Prometheus monitoring components
• Paperless-ngx
• n8n
• Uptime Kuma
• Ghost
• MariaDB
• Self Made Rabbit Breeding Tracker
• Self Made Breeding Tracker for my side business breeding and keeping exotic animals.
• Glance
• Trivy Operator

I have also been working with Kubernetes scheduling and node management. This includes configuring node selectors for workloads that require specific hardware or network access, deciding when additional systems should join as worker-only nodes, and safely removing unavailable nodes from both Kubernetes and Longhorn.

Managing Stateful Kubernetes Workloads

Persistent storage has been an important part of the cluster’s development.

I use Longhorn as the default replicated storage platform and a Synology NAS through the NFS external provisioner for workloads that benefit from shared network storage. Working with both systems has required troubleshooting storage classes, persistent volume claims, failed replicas, unavailable nodes, mount behavior, and application-specific storage requirements.

Several applications rely on persistent data, including Paperless-ngx, MariaDB, Forgejo, Ghost, n8n, and Uptime Kuma. Managing these applications has given me practical experience with the additional planning required for stateful Kubernetes workloads.

Deploying an application is only one part of the process. Its storage must remain available when a pod is recreated, moved to another node, upgraded, or recovered after a failure.

Improving Kubernetes Networking and Service Delivery

Networking has been another major area of focus.

Using Traefik for ingress traffic, cert-manager with Cloudflare DNS for TLS certificates, and internal DNS records to keep services available within my network. I have worked through issues involving ingress routing, TCP services, DNS resolution, reverse proxies, VLANs, and communication between Kubernetes workloads and services outside the cluster.

Internal services such as Forgejo also needed to remain reachable by Argo CD and CI runners during the transition. Troubleshooting this environment involved tracing requests through DNS, Docker networking, reverse proxies, ingress controllers, Kubernetes services, and individual pods.

This work has reinforced how important DNS and network design are to a reliable Kubernetes platform.

Building Better CI/CD and GitOps Workflows

Another area I have been improving is the path from source code to a running Kubernetes workload.

Operating a self-hosted Forgejo instance and container registry. Application images are built, versioned, pushed to the registry, and deployed through Kubernetes manifests managed by Argo CD.

For my custom applications, this provides a complete workflow:

1. Application code is updated in Git.
2. The CI runner builds the client or server image.
3. The image is tagged and pushed to the internal registry.
4. Kubernetes deployment configuration references the new version.
5. Argo CD synchronizes the change into the cluster.
6. Kubernetes performs the rollout and reports the workload’s health.

I have also been refining my image-tagging strategy. Mutable tags such as latest can be convenient, but explicit version tags provide clearer deployment history and more predictable rollbacks. Understanding Kubernetes image-pull behavior is an important part of making these deployments dependable.

Expanding Observability and Security

As the number of workloads has increased, observability has become increasingly important.

The cluster uses Prometheus, Grafana, kube-state-metrics, node exporters, and Grafana Alloy to provide visibility into infrastructure and application health. I have also created and exposed application-specific metrics, including metrics related to active client sessions.

Trivy Operator adds continuous security scanning for Kubernetes workloads. Its ServiceMonitor allows vulnerability and configuration findings to be collected alongside other cluster metrics.

I have also been working with Wazuh as part of a broader security-monitoring strategy. This includes estimating the storage required to move from Wazuh Cloud to a self-hosted AWS deployment and evaluating the amount of GP3 storage needed for log ingestion and retention.

Together, these tools help move the environment from simply running applications to actively measuring their reliability, security, and performance.

Working with AWS Infrastructure

My AWS work has included managing EC2-based systems, planning cloud storage, using Amazon S3 for application migrations, and integrating applications with AWS identity-based authentication.

One migration project involved transferring application directories and database components through S3. This required planning how data would be packaged, uploaded, protected, transferred, and restored in the destination environment.

I have also been evaluating AWS storage for self-hosted security workloads. For a Wazuh deployment, the size of the environment cannot be based only on the number of monitored systems. Log volume, index growth, retention periods, replicas, and performance requirements all affect the amount and type of EBS storage required.

GP3 provides the ability to size capacity, IOPS, and throughput independently, but those values still need to be based on real ingestion and retention requirements. This has led me to focus more closely on measuring workloads before selecting cloud resources.

Strengthening AWS IAM Integration

Identity and access management has also been a key part of my AWS work.

I have worked with AWS IAM authentication for Kafka clients, including integrating an AWS MSK IAM authentication library with newer Kafka client versions. This requires more than enabling a single setting. The application dependencies, credential provider, IAM policy, Kafka configuration, and network path all have to work together.

I have been applying a least-privilege approach by limiting permissions to the specific resources and actions required by each application. The goal is to avoid broad permissions and long-lived credentials wherever possible.

This same principle applies to Kubernetes. Sensitive values are stored as secrets, deployment keys are scoped to their intended purpose, and workloads are given only the access required to perform their functions.

Connecting Kubernetes and AWS

The most important lesson from this work has been that Kubernetes and AWS are most powerful when they are treated as parts of one platform.

Kubernetes provides scheduling, service discovery, application recovery, declarative configuration, and deployment automation. AWS provides flexible compute, storage, networking, managed services, and identity controls. GitOps ties the deployment process together, while monitoring and security tools provide the visibility needed to operate it responsibly.

My recent projects have helped me become more comfortable following an issue across these boundaries. An application problem may begin inside a Kubernetes pod, but the underlying cause could involve storage performance, an IAM policy, an unavailable cloud resource, DNS resolution, or a failed deployment.

My next steps are to continue improving deployment automation, strengthen disaster-recovery procedures, expand centralized monitoring and logging, and build more repeatable workflows between Git, Kubernetes, and AWS.

The goal is not simply to deploy more applications. It is to build infrastructure that is secure, observable, scalable, and capable of recovering when something goes wrong.