Infrastructure, Batch Processing & Kubernetes

Kubernetes deployment automation, distributed batch processing, database coordination, and custom GitHub Actions for the admin platform ecosystem.

Role: Software Engineer — Infrastructure & Platform AutomationPeriod: May 2024 — Present8 min read

Java 21Spring BootShedLockKafkaMySQLFlywayKubernetesKustomizeDockerJibGitHub ActionsTypeScript

430+

Batch repo commits

Batch contributor rank

50+

Kubernetes config commits

Environments (dev/stg/prod)

Data verified from local repository analysis

A payment platform doesn't stop processing when everyone goes home. Settlements run at midnight. Reconciliation jobs execute on schedule. Reports are generated and distributed to merchants before they open their shops. Failures aren't acceptable — missed settlements mean delayed payouts.

This is the world of background processing. And in Kubernetes, where every service runs as multiple replicas for resilience, background jobs have a fundamental problem: who decides which replica runs the job?

The Replica Problem

Imagine three batch pod replicas. Each one wakes up at midnight and checks: "Is it time to run the daily settlement job?" Without coordination, all three start processing simultaneously. Settlements get duplicated. Merchants get paid twice. Reconciliation becomes a nightmare.

Warning·

Why not CronJob?

Kubernetes CronJob doesn't solve this — it creates a new pod per schedule, but multiple replicas of the same service all trigger the same logic. The correct approach is distributed coordination: a mechanism where replicas agree on who executes.

ShedLock: Distributed Lock Coordination

The solution was ShedLock, a Java library that uses a database-backed lock table. When a scheduled job triggers on any replica, that replica attempts to acquire a named lock. If successful, it executes the job. If another replica holds the lock, it skips gracefully.

ShedLock — distributed lock coordination

Java

1@Configuration2@EnableScheduling3@EnableSchedulerLock(defaultLockAtMostFor = "30m")4public class BatchSchedulingConfig {5 6  @Bean7  public LockProvider lockProvider(DataSource dataSource) {8      return new JdbcTemplateLockProvider(dataSource);9  }10}11 12@Component13public class SettlementJob {14 15  @Scheduled(cron = "0 0 0 * * ?")16  @SchedulerLock(17      name = "daily-settlement",18      lockAtMostFor = "30m",19      lockAtLeastFor = "5m"20  )21  public void processDailySettlements() {22      List<Settlement> pending = fetchPendingSettlements()23      for (Settlement s : pending) {24          processAndReconcile(s)25      }26  }27}

Representational configuration. Production includes additional monitoring hooks, lock timeout alerting, and graceful degradation paths.

Every new infrastructure dependency is a new thing that can fail. ShedLock uses the MySQL database that already exists. Adding a lock table added zero operational complexity. It was the smallest possible change to solve the replica problem.
Architecture principleUse existing infrastructure

The lock provider backed by a JDBC DataSource. The same MySQL database the service already depends on — zero new infrastructure to manage.

Java

1@Configuration2@EnableScheduling3@EnableSchedulerLock(defaultLockAtMostFor = "30m")4public class BatchSchedulingConfig {5 6  @Bean7  public LockProvider lockProvider(DataSource dataSource) {8      return new JdbcTemplateLockProvider(dataSource);9  }10}11 12@Component13public class SettlementJob {14 15  @Scheduled(cron = "0 0 0 * * ?")16  @SchedulerLock(17      name = "daily-settlement",18      lockAtMostFor = "30m",19      lockAtLeastFor = "5m"20  )21  public void processDailySettlements() {22      List<Settlement> pending = fetchPendingSettlements()23      for (Settlement s : pending) {24          processAndReconcile(s)25      }26  }27}

Kubernetes topology with admin pods, batch pods, shared MySQL, Kafka, and ingress boundaries — Kubernetes deployment topology: admin service pods and batch service pods share a MySQL database and Kafka cluster. The batch pods use ShedLock — a JDBC-backed lock table — to coordinate scheduled job execution.

The Deployment Puzzle

The admin platform and batch service run in three environments: development, staging, and production. Each needs different configurations — replica counts, resource limits, image tags. But the core deployment structure is identical across all three.

Kustomize solved this cleanly with a base + overlay structure:

Environment-agnostic manifests describing the service's deployment, service account, and resource shape. No hardcoded URLs or environment-specific values.

Kustomize — base + overlay structure

YAML

1# base/deployment.yaml2apiVersion: apps/v13kind: Deployment4metadata:5name: batch-service6spec:7replicas: 18template:9  spec:10    containers:11      - name: batch12        image: batch-service:latest13        ports:14          - containerPort: 808015        livenessProbe:16          httpGet: { path: /health, port: 8080 }17        readinessProbe:18          httpGet: { path: /health, port: 8080 }19 20---21# overlays/prod/kustomization.yaml22apiVersion: kustomize.config.k8s.io/v1beta123kind: Kustomization24resources:25- ../../base26patches:27- patch: |-28    - op: replace29      path: /spec/replicas30      value: 331    - op: replace32      path: /spec/template/spec/containers/0/image33      value: batch-service:v1.2.3

Representational configuration. Production overlays include additional resource limits, HPA configs, and pod disruption budgets.

Kustomize

Base + overlay

Docker / Jib

Container build

GitHub Actions

CI/CD pipeline

Flyway

Schema migration

Kubernetes

Orchestration

Horizontal Pod Autoscaler

Auto-scaling

Tip·

Transparency over templating

The key insight with Kustomize was that final YAML is always inspectable. Unlike Helm, where you need to render templates to see the final output, Kustomize produces plain YAML that can be reviewed, diffed, and audited. For a financial platform handling settlements, auditability was the deciding factor.

Automating the Repetitive

The team had recurring workflow tasks that marketplace GitHub Actions couldn't handle cleanly — branch synchronization across multiple repositories, release branch preparation, and pre-deployment validation checks.

Custom GitHub Actions

I built custom TypeScript GitHub Actions for these workflows. Each action was small — typically under 100 lines — but eliminated minutes of manual work per task, repeated dozens of times per sprint. A shared TypeScript library for GitHub API interactions made developing future actions faster.

Custom GitHub Action — branch synchronization

TypeScript

1import * as core from '@actions/core'2import * as github from '@actions/github'3 4async function syncToDev(): Promise<void> {5const octokit = github.getOctokit(core.getInput('token'))6const { repo } = github.context7 8const devBranch = await octokit.rest.repos.getBranch({9  ...repo, branch: 'dev',10})11 12const latestSha = devBranch.data.commit.sha13await octokit.rest.repos.merge({14  ...repo,15  base: 'dev',16  head: core.getInput('source-branch'),17})18 19core.notice('Dev synchronized')20}21 22syncToDev().catch((err) => core.setFailed(err.message))

Representational — simplified from production automation. Actual actions include additional validation, rollback handling, and notification hooks.

The ROI of Small Automations

5 min/day saved

A 50-line GitHub Action that saves one engineer 5 minutes per day pays for itself in less than two weeks. Across a team of 20, the ROI on small automations is enormous — and they compound as the team grows.

Metrics dashboard showing contribution distribution across repositories — Engineering contribution overview across the ecosystem. The batch platform (430+ commits, #3 contributor) and admin platform (1,200+ commits, #7 contributor) represent the deepest areas of contribution.

Engineering Decisions

Engineering Decision

ShedLock over a Dedicated Scheduler

Kubernetes replicas all trigger scheduled methods. Without coordination, jobs execute on every replica simultaneously, causing duplicates in settlement processing.

Decision: Use ShedLock with a JDBC-backed lock table in existing MySQL. No new infrastructure. Locks expire automatically if the holder crashes.

Consequences:

Zero new infrastructure — uses the database the service already depends on
Simple first-lock-wins model that the entire team understood immediately
Lock expiration provides crash recovery without manual intervention

Tradeoffs:

MySQL becomes a coordination dependency — database issues affect job scheduling
Not suitable for complex DAG-style workflows (use Airflow / Quartz for that)

Engineering Decision

Kustomize over Helm

Deployment configurations needed environment-specific customization without introducing a full template language or external chart repositories.

Decision: Organize Kubernetes manifests as Kustomize base + overlay structure. No external dependencies, no template rendering step.

Consequences:

Final manifests are plain YAML — always inspectable and auditable
No template debugging — what you see in the overlay is what gets applied
Native Kubernetes tooling with zero external dependencies

Tradeoffs:

Less suitable for distributing as reusable charts to external teams
Complex conditional logic requires Kustomize patches instead of template functions

Engineering Decision

Custom GitHub Actions over Marketplace Actions

Team branching strategy required manual git operations that were error-prone and time-consuming.

Decision: Build custom TypeScript GitHub Actions with a shared library for GitHub API interactions.

Consequences:

Automated branch synchronization — no manual merges, no forgotten syncs
Consistent CI behavior across all team repositories
TypeScript provides type safety, testability, and maintainability

Tradeoffs:

Custom actions require ongoing maintenance and team debugging knowledge
Version management of action releases across consuming repositories

Practical Lessons

The best infrastructure decisions eliminate categories of problems without creating new categories of complexity. ShedLock didn't add a new system to manage — it used one that already existed. Kustomize didn't introduce a new language to learn — it was just YAML with inheritance.

Distributed lock tuning is more art than science. A lock duration too short causes contention. Too long blocks crash recovery. The right value depends on job execution time, replica count, and acceptable failure windows. We tuned by monitoring lock contention in production and adjusting incrementally.

Overlays should stay small. When a Kustomize overlay needs more than a few patches, it's usually a sign that the base manifest needs refactoring. Large overlays defeat the purpose of centralization.

Database sharing requires explicit ownership. When batch and admin services share MySQL, every schema migration needs cross-team awareness. We established a convention: each service's tables are prefixed by domain, and migration files include ownership metadata.

Confidentiality note: Case study details are anonymized. Repository existence and primary patterns verified. No proprietary infrastructure configurations, internal endpoints, or business logic are exposed.