Spark on K8s Log Collection Nightmare: A Complete Guide to Fixing Driver/Executor Logs in S3 and History Server Display

Symptom: History Server Shows Nothing, Where Did the Logs Go?

Last Friday afternoon, our monitoring went haywire—users reported that the Spark History Server was completely blank. The spark-submit jobs had finished, I could see the spark-events directory in the S3 bucket, but clicking into it showed nothing.

To make it weirder, I ran kubectl logs on the driver pod—logs were there. Executor pods had stdout/stderr too. But there was no application_xxx.inprogress file in S3.

My first thought: “The log pipeline is broken.”

Root Cause Analysis: Three “I Thought So” Traps

After an afternoon of digging through code and community posts, I found this issue is incredibly common in Spark on Kubernetes. Someone on Reddit complained: “I am looking for an option to add sparkApplicationId to driver and executor pod names.” — The real problem underneath is the chaos in log path mapping.

Three core issues:

1. spark.eventLog.dir is misconfigured. Many people set s3a://bucket/spark-events, but Spark on K8s doesn’t use the Hadoop S3A connector by default. You need to explicitly add spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem.

2. Driver and executor log paths are not unified. By default, Spark’s event log is only written by the driver. Executor stdout/stderr never make it to S3. You need extra configs: spark.kubernetes.driver.log.persist and spark.kubernetes.executor.log.persist.

3. History Server read permission issues. Even if logs are written, the History Server’s service account might not have s3:GetObject permission on the S3 bucket. This trap cost us two days.

Fix Steps: From Zero to a Working History Server

1. Verify S3 Path and Hadoop Configuration

First, check if your spark-submit command or config file includes these critical items:

--conf spark.eventLog.enabled=true
--conf spark.eventLog.dir=s3a://my-bucket/spark-events/
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
--conf spark.hadoop.fs.s3a.access.key=YOUR_ACCESS_KEY
--conf spark.hadoop.fs.s3a.secret.key=YOUR_SECRET_KEY
--conf spark.hadoop.fs.s3a.endpoint=https://s3.ap-southeast-1.amazonaws.com

Note: If you’re using an S3-compatible store (like MinIO) with self-signed certs, you’ll also need --conf spark.hadoop.fs.s3a.connection.ssl.enabled=false (don’t ask how I know).

2. Enable Driver and Executor Log Persistence

This is the step most people miss. By default, Spark only writes event logs to S3—driver and executor stdout/stderr don’t get uploaded automatically. You need to explicitly enable them:

--conf spark.kubernetes.driver.log.persist=true
--conf spark.kubernetes.executor.log.persist=true
--conf spark.kubernetes.driver.log.persist.dir=s3a://my-bucket/driver-logs/
--conf spark.kubernetes.executor.log.persist.dir=s3a://my-bucket/executor-logs/

Note: spark.kubernetes.driver.log.persist only became stable after Spark 3.1.1. If you’re on an older version, you’ll need to upgrade.

3. Configure IAM Permissions for History Server

The History Server is itself a Spark application—it needs to read S3 event logs. If you’re using IRSA (IAM Roles for Service Accounts), make sure the service account has the right policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::my-bucket",
        "arn:aws:s3:::my-bucket/*"
      ]
    }
  ]
}

If you’re using access key/secret key, just set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the History Server Deployment’s environment variables.

4. Verify Log Writes

Run a simple Spark Pi job to test:

spark-submit \
  --master k8s://https://<api-server> \
  --deploy-mode cluster \
  --conf spark.kubernetes.container.image=apache/spark:3.5.0 \
  --conf spark.eventLog.enabled=true \
  --conf spark.eventLog.dir=s3a://my-bucket/spark-events/ \
  --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
  --conf spark.hadoop.fs.s3a.access.key=xxx \
  --conf spark.hadoop.fs.s3a.secret.key=xxx \
  local:///opt/spark/examples/jars/spark-examples.jar 100

After the job finishes, check the S3 bucket:

aws s3 ls s3://my-bucket/spark-events/

You should see files like app-20260624020000-0000.inprogress. If not, check the driver pod logs:

kubectl logs <driver-pod> | grep -i "eventlog"

A common error is java.io.IOException: No FileSystem for scheme: s3a—this means the Hadoop S3A dependency isn’t loaded.

5. Restart History Server and Verify

If logs are in S3 but History Server is still empty, check the History Server’s startup config. The correct setup looks like this:

# history-server-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spark-history-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: spark-history-server
  template:
    metadata:
      labels:
        app: spark-history-server
    spec:
      containers:
      - name: history-server
        image: apache/spark:3.5.0
        args:
          - "/opt/spark/bin/spark-class"
          - "org.apache.spark.deploy.history.HistoryServer"
        env:
        - name: SPARK_HISTORY_OPTS
          value: "-Dspark.history.fs.logDirectory=s3a://my-bucket/spark-events/ -Dspark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem -Dspark.hadoop.fs.s3a.access.key=xxx -Dspark.hadoop.fs.s3a.secret.key=xxx"
        ports:
        - containerPort: 18080

Note: The SPARK_HISTORY_OPTS environment variable is the key to History Server config. Many people put configs in spark-defaults.conf, but History Server doesn’t always load that file on startup.

Configuration Comparison Table

FAQ

Q1: Why does my History Server show “No completed applications found” even though there are files in S3?

A: The most common cause is a mismatch between the History Server’s spark.history.fs.logDirectory and the driver’s spark.eventLog.dir. Check two things:

1. The paths must be identical (including bucket name and prefix).
2. The History Server must be able to access S3—if using IRSA, check the service account annotation; if using access keys, check the environment variables.

Q2: After enabling spark.kubernetes.driver.log.persist, where are the log files?

A: The default path is s3a://<eventLog.dir>/<app-id>/driver/. You can customize it via spark.kubernetes.driver.log.persist.dir. Note: This feature is only available in Spark 3.1.1+ and requires the Hadoop S3A dependency.

Q3: I’m on Spark 3.0.0. Can I persist logs to S3?

A: Barely, but I wouldn’t recommend it. The spark.kubernetes.driver.log.persist feature in Spark 3.0.0 is experimental and doesn’t support executors. Upgrade to 3.1.1 or later. If you’re stuck on 3.0.0, consider using a sidecar container to manually upload logs.

Q4: Do I need extra config for S3-compatible stores like MinIO?

A: Yes. Besides fs.s3a.impl, you need:

• spark.hadoop.fs.s3a.endpoint: Pointing to the MinIO API address.
• spark.hadoop.fs.s3a.path.style.access=true: MinIO requires path-style access.
• If MinIO doesn’t have HTTPS, add spark.hadoop.fs.s3a.connection.ssl.enabled=false.

Q5: History Server loads slowly after logs are written to S3. What can I do?

A: Consider enabling spark.history.fs.updateInterval to reduce polling frequency (default is 10 seconds). If log files are large, set spark.history.fs.maxLogEntries to limit the maximum number of log entries per application. For production, set up an S3 lifecycle policy to archive old logs to Glacier, reducing the History Server’s scan scope.

References & Community Insights

The following authoritative resources were referenced for architectural best practices and specifications:

• CNCF Cloud Native Interactive Landscape
• AWS Architecture Center
• Kubernetes Official Documentation