Smoke Testing with the Bash Automated Testing System 🦇

Tansu is an Apache Kafka® compatible broker that stores data in S3 or PostgreSQL — no code changes needed! 🔥 But how does it work? This article dives into our final stage of automated smoke testing to ensure every build is rock solid.

Our build process starts with an automated suite of unit ✅ and integration tests ✅. Only after passing these checks is the code built into a multi-architecture Docker container, tagged, and published to the GitHub Container Registry.

🔍 What’s new? We used to run a final manual "smoke" test — firing up an Apache Kafka client against the published container to confirm we were truly done (as in done done).

🔥 Now, it’s fully automated! We've extended our GitHub Actions workflow with the Bash Automated Testing System (bats), eliminating manual checks. Plus, we’re expanding our tests to cover multiple Kafka client libraries, including Python, JavaScript, and librdkafka.

Smarter testing, fewer headaches. 💡

We use the matrix strategy in our GitHub action workflow representing the combinations of storage (S3/PostgreSQL), cpu (ARM/AMD64) and kafka client (3.8/3.9) that we smoke test:

  strategy:
    fail-fast: false
    matrix:
      os:
        - ubuntu-latest
        - ubuntu-24.04-arm
      storage:
        - s3://tansu/
        - postgres://postgres:postgres@db
      kafka:
        - apache/kafka:3.8.1
        - apache/kafka:3.9.0
  runs-on: ${{matrix.os}}
  env:
    KAFKA_IMAGE: ${{matrix.kafka}}
    STORAGE_ENGINE: ${{matrix.storage}}

GitHub hosted ARM runners are finally now available to public repositories using the ubuntu-24.04-arm label 🎉. The runs-on uses both the AMD64 and ARM variants of our multi architecture docker image ✅.

The combinations run in parallel, independently using their own Tansu broker. Switching off fail-fast so that everything runs to completion. The env maps the combination into per job environment variables.

We define the test environment using docker compose, parameterised the KAFKA_IMAGE, and the STORAGE_ENGINE being used by Tansu:

services:
  db:
    image: postgres:17
    ...
  minio:
    image: quay.io/minio/minio
    ...
  kafka:
    image: ${KAFKA_IMAGE}
    command: /bin/sleep infinity
  tansu:
    image: ghcr.io/tansu-io/tansu:main
    environment:
      STORAGE_ENGINE: ${STORAGE_ENGINE}
      ...

Rather than running a broker, the Apache Kafka service sleeps indefinitely, so that we can use the Java CLI that is embedded in that container. Tansu is configured to use either S3 or PostgreSQL using the STORAGE_ENGINEenvironment variable having been set by the matrix strategy.

Workflow Steps 🚀

1. Start up Compose
2. Create a "tansu" bucket for MinIO
3. Ensure PostgreSQL is ready
4. Install and run BATS

steps:
  - uses: actions/checkout@v4
  - run: 1⃣ docker compose up --detach
  - run: 2⃣ docker compose exec minio mc ready local
  - run: 2⃣ docker compose exec minio mc alias set local http://localhost:9000 ${{ secrets.AWS_ACCESS_KEY_ID }} ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  - run: 2⃣ docker compose exec minio mc mb local/tansu
  - run: 3⃣ docker compose exec db pg_isready --timeout=60
  - run: 4⃣ sudo apt-get update
  - run: 4⃣ sudo apt-get install -y bats
  - run: 4⃣ bats --trace --verbose-run tests

For the smoke test, we execute commands in the Apache Kafka container using Tansu as the bootstrap server. We start by trying to delete a non-existant topic. Unix uses a non-zero status code to indicate an error. We assert that 1 is the resulting status code:

@test "delete non-existing topic" {
  run -1 docker compose exec kafka kafka-topics.sh \
    --bootstrap-server tansu:9092 \
    --delete \
    --topic test
}

The @test "delete non-existing topic" is the description used in our test output. While run -1 will run the command checking that it exits with a status code of 1. The docker compose exec kafka allows us to run kafka-topics.sh in the context of the kafka container using the version defined in the matrix. Finally, the tansu:9092 is the host name and port defined defined in the compose for our Tansu broker.

We create a topic using the CLI, verifying the output is "Created topic test." as expected:

@test "create topic" {
  run docker compose exec kafka kafka-topics.sh \
    --bootstrap-server tansu:9092 \
    --create \
    --topic test

  [ "${lines[0]}" = "Created topic test." ]
}

The run is also checking that the exit code from kafka-topics.sh is 0 (successful).

Listing the topics, to verify that our "test" topic is present:

@test "list topics" {
  run docker compose exec kafka kafka-topics.sh \
    --bootstrap-server tansu:9092 \
    --list

  [ "${lines[0]}" = "test" ]
}

Verify that we can't create a duplicate topic:

@test "create duplicate topic" {
    run docker compose exec kafka kafka-topics.sh \
      --bootstrap-server tansu:9092 \
      --create \
      --topic test

    [ "${lines[0]}" = "Error while executing topic command : Topic with this name already exists." ]
}

Being a duplicate doesn't cause kafka-topics.sh to return a non-zero status code, so we use a vanilla run checking the output is as expected instead.

Tansu is a stateless broker. Brokers are leaders of all topic partitions in a cluster. When describing a topic, as a result it looks a little different:

@test "describe topic" {
    run docker compose exec kafka kafka-topics.sh \
      --bootstrap-server tansu:9092 \
      --describe \
      --topic test

    [ "${lines[1]}" = "	Topic: test	Partition: 0	Leader: 111	Replicas: 111	Isr: 111	Adding Replicas: 	Removing Replicas: 	Elr: N/A	LastKnownElr: N/A" ]
    [ "${lines[2]}" = "	Topic: test	Partition: 1	Leader: 111	Replicas: 111	Isr: 111	Adding Replicas: 	Removing Replicas: 	Elr: N/A	LastKnownElr: N/A" ]
    [ "${lines[3]}" = "	Topic: test	Partition: 2	Leader: 111	Replicas: 111	Isr: 111	Adding Replicas: 	Removing Replicas: 	Elr: N/A	LastKnownElr: N/A" ]
}

For each of the 3 partitions, the leading broker, replicas and ISR are node 111. In Tansu all brokers are node 111.

We then verify that the offsets for an empty topic are as expected:

@test "before produce, earliest offsets" {
    run docker compose exec kafka kafka-get-offsets.sh \
      --bootstrap-server tansu:9092 \
      --topic test \
      --time earliest

    [ "${lines[0]}" = "test:0:0" ]
    [ "${lines[1]}" = "test:1:0" ]
    [ "${lines[2]}" = "test:2:0" ]
}

@test "before produce, latest offsets" {
    run docker compose exec kafka kafka-get-offsets.sh \
      --bootstrap-server tansu:9092 \
      --topic test \
      --time latest

    [ "${lines[0]}" = "test:0:0" ]
    [ "${lines[1]}" = "test:1:0" ]
    [ "${lines[2]}" = "test:2:0" ]
}

Produce 3 messages to the topic:

@test "produce" {
    run bash -c "echo 'h1:pqr,h2:jkl,h3:uio	qwerty	poiuy' | docker compose exec kafka kafka-console-producer.sh --bootstrap-server tansu:9092 --topic test"
    run bash -c "echo 'h1:def,h2:lmn,h3:xyz	asdfgh	lkj' | docker compose exec kafka kafka-console-producer.sh --bootstrap-server tansu:9092 --topic test"
    run bash -c "echo 'h1:stu,h2:fgh,h3:ijk	zxcvbn	mnbvc' | docker compose exec kafka kafka-console-producer.sh --bootstrap-server tansu:9092 --topic test"
}

Verify the content of the message on consumption:

@test "consume" {
    run docker compose exec kafka kafka-console-consumer.sh \
      --bootstrap-server tansu:9092 \
      --topic test

    [ "${lines[0]}" = "Partition:1	Offset:0	h1:stu,h2:fgh,h3:ijk	zxcvbn	mnbvc" ]
    [ "${lines[1]}" = "Partition:0	Offset:0	h1:pqr,h2:jkl,h3:uio	qwerty	poiuy" ]
    [ "${lines[2]}" = "Partition:2	Offset:0	h1:def,h2:lmn,h3:xyz	asdfgh	lkj" ]
    [ "${lines[5]}" = "Processed a total of 3 messages" ]
}

Conclusion

I'm not a huge fan of writing tons of shell scripts, but BATS for smoke tests feels like a solid compromise. The biggest factor? Using a CLI for the tests. During development, every script is run through shellcheck to catch hidden 💣 — highly recommended!

They also kind of remind me of writing expect scripts back in the day. They're also not too heavy weight, smoke testing that the application responds as we expect at a high level. Only time will tell if they're too tightly coupled to the output format - maybe exit codes will end up being sufficient.

They're also not the only tests that we run, the unit and integration tests examine specifics of encoding and decoding the protocol, and the integration with PostgreSQL and S3.

We're also looking at running similar tests against various other non-Java based clients. They're in various stages of readiness right now:

• Kafka Python, an example repository that will probably get wrapped in some pytest.
• Similarly, KafkaJS another example repository that will get wrapped into some further tests
• We currently test against librdkafka manually but are exploring automation options. One approach is using existing examples built with the library, likely integrating BATS. Speed is key—we want to avoid building the library from scratch. If no suitable alternative exists, we may package it into a multi-arch Docker container.
• Sarama is a Go client that has implemented the Kafka protocol. It also has a (small) CLI, that could use BATS in a similar way to the Java CLI.

Every pull request already runs the unit and integration tests, triggering a labelled Docker build, using the pr# as a suffix. This means we can also run automated smoke tests during the PR process ✅, catching issues long before they reach main. 🔥

Licensed under the GNU AGPL, Tansu is written in 100% safe 🦺 async 🚀 Rust 🦀 and 🌟 is available on GitHub.