This article is based on a talk by Beau Nelford of BILL at Sedai’s autocon conference as part of a session I led on developer environments.  From here on, we’ll let Beau tell the story,

Introduction

Creating efficient, scalable, and reliable development environments is crucial for innovation and productivity. Here at BILL, who is a leading financial automation software company, we have embraced Kubernetes to address this challenge, developing a unique approach that streamlines their development process. 

In this article, we will explore how our team at BILL leverages Kubernetes, Team Spaces, and custom-built tools to create a robust, flexible development ecosystem that supports our large-scale operations and rapid deployment cycles. We'll delve into the specifics of our approach, examining how it enhances developer productivity, ensures code quality, and maintains system reliability across a complex, high-transaction environment.  

Optimizing Kubernetes Development: The BILL Approach

At BILL, Kubernetes is central to our platform, which supports over 460,000 customers and manages transactions amounting to 1% of the U.S. GDP. Built on a combination of Java, Elixir, and AWS, the platform is designed to be scalable, reliable, and fast. Over 600 BILL engineers continuously deploy updates every 11 minutes, ensuring that the platform stays up-to-date and bug-free. 

Kubernetes helps create environments that mirror production, allowing developers to test isolated systems without affecting live services. Automated pipelines further guarantee that code is thoroughly tested before deployment, reducing the risk of bugs entering production.

Team Spaces vs. Local Spaces: A Comparison

BILL's Team Spaces are Kubernetes-based environments, deployed with Helm charts and namespaces, that replicate production. These ephemeral spaces allow developers to test features in isolation, then remove them after testing, optimizing resource use.

Local Spaces provide a lightweight, Docker Compose environment for faster, local testing on developer machines. This setup supports parallel development while maintaining the fidelity of testing.

Developing with Team Spaces and the Developer Portal

As software development becomes more complex and distributed, tools that provide developers with isolated, ephemeral environments for testing and deploying changes become essential. BILL’s Team Spaces, in conjunction with their Developer Portal, offers an innovative solution to this challenge. These environments allow developers to work in a production-like setup while isolating their changes from the live environment, which helps minimize the risk of introducing bugs into production and provides a faster feedback loop for iterative development.

How Team Spaces Work

A Team Space is essentially a temporary, isolated environment created using Kubernetes and Helm charts. Developers use these spaces to test their changes against the current master branch in isolation, ensuring that their work does not interfere with the work of other engineers. These spaces are ephemeral, meaning they are automatically destroyed after a set period (thanks to a time-to-live (TTL) mechanism), which helps in optimizing resource usage.

To break it down:

1. Creation of a Team Space: When a developer begins working on a new feature or a bug fix, they create a Team Space through the Developer Portal. This automatically triggers a GitLab pipeline, deploying the necessary services, databases, and other components into the Team Space.
2. Helm Chart Integration: The Team Spaces are defined using Helm charts, which include all parameters necessary for setting up the environment, such as URLs, service mesh, proxies, and database snapshots. This ensures that the environment is as close to production as possible.
3. Sanitized Data for Testing: Team Spaces also come with managed seed data, which is a sanitized snapshot of production databases. Engineers own these data snapshots, ensuring they have all the necessary information to perform accurate tests without compromising sensitive production data.
4. Isolation from Other Environments: Since each Team Space is isolated, developers can run tests without affecting other ongoing work. This isolation significantly reduces the likelihood of conflicts and ensures that the environment behaves exactly as it would in production, providing a realistic context for testing.
5. Automatic Cleanup: Once testing is complete, or after a set period, the Team Space is automatically destroyed, freeing up resources and ensuring that the system remains efficient and cost-effective.

Leveraging the Developer Portal

‍

The Developer Portal at BILL plays a crucial role in managing Team Spaces. It is a centralized platform where developers can view, create, and manage their environments in a user-friendly interface. The portal provides critical details about each Team Space, such as its status, creation time, and time-to-live (TTL). This level of visibility ensures that developers are fully aware of the state of their environments, reducing the chances of errors during development and testing.

The Developer Portal not only allows developers to manually interact with their environments but also integrates with the company's CI/CD pipelines to automatically trigger deployments. This integration helps streamline the process, making it easy for developers to push changes, run tests, and deploy updates without manual intervention.

Command-Line Interface (CLI) for Advanced Users

In addition to the Developer Portal, our BILL’s Divvy team has developed a Command-Line Interface (CLI) called Divvy Mission Control (DMC), which caters to engineers who prefer terminal-based interactions. This CLI allows developers to interact with their Team Spaces directly from the command line, providing the same functionality as the portal but in a more flexible, code-based manner.

• Creating and Managing Team Spaces: With DMC, developers can create new Team Spaces, view active spaces, and delete or update environments directly from their terminal. This is especially useful for more advanced users who want to automate repetitive tasks or integrate environment management into their scripts.
• Seamless Integration: The CLI integrates seamlessly with BILL’s identity and authorization platform, ensuring that engineers can securely manage their environments without leaving their preferred development setup.

Developer Tools at BILL: Portal and CLI

To further support our development teams, at BILL, we have developed an internal developer portal and a command-line interface (CLI) called DIVI Mission Control (DMC). These tools make it easy for developers to create, manage, and monitor their Team Spaces and Local Spaces. The developer portal provides a user-friendly dashboard that displays the status of all active Team Spaces, while the CLI allows more advanced users to interact with the environment directly from their terminals.

The ability to quickly manage environments via the developer portal or DMC ensures that developers can focus on their code rather than the underlying infrastructure. This combination of tools fosters efficiency and scalability within the development process.

Here is a breakdown of the tools BILL developers use to manage Team Spaces and Local Spaces:

Conclusion 

BILL's innovative approach to Kubernetes development environments demonstrates the power of custom-tailored solutions in addressing the unique challenges of large-scale, high-transaction software platforms. By implementing Team Spaces and Local Spaces, we have created a flexible, efficient system that allows their 600+ engineers to work in isolated, production-like environments, significantly reducing the risk of bugs and conflicts. The combination of Kubernetes, Helm charts, and custom tools like the Developer Portal and Divvy Mission Control (DMC) CLI provides a comprehensive ecosystem that streamlines the development process from local testing to production deployment.

Key takeaways from our approach at BILL include:

1. The importance of isolated, ephemeral testing environments that closely mirror production.
2. The value of automated resource management and cleanup in maintaining efficiency.
3. The benefits of providing both GUI (Developer Portal) and CLI (DMC) interfaces to cater to different developer preferences.
4. The significance of integrating development environments with CI/CD pipelines for seamless deployments.
5. The role of sanitized production data in creating realistic testing scenarios.

By adopting these practices and tools, we have created a development environment that supports rapid iteration, maintains high code quality, and enables continuous deployment every 11 minutes. This approach not only enhances developer productivity but also ensures the reliability and scalability of their platform, which manages transactions amounting to 1% of the U.S. GDP. If your organization is facing similar challenges in managing complex, large-scale development environments may be able to draw valuable insights from our use of Kubernetes and custom tooling to streamline your development processes.