Skip to content

January 2025

  • in DevOps
  • 6 min read

The Production-Ready Python Template for Scalable Projects 🌟

📖 Overview

⚙️ This python-template is built with modularity, scalability, and security in mind, leveraging cutting-edge tools and best practices to streamline development, testing, and deployment workflows for CloudOps, FinOps, and Data Science projects.

🔥 The project integrates VSCode DevContainers, GitHub Actions CI/CD workflows and uv/ruff/pytest to simplify development workflows and ensure security and compliance with Trivy and Chainguard Wolfi.

🛠️ Features

Complete Walkthrough for CloudOps Python Template

🚀 The Production-Ready Python Template for Scalable CloudOps and FinOps Projects 🌟

📣 Attention Cloud Engineers, DevOps Pros, and Data Scientists! 🥇

💬 Are you tired of setting up Python environments, debugging CI/CD pipelines, or dealing with dependency management?

💼 Today, I’m thrilled to introduce a powerful, scalable, and production-ready Python template—designed to streamline CloudOps automation, DevOps workflows, and FinOps visual analytics like never before!

🔥 Why This Python Template?

  • 🥉 ✅Lightning-Fast Toolchain: Powered by 📦 uv - Next-generation Python dependency and build management, 💅 ruff - Linting and formatting at blazing speed, and 🧪 pytest - Robust testing framework with coverage reports.
  • 🥈 ✅Effortless CI/CD Pipelines: 🛠️ Taskfile Automation - Say goodbye to manual SDLC repetitive tasks, 🐳 Containerized Workflows – 🛡️ Security-first practices and Multi-stage Wolfi-based Docker builds for scalable production-ready environments, and ⚙️ Auto-publish to PyPI and GitHub Container Registry (GHCR) with GitHub Actions.
  • 🥇 ☑️CloudOps Automation and FinOps Toolkit – Pre-configured hybrid-cloud workflows and seamlessly integrations (jupyterlab, mkdocs, boto3, moto) for managing cloud infrastructure 🌐.

⚙️ Key Features

Feature Toolchain Purpose
🛠️ Configuration pyproject.toml Centralized configuration for dependencies, testing, and linting.
🧹 Task Automation Taskfile Automates repetitive tasks like linting, testing, and publishing.
📦 Python Dependencies uv Lightning-fast dependency resolution, caching, and builds.
💅 Linting & Formatting ruff Enforces code quality standards, auto-formatting, and import sorting.
🧪 Testing Framework pytest Comprehensive unit tests, integration tests with coverage reporting.
🐳 Docker Integration Dockerfile + DevContainer Optimized wolfi-based multi-stage builds for CI/CD and local development environments.
🦾 CI/CD Pipelines GitHub Actions Automated builds, tests, and deployments to PyPI and GHCR.
📝 Security Compliance chainguard/wolfi-base + SBOM + Attestations Ensures compliance, vulnerability scanning, and security transparency.

💬 Let’s Connect for insights on DevOps Accelerate, CloudOps Automation, and FinOps Toolkit! 💼 Build Faster and Smarter: Drop your thoughts, questions, or suggestions below!

🔗 #CloudOps #DevOps #Python #CI/CD #GitHubActions #Docker #PyPI #AWS #Automation #InfrastructureAsCode #FinOps 🚀

🧑‍💻 Software Development Lifecycle (SDLC) with CI/CD 🔄

🧑‍💻 Software Development Lifecycle (SDLC) with CI/CD 🔄

Stage Task Name Description Command
Dev install Install Python dependencies. task install
Dev format Format code using Ruff. task format
Dev lint Perform Python linter and code formatter with Ruff for errors and formatting issues. task lint
Dev code_quality Validcate Code Quality with SonarQube task code_quality
Dev render-docs Generate documentation for components, modules, and GitHub actions. task render-docs
Test test Run unit tests with Pytest and generate coverage reports. task test
Test ci Full CI pipeline with clean, lint, build, and test steps. task ci
Deploy build Build artifacts for deployment. task build
Deploy publish Publish the project to PyPI. task publish
Deploy sync_data_up Sync local data directory to S3 bucket. task sync_data_up
Deploy sync_data_down Download data from S3 bucket to local directory. task sync_data_down
Utility clean Remove Python cache files and temporary directories. task clean
Utility real-clean Perform a deep clean, including dependencies and cache files. task real-clean
Docs help List all available tasks with descriptions. task help 
cloudops-template/
├── .github/
│   ├── workflows/               ## CI/CD pipelines for testing, building, and publishing
│   │   ├── test.yml
│   │   ├── build.yml
│   │   └── publish.yml
├── .devcontainer/               ## DevContainer configurations for VSCode
│   ├── Dockerfile
│   └── devcontainer.json
|-- notebooks                    ## A directory to place all notebooks files.
|   ├── *.ipynb                  ## Jupyter notebooks
|   └── my_nb_path.py            ## Imported by *.ipynb to treat src/ as PYTHONPATH
├── src/                         ## Python modules developed in this project
│   ├── cloudops/                ## Core source code for cloudops module
│   │   ├── __init__.py
│   │   └── toolkit.py           ## Toolkit utilities. Imported by *.ipynb to XXX their outputs
├── tests/                       ## Unit and integration tests
│   └── test_toolkit.py
├── Taskfile.yaml                ## Task automation
├── Dockerfile                   ## Multi-stage Docker build
├── pyproject.toml               ## Configuration for dependencies and tools for Python toolchains
├── README.md                    ## Documentation
└── LICENSE                      ## License

🚀 Quick Start

Enterprise-grade Development Workflows Best Practices
  • 1. Local Development Tools: DevContainer, Docker, Python Packaging Standards* using pyproject.toml.
  • 2. Automated CI/CD Pipelines ensure testing, packaging, and deployment are reliable and scalable.
  • 3. Dockerized Builds provide consistent runtime environments and secure image hosting on GHCR.
  • 4. MkDocs Documentation simplifies documentation management and enhances team collaboration.
  • 5. Taskfile Workflow streamlines local development and integrates seamlessly with CI/CD pipelines.

1️⃣ Local Development Setup

Step 1.1. Clone the Repository
git clone https://github.com/nnthanh101/cloudops.git
cd runbooks
Step 1.2. Run in DevContainer

✅ Install VSCode and 🐳 Docker Desktop.

  1. Open the project in VSCode.
  2. Install the Remote - Containers extension.
  3. Reopen in Container:
    Command Palette Cmd+Shift+P (Cmd+Shift + P) in MacOS or Ctrl+Shift+P in Windows(Ctrl+Shift + P) → Reopen in Container.

2️⃣ SDLC Taskfile Commands

Step 2.1. Install Python dependencies using UV
task install
Step 2.2. Run CI/CD Pipelines
echo "Run CI/CD pipeline tasks: clean, lint, format, test, and build"
task ci

echo "Remove Build Artifacts"
task clean
Step 2.3. 🧪 Testing and Code Quality

  • Code Formatting and Linting

    # task format
task lint

  • Validcate Code Quality with SonarQube

    task code_quality

  • Test the Codebase with Coverage Reports

    task test --verbose

---------- coverage: platform darwin, python 3.12.6-final-0 ----------
Name                       Stmts   Miss  Cover   Missing
--------------------------------------------------------
src/cloudops/__init__.py       2      0   100%
src/cloudops/toolkit.py       33      7    79%   107-114
--------------------------------------------------------
TOTAL                         35      7    80%
Step 2.4. Build Docker Image
task build
Step 2.5. Publish Artifacts to PyPI and GHCR

  • Build and Publish Docker Image to GHCR and DockerHub to nnthanh101/runbooks:latest

    ./docker-build-push.sh runbooks latest

  • Build and Publish the Project Package to PyPI: runbooks

    Automatically deploys Docker images and Python packages to PyPI.
    • In order to publish to PyPI you need to create a secret called UV_PUBLISH_TOKEN with your PyPI access token under API tokens.
    • If you are using Twine to upload your projects to PyPI, set up your $HOME/.pypirc file.
    UV_PUBLISH_TOKEN=your-uv-publish-token

    task publish

3️⃣ 📦 CI/CD using GitHub Actions

🐳 Tests, linting, and publishing workflows integrated via GitHub Actions.

Step 3.1. PyPI Publishing Workflow
  • Triggered on main branch push.
  • Tests, formats, and builds the package.
  • Publishes the package to PyPI pip install runbooks using trusted OIDC authentication.
Step 3.2. Docker Image Workflow
Push Docker Image

Triggered automatically on GitHub Actions:

  1. Push a new tag or release: 

    git tag v0.1.0
git push origin v0.1.0

  2. Monitor the logs: - GitHub Actions → Workflows → Create and publish Docker image

4️⃣ Documentation with MkDocs

Step 4.1: Build the Documentation
mkdocs build
Step 4.2: Serve the Documentation Locally
mkdocs serve
https://nnthanh101.github.io/runbooks/

Access URL: http://127.0.0.1:8000

5️⃣ Usage Examples for Python API from PyPI Installation

Step 5.1. Install the Package:
pip install runbooks
Step 5.2. Use the Toolkit API:
from runbooks.toolkit import add, subtract, multiply, divide
print(add(5, 10))  ## Output: 15.0
Step 5.3. Use the Calculator API:
from runbooks.calculator import Calculator
calc = Calculator(precision=2)
print(calc.add(5, 10))  ## Output: 15.0

6️⃣ Usage Examples for Docker Image

Step 6.1. Pull and Run Container:
docker pull ghcr.io/nnthanh101/runbooks:latest
docker run --rm ghcr.io/nnthanh101/runbooks:latest runbooks toolkit add 5 10
Step 6.2. Shell Access for Debugging:
docker run -it --rm ghcr.io/nnthanh101/runbooks:latest /bin/sh

📚 Conclusion

This Python template provides a production-ready framework for building scalable, secure, and automated systems. It incorporates:

  1. Efficient DevOps pipelines for end-to-end CI/CD workflows for testing, publishing, and containerized deployments.
  2. Security-first practices with Chainguard's Wolfi, SBOM, artifact attestations, and vulnerability scanning.
  3. Simplified Development: Task automation & Streamlined dependency management with Taskfile and uv for fast builds and reliable testing. Also, Pre-configured DevContainer and Taskfile streamline workflows for new developers. 🚀
🏁 Ready to get started? ⚡️

🛠️ Clone the repo, run the tasks, and deploy your pipelines today! Let me know if you need enhancements like DevOps Accelerate using Terraform IaC, CloudOps Automation, and FinOps FOCUS™ Toolkit! 🥇