🛠️ DevSecOps for Multi-Cloud 🌥️

Cloud DevSecOps using Docker, Terraform, and Multi-Cloud 🚀

🎯 The CloudOps/GitOps Platform utilizes HashiCorp's Terraform Infrastructure as Code (IaC) on the AWS Cloud 🌥️, offering speed, scalability, agility, and cost efficiency. ⚡

1. AWS-Terraform DevContainer

This DevContainer provides a pre-configured development VSCode Dev-Container environment based on Ubuntu 24.04 LTS with essential DevOps & CloudOps Tools. It simplifies building and testing cloud-native applications for hybrid-cloud platforms (AWS, Azure) by providing comprehensive set of tools for infrastructure automation, container orchestration, Kubernetes management, CI/CD pipelines, and cloud-native application development, including Terraform, Kubernetes, AWS CLI, Azure CLI, and more.

Key Features: Ubuntu Docker, K8s & CI/CD, Python | Node | Go

• Ubuntu 24.04 LTS (Noble): A stable and secure base image optimized for long-term support and compatibility with modern cloud-native tools.
• DevOps toolchain: Pre-installed with tools like awscli, azure-cli, terraform, kubectl, helm, and more for seamless hybrid-cloud development.
  • Docker-in-Docker support: Enables containerized development and testing within isolated environments.
  • Kubernetes-ready: Includes Minikube | Kind | k3s, K8s tools: Kubectl, Helm, and other lightweight Kubernetes tools for local testing and development microservices such as K9s.
  • CI/CD tools: Integrated with tools like terraform, tfsec, act, and tflint to enable automated pipelines and code quality checks.
• Python, Node.js, Go support: Pre-installed with interpreters and runtime environments for cross-language development and multi-service applications.

1.1. Prerequisites

Prerequisites: Docker and VSCode

Before using this development environment, ensure you have Docker and VSCode installed on your local machine.

• Docker Desktop - The fastest way to containerize applications.
• VisualStudio Code - Visual Studio Code is a code editor redefined and optimized for building and debugging modern web and cloud applications.
• VSCode Remote Development Extension Pack - An extension pack that lets you open any folder in a container, on a remote machine, or in WSL and take advantage of VS Code's full feature set.

1.2. Getting Started

Reopen in Container in VSCode

1. Clone the repository: Clone the repository that includes the devcontainer.json and the Dockerfile for this setup. On a terminal, inside your Terraform project, execute the following on Mac, Linux or WSL:

2. Open in VSCode: Open the repository in VSCode.

3. Build the Dev Container or Reopen in Container: * VSCode will automatically detect the devcontainer.json file and prompt you to Reopen in Container. This will automatically build the container based on the provided Dockerfile. * To reopen in container manually, open the command pallete on VS Code and select Rebuild and Reopen in Container

4. Use the pre-configured tools: Once inside the container, all tools like terraform, kubectl, and awscli are available for use directly from the terminal and ready for use.

5. [Advanced] Running Development Services

   • Use Docker Compose to spin up services like the API, PostgreSQL, and Elasticsearch locally.
   • K3s is used to deploy Kubernetes resources locally.

curl -sL https://raw.githubusercontent.com/1xOps/CloudOps/.dev-container/init-devcontainer.sh | bash

# git clone https://github.com/1xOps/CloudOps
# cd CloudOps/.dev-container

echo "The list of Tools and Utilities to help you build and manage AWS infrastructure with Terraform !!!"
tools.sh

1.3. Taskfile | Makefile Usage

Taskfile

111

Makefile

make [tab][tab]

For example, if you want to explore the most common terraform commands:

make terraform/[tab]
apply     clean     destroy   fmt       init      init/     plan      validate  version

To display all available commands:

make help

For more information about each Make targets available.

---

2. AWS-Terraform Modules

• module folder: This is the official terraform aws modules which we can use to save our efforts.
  • https://github.com/terraform-aws-modules
  • If there is one new/latest module, you can clone/update here. Do a fully testing when update the module.
• global folder: It includes all the aws provider information.

---

3. Tools & Utilities

The list of pre-installed tools and utilities that are boostrapped to your Multi-Cloud environment

OS Utilities

• jq = JSON processor
• yq = YAML processor
• envsubst = The envsubst command is used to get a substitute of environment variables and that's what its name suggests.
• bash-completion = Bash completion is a bash function that allows you to auto complete commands or arguments by typing partially commands or arguments, then pressing the [Tab] key.
• aliases (k, kgn, kgp, tfi, tfp, tfy) - type alias to see them

[x] AWS Utilities

• AWS CLI version 2
• IAM role

[x] Terraform and Related Tools

| Apply | Tool                                                    | Description                                                                | Language | Advantages                                                                                                                        | Disadvantages                                              |
|-------|---------------------------------------------------------|----------------------------------------------------------------------------|----------|-----------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------|
| [x]   | **Terraform**                                           | Infrastructure as Code tool for building, changing, and versioning infrastructure | Go       | Widely used, supports multiple providers, strong community support                                                                | State management can be complex                            |
| [x]   | **Terraform Commands**                                  | Built-in Terraform commands (e.g., `validate`, `plan`, `apply`)           | Go       | Essential for managing Terraform configurations, built-in commands                                                                | Limited to syntax and execution, not security checks        |
| [x]   | **Terraform Docs**                                      | Generate documentation from Terraform modules                             | Go       | Automates documentation generation, supports various output formats                                                               | Limited to Terraform modules                               |
| [x]   | **Tfsec**                                               | Security scanner for Terraform code                                       | Go       | Fast, checks for security best practices, easy to use                                                                             | Limited to security checks                                 |
| [x]   | **Terrascan**                                           | Detects security vulnerabilities and compliance violations                | Go       | Supports multiple IaC tools, extensive policy library (supports OPA)                                                              | Can be slow on large codebases                             |
| [x]   | **TFLint AWS Ruleset** (`tflint-ruleset-aws`)           | Linter for Terraform code with AWS-specific rules                         | Go       | Pluggable, supports custom AWS rules, actively maintained                                                                         | Requires configuration                                      |
| [x]   | **TFLint AzureRM Ruleset** (`tflint-ruleset-azurerm`)   | Linter for Terraform code with Azure-specific rules                       | Go       | Pluggable, supports custom Azure rules, actively maintained                                                                       | Requires configuration                                      |
| [x]   | **Checkov Terraform Resource Scans**                    | Static code analysis tool for infrastructure-as-code                      | Python   | Supports multiple IaC tools, checks for security best practices, supports custom checks                                           | Might have false positives/negatives                        |
| [x]   | **Terraform Compliance** (`terraform-compliance`)       | Compliance-focused tool for Terraform                                     | Python   | Scenario-based tests, easy to read, supports custom compliance checks                                                            | Not a comprehensive linter, focused on behavior testing     |

---

Kubernetes Utilities

• kubectl = The Kubernetes command-line tool, kubectl, allows you to run commands against Kubernetes clusters. You can use kubectl to deploy applications, inspect and manage cluster resources, and view logs.
• helm = Helm helps you manage Kubernetes applications — Helm Charts help you define, install, and upgrade even the most complex Kubernetes application.
• kns = kns is a very small shellscript that utilizes fzf to switch between Kubernetes namespaces fast.

• kctx = kubectx is a tool to switch between contexts (clusters) on * kubectl faster. A Kubernetes context is a group of access parameters that define which cluster you're interacting with, which user you're using, and which namespace you're working in. It's helpful if you need to access different clusters for different purposes or if you want to limit your access to certain parts of a cluster.

• c9 = Cloud9 CLI and tools (c9 open file.txt)

• k9s = K9s is a terminal based UI to interact with your Kubernetes clusters. The aim of this project is to make it easier to navigate, observe and manage your deployed applications in the wild. K9s continually watches Kubernetes for changes and offers subsequent commands to interact with your observed resources.
• eks-node-viewer = eks-node-viewer is a tool for visualizing dynamic node usage within a cluster. It was originally developed as an internal tool at AWS for demonstrating consolidation with Karpenter. It displays the scheduled pod resource requests vs the allocatable capacity on the node. It does not look at the actual pod resource usage.

4. Makefile

aws-terraform-dev-container > available targets:

  aws/cfn-lint/install                Install AWS CloudFormation Linter
  aws/cli/install                     Install AWS Command Line Interface v2
  aws/cli/version                     Display AWS CLI version
  aws/cloudformation/create-change-set Creates a list of changes that will be applied to a stack so that you can review the changes before executing them.
  aws/cloudformation/create-change-set-without-parameters Creates a list of changes that will be applied to a stack so that you can review the changes before executing them.
  aws/cloudformation/create-folder-structure Create a folder structure for CloudFormation projects
  aws/cloudformation/create-parameters Copy a CloudFormation parameters to be used as example
  aws/cloudformation/create-project   Create a CloudFormation project structure
  aws/cloudformation/create/service-linked-role Creates  an  IAM  role that is linked to a specific Amazon Elasticsearch service.
  aws/cloudformation/create-stack     Creates a stack as specified in the template.
  aws/cloudformation/create-stack-without-parameters Creates a stack as specified in the template. (don't pass --parameters flag)
  aws/cloudformation/create-template-yaml Copy a CloudFormation template to be used as example
  aws/cloudformation/delete-change-set Delete latest change-set created
  aws/cloudformation/delete/service-linked-role Deletes  an  IAM  role that is linked to a specific Amazon Web Services service.
  aws/cloudformation/delete-stack     Delete CloudFormation Stack
  aws/cloudformation/describe-stack-events Returns  all  stack  related  events  for  a specified stack in reverse chronological order.
  aws/cloudformation/describe-stack   Returns  the  description for the specified stack; if no stack name was specified, then it returns the description for all the stacks created.
  aws/cloudformation/detect-stack-drift Detects  whether a stack's actual configuration differs, or has drifted , from it's expected configuration, as defined in  the  stack  template and  any  values specified as template parameters.
  aws/cloudformation/estimate-template-cost Returns  the  estimated monthly cost of a template
  aws/cloudformation/execute-change-set Execute latest change-set
  aws/cloudformation/hygiene          Execute CFN Lint and pre-commit rules
  aws/cloudformation/latest-change-set Display latest change-set
  aws/codeartifact/login              Login into AWS CodeArtifact
  aws/ssm/install-plugin              Install AWS SSM plugin
  aws/ssm/start-session               Start session with AWS Systems Manager Session Manager
  aws/sso/login                       Login into AWS account and export credentials to ~/.aws/credentials
  aws/sts/get-caller-identity         Returns  details  about the IAM user or role whose credentials are used to call the operation.
  checkov/run                         Run Checkov
  checkov/version                     Display checkov version
  doc/build                           Builds documentation
  doc/init                            Initialize documentation
  docker/prune                        Remove unused images and all stopped containers
  docker/remove-containers            Remove all Docker containers
  docker/remove-images                Remove all Docker images
  docker/remove-volumes               Remove all Docker volumes
  git/config/init                     Initialize git configuration for project
  github/actions/init                 Initialize .github/actions directory
  github/issues/init                  Initialize .github/issues directory
  github/pull-request/init            Initialize .github/pull-request directory
  github/workflows/init               Initialize .github/workflows directory
  gitignore/init                      Create .gitignore file
  gitignore/install                   Install gitignore
  gitignore/list                      List all gitignore templates
  go/install                          Install Golang
  gomplate/version                    Display Gomplate version
  go/version                          Display Go version
  habits/check                        Performs checks
  habits/init                         Initialize gitignore, documentation, pre-commit, github workflows, issues and pull-request
  habits/install                      Install Habits dependencies
  habits/remove                       Uninstall Habits
  habits/update                       Update Habits
  help/clean                          Help screen
  nodejs/install                      Install NodeJS
  npm/install                         Install NPM
  pre-commit/hooks/install            Install pre-commit hooks
  pre-commit/init                     Initialize .pre-commit-config.yaml to working directoy
  pre-commit/install                  Install pre-commit using Pip3
  pre-commit/remove                   Remove .pre-commit-config.yaml
  pre-commit/run                      Execute pre-commit hooks on all files
  pre-commit/update                   Update pre-commit-config.yaml with the latest version
  pre-commit/version                  Display pre-commit version
  python/install                      Install Python 3
  python/pip/install                  Install Python 3 Pip
  python/version                      Display Python & Pip version
  python/virtualenv/init              Initialize a Python 3 virtualenv in the current directory
  python/virtualenv/install           Install Python 3 virtualenv
  python/virtualenv/remove            Remove Python 3 virtualenv in the current directory
  terraform/apply                     Builds or changes infrastructure according to Terraform configuration files in DIR
  terraform/clean                     Remove temporary files and directories
  terraform/destroy                   Destroy Terraform-managed infrastructure.
  terraform-docs/build                Build doc/terraform-docs.md with Terraform Docs
  terraform-docs/version              Display Terraform Docs version
  terraform/fmt                       Check if the input is formatted. Exit status will be 0 if all input is properly formatted and non-zero otherwise.
  terraform/init/backend              Initialize a new or existing Terraform working directory by creating initial files, loading any remote state, downloading modules, etc.
  terraform/init                      Initialize a new or existing Terraform working directory by creating initial files, loading any remote state, downloading modules, etc.
  terraform/install                   Install Terraform latest version
  terraform/plan                      Generates an execution plan for Terraform
  terraform/validate                  Validate the configuration files in a directory, referring only to the configuration and not accessing any remote services such as remote state, provider APIs, etc.
  terraform/version                   Display Terraform version
  terrascan/run                       Run Terrascan
  terrascan/version                   Display Terrascan version
  tflint/init/force                   Init AWS TFLINT, overwrite current configuration
  tflint/init                         Init AWS TFLINT
  tflint/run                          Run TFLINT
  tflint/version                      Display TFLINT version
  tfsec/run                           Run TFSEC
  tfsec/version                       Display TFSEC version
  tfswitch/run                        Execute tfswitch
  tfswitch/version                    Display tfswitch version
  ubuntu/install                      Install most common packages
  ubuntu/update                       Update and upgrade Ubuntu packages

5. DevBox & Taskfile

5.1. Devbox Packages

Apply	Package	Description	Version
[x]	awscli2	Official AWS CLI version 2	latest
[x]	envsubst	Substitute environment variables in shell format strings	latest
[x]	gh	GitHub command-line interface	latest
[x]	go	Go programming language	latest
[x]	go-task	Task runner / simpler Make alternative written in Go	latest
[x]	google-cloud-sdk	SDK for Google Cloud Platform	latest
[x]	gum	Tool for rich shell scripts with minimal code	latest
[x]	jq	Command-line JSON processor	latest
[x]	k9s	Kubernetes CLI to manage clusters	latest
[x]	kind	Kubernetes IN Docker - local clusters using Docker container nodes	latest
[ ]	kluctl	Deployment tool for Kubernetes using a declarative approach	latest
[ ]	ko	Build and deploy Go applications on Kubernetes	latest
[x]	kubectl	Kubernetes command-line tool	latest
[x]	kubectx	Switch between Kubernetes contexts and namespaces	latest
[x]	kubernetes-helm	Kubernetes package manager (Helm)	latest
[x]	kustomize	Customize Kubernetes YAML configurations	latest
[x]	oras	OCI Registry As Storage CLI	latest
[x]	nodejs_20	Node.js JavaScript runtime	latest
[ ]	poetry	Python dependency management and packaging tool	latest
[x]	python312	Python programming language version 3.12	latest
[x]	tilt	Control microservices locally during development	latest
[x]	yq or yq-go	YAML processor (like jq but for YAML files)	latest
[x]	act	Run GitHub Actions locally	latest
[x]	kubent	Kubernetes deprecated APIs checker	latest
[x]	yarn	Fast, reliable, and secure dependency management for JavaScript	latest
[x]	pulumictl	CLI for Pulumi infrastructure as code platform	latest
[x]	go	Go programming language	1.23.*
[ ]	dotnet-sdk	Microsoft .NET SDK	6.0.*
[ ]	gradle_7	Build automation tool	7.6
[x]	curl	Command-line tool for transferring data with URLs	8

5.2. Taskfile

6. AWS-Terraform Best Practices

6.1. Terraform Behavior

aws2-wrap: CAUTION that doing terraform with correct profile, which points to the correct account and assume-role.

$ aws2-wrap --profile Cloudteam-Admin-123456789012 --exec "terraform plan"
$ cat ~/.aws/config

[profile Cloudteam-Admin-123456789012]
sso_start_url = https://d-1234567.awsapps.com/start#/
sso_region = ap-southeast-2
sso_account_id = 123456789012
sso_role_name = Cloudandplatformteam-Admin

6.2. Terraform State file

Organization's accounts have a centralized bucket to keep its state.

terraform {
  backend "s3" {
    bucket  = "ams-terraform-org-state"
    key     = "terraform-aws/<account_number>/*/terraform.tfstate"
    region  = "ap-southeast-2"
    encrypt = true
  }
}

ams-terraform-state               <--- S3 bucket in account 335083429030
└── terraform-aws                    <--- prefix for terraform-aws, in case there are others, like terraform-pagerduty, etc
    ├── management                 <--- account management
    │   ├── organization             <--- subdirectory of account management
    │   │   ├── terraform.tfstate
    │   ├── policy
    │   │   └── terraform.tfstate
    │   └── sso
    │       └── terraform.tfstate
    └── 335083429030                <--- account 335083429030
     ...                            <--- other aws account

The bucket is at account aws-admin (management) S3 bucket - ams-terraform-org-state. The prefix is terraform-aws/<account_name>. current only Cloudteam-Admin has permission to write to this bucket.

6.3. Other files

• terraform.tf -- this is where the remote state storage is configured.
• variables.tf -- where input variables are declared possibly with defaults.
• main.tf -- where the actual terraform resources are configured.
• outputs.tf -- where output variables are declared.
• locals.tf -- specific variables only applicable to a particular environment (prod or dev).

6.4. Terraform

• Checkout the repository and change directory into the account
• Run terraform init to setup the remote state from s3, will need aws credentials.
• to get the credential from sso aws sso login --profile=Cloudandplatformteam-Admin-939936386129
• Edit .tf files as needed.
• Run terraform plan to check what will be changed, possibly save a plan file to use in the next step.
• Run terraform apply to actually make the changes, possible use a plan file to limit the actual changes.
• Run terraform show or terraform output to list the useful outputs.

6.5. Terraform Naming

• Use _ (underscore) instead of - (dash) in all: resource names, data source names, variable names, outputs.
• Beware that actual cloud resources have many hidden restrictions in their naming conventions. Some cannot contain dashes, some must be camel cased. These conventions refer to Terraform names themselves.
• Only use lowercase letters and numbers.

Resource and data source arguments

• Do not repeat resource type in resource name (not partially, nor completely):

  • Good: resource "aws_route_table" "public" {}
  • Bad: resource "aws_route_table" "public_route_table" {}
  • Bad: resource "aws_route_table" "public_aws_route_table" {}

• Resource name should be named this if there is no more descriptive and general name available, or if resource module creates single resource of this type (eg, there is single resource of type aws_nat_gateway and multiple resources of typeaws_route_table, so aws_nat_gateway should be named this and aws_route_table should have more descriptive names - like private, public, database).

• Always use singular nouns for names.
• Use - inside arguments values and in places where value will be exposed to a human (eg, inside DNS name of RDS instance).
• Include count argument inside resource blocks as the first argument at the top and separate by newline after it. See example.
• Include tags argument, if supported by resource as the last real argument, following by depends_on and lifecycle, if necessary. All of these should be separated by single empty line. See example.
• When using condition in count argument use boolean value, if it makes sense, otherwise use length or other interpolation. See example.
• To make inverted conditions don't introduce another variable unless really necessary, use 1 - boolean value instead. For example, count = "${1 - var.create_public_subnets}"