YAML 2 Terraform Generator

Table of contents

1. Description
2. Info
3. Repositories
4. Requirements
4.1 Python
4.2 Packages
5. Documentation
5.1 Arguments
5.1.1 Examples
5.2 Jenkins Pipeline
5.3 Data Structure of YAML Files
5.3.1 Supported Metrics
5.4 Jinja Templates
5.5 Terraform Output Folder Structure

1. Description

This repository contains a first draft script that can use a specific YAML file datastructure in order to generate Terraform configuration files. Afterwards it is used to be part of a Jenkins pipeline.

2. Info

- The current version of this script is able to generate Terraform files that support the metrics mentioned within section Supported Metrics. Therefore not all use cases might be covered!
- YAML files should be stored according to the business line (DE-3, DE-4, DE-7, EC-DE) within the respective business line folder!
- Some YAML files have already been created for testing purposes based on the shared_configuration and coco_terraform_config repositories and were therefore added in this repository.

⚠️ Currently YAML filenames starting with the prefix TP_ are supported!

3. Repositories

The following repositores have been used to create some YAML files.

• shared_configuration
• coco_terraform_config

4. Requirements

4.1 Python

The used python version for running the script is 3.11.2.

4.2 Packages

Some python packages need to be installed. Please use the requirements.txt file to install missing packages. The command for installing the missing packages is:

pip3 install --user -r requirements.txt

5. Documentation

The following section is just a small exhibit tailored to the use of the script.

5.1 Arguments

If the script needs to be used for local execution, then the following command should be used:

.\generate.py <path to yaml file/folder> <path to output folder>

Please note that two arguments are necessary:

• <path to yaml folder> - Here it is necessary to provide the path to the folder where the yaml files are stored. Fore example, if the yaml files are stored within the DE-3 business line folder, then it is important to convey the correct path as an argument for the script, in this case .\DE-3\. It is also possible to provide multiple folders, but they have to be delimited by a comma when parsing, in this case .\DE-3\,.\DE-4\,.\DE-7\!
• <optional:path to output folder> - If it is necessary to store the generated Terraform files to a specific location, then the correct path to that particular folder needs to be provided! Alternatively, the default path .\output\ will be used which is within the current working directory of the script.

5.1.1 Examples

Here are some examples when executing the script:

# The YAML Files are stored within the business line folder DE-3, in the same working directory as the script, and by the following command it is possible to generate the Terraform files and store them into the default output folder.
.\generate.py .\DE-3\

# The YAML Files are stored within the business line folder DE-3, in the same working directory as the script, and by the following command it is possible to generate the Terraform files and store them to a folder at a different location.
.\generate.py .\DE-3\ ..\coco_terraform_config\

# The YAML Files are stored within the business line folder DE-3, outside the working directory of the script, and by the following command it is possible to generate the Terraform files and store them to a folder at a different location.
.\generate.py ..\shared_configuration\DE-3\ ..\coco_terraform_config\

# The YAML Files are stored within multiple business line folders (DE-3, DE-4, DE-7), in the same working directory as the script, and by the following command it is possible to generate the Terraform files and store them into the default output folder.
.\generate.py .\DE-3\,.\DE-4\,.\DE-7\

# The YAML Files are stored within multiple business line folders (DE-3, DE-4, DE-7), in the same working directory as the script, and by the following command it is possible to generate the Terraform files and store them to a folder at a different location.
.\generate.py .\DE-3\,.\DE-4\,.\DE-7\ ..\coco_terraform_config\

# The YAML Files are stored within multiple business line folders and outside the working directory of the script. By using the following command it is possible to generate the Terraform files and store them to a folder at a different location.
.\generate.py ..\shared_configuration\DE-3\,..\shared_configuration\DE-4\,..\shared_configuration\DE-7\ ..\coco_terraform_config\
# Note here: If all yaml files of the shared_configuration repository need to be provided as an input argument it would also suffice to use the following command:
.\generate.py ..\shared_configuration\ ..\coco_terraform_config\

5.2 Jenkins Pipeline

This is the link to the pipeline: Pipeline CoCo SLO YAML 2 TF. There are basically five main stages:

• Fetch Repositories:
  • shared_configuration
  • coco_terraform_config
  • coco_apm_slo_yaml2tf
• Checkout New Branch: A new branch is checked out (with a timestamp) which will be needed before any changes can be made after the script is run. The branch name will be something like -automatedSLOGeneration.
• Install Required Python Packages: It is necessary to install the required packages before the script can be run!
• Execute Script: The script is executed and the YAML files with the corresponding business lines
• Push Changes: Once the script finished with its exection the changes will be pushed with a commit that just includes a timestamp. The changes will be pushed to the newly created branch as mentioned above!

5.3 Data Structure of YAML Files

The majority of the key and value pairs have been taken respectively from the shared_configuration repository.

⚠️ Please make sure that the YAML File does not include any tabs otherwise the correct functionality cannot be assured!

slo_id: # <type:string>            - Order of the given SLO within the dashboard
slo_name: # <type:string>          - Touchpoint Mobile or Vehicle
displayname: # <type:string>       - Display name of the SLO to be displayed on the dashboard
department: # <type:string>        - Department/Business Line, responsible for this SLO
description: # <type:string>       - Description Field: CoCo QM-Report_Mobile or CoCo QM-Report_Vehicle
doc_url: # <type:string>           - Link to the documenation of the particular SLO
slo_definition:
  tresholds:
    warning: # <type:integer>      - Traffic light orange
    failure: # <type:integer>      - Traffic light red
  EMEA-Prod:
  - metric: # <type:string>        - Used metric, see section Supported Metrics
    filter:
    - service: # <type:string>     - Name of the Service
      keyRequests:
      - # <type:string>            - The name of the particular Key-Request of that Service
      - # <type:string>            - ...
    - service: # <type:string>     - ...
      keyRequests:
      - # <type:string>
  NA-Prod:
  - metric: # <type:string>
    filter:
    - service: # <type:string>
      keyRequests: # <type:string>
      - # <type:string>
      - # <type:string>
    - service: # <type:string>
      keyRequests:
      - # <type:string>
  CN-Prod:
  - metric: # <type:string>
    filter:
    - service: # <type:string>
      keyRequests: # <type:string>
      - # <type:string>
      - # <type:string>
    - service: # <type:string>
      keyRequests: 
      - # <type:string>

5.3.1 Supported Metrics

Currently the following metrics are supported:

builtin:service.keyRequest.errors.fivexx.rate
builtin:service.keyRequest.errors.server.successCount
builtin:service.keyRequest.errors.fivexx.count
builtin:service.errors.server.successCount
builtin:service.errors.fivexx.successCount

5.4 Jinja Templates

Currently there are only five Jinja Templates setup to cover mainly five different types of metrics as mentioned before. They can be found within the jinja-templates folder.

├───jinja-templates
│       terraform-template-1.j2
│       terraform-template-2.j2
│       terraform-template-3.j2
│       terraform-template-4.j2
│       terraform-template-5.j2

• terraform-template-1.j2 is used for builtin:service.keyRequest.errors.fivexx.rate
• terraform-template-2.j2 is used for builtin:service.keyRequest.errors.server.successCount
• terraform-template-3.j2 is used for builtin:service.keyRequest.errors.fivexx.count
• terraform-template-4.j2 is used for builtin:service.errors.server.successCount
• terraform-template-5.j2 is used for builtin:service.errors.fivexx.successCount

5.5 Terraform Output Folder Structure

The output folder structure will look like the following example:

C:.s
├───CN_Prod
│   └───slo
│           TP_Mobile_DigitalKey.tf
│           TP_Mobile_Login.tf
│           TP_Mobile_Mapping.tf
│           TP_Mobile_PersonalFavorites.tf
│           TP_Mobile_Remote360.tf
│           TP_Mobile_RemoteServices.tf
│           TP_Mobile_Send2VehicleLegacy.tf
│           TP_Mobile_Send2VehicleMGU.tf
│           TP_Mobile_VehicleData.tf
│           TP_Mobile_VehicleList.tf
│           TP_Vehicle_FTS.tf
│           TP_Vehicle_RTTI.tf
│
├───EMEA_Prod
│   └───slo
│           TP_Mobile_DigitalKey.tf
│           TP_Mobile_Login.tf
│           TP_Mobile_Mapping.tf
│           TP_Mobile_PersonalFavorites.tf
│           TP_Mobile_Remote360.tf
│           TP_Mobile_RemoteServices.tf
│           TP_Mobile_Send2VehicleLegacy.tf
│           TP_Mobile_Send2VehicleMGU.tf
│           TP_Mobile_VehicleData.tf
│           TP_Mobile_VehicleList.tf
│           TP_Vehicle_eRoute.tf
│           TP_Vehicle_FTS.tf
│           TP_Vehicle_RTTI.tf
│
└───NA_Prod
    └───slo
            TP_Mobile_DigitalKey.tf
            TP_Mobile_Login.tf
            TP_Mobile_Mapping.tf
            TP_Mobile_PersonalFavorites.tf
            TP_Mobile_Remote360.tf
            TP_Mobile_RemoteServices.tf
            TP_Mobile_Send2VehicleLegacy.tf
            TP_Mobile_Send2VehicleMGU.tf
            TP_Mobile_VehicleData.tf
            TP_Mobile_VehicleList.tf
            TP_Vehicle_eRoute.tf
            TP_Vehicle_FTS.tf
            TP_Vehicle_RTTI.tf