How to Transition from Testing to Data Engineering

import pandas as pd

# O*NET 31.0: technologies reported in each occupation (Hot Technology = frequently in job postings) and task statements
tech = pd.read_csv("onet/Software Skills.txt", sep="\t")
tasks = pd.read_csv("onet/Task Statements.txt", sep="\t")
QA, DW = "15-1253.00", "15-1243.01"          # Software QA Analysts and Testers -> Data Warehousing Specialists

qa_tools = set(tech.loc[tech["O*NET-SOC Code"].eq(QA), "Workplace Example"])
dw = tech[tech["O*NET-SOC Code"].eq(DW)]
dw_tools = set(dw["Workplace Example"])
hot = set(dw.loc[dw["Hot Technology"].eq("Y"), "Workplace Example"])
print(f"G1 technologies O*NET lists: testers {len(qa_tools)}, data warehousing specialists {len(dw_tools)}, in both {len(qa_tools & dw_tools)}")
print(f"   warehousing hot technologies also listed for testers: {len(hot & qa_tools)} of {len(hot)}")
print("   not listed for testers: " + ", ".join(sorted(hot - qa_tools)))

print("\nG2 data warehousing tasks: which build on testing work?")
testing_words = ("verify", "test", "troubleshoot", "quality", "accuracy", "document", "error")
dw_tasks = tasks[tasks["O*NET-SOC Code"].eq(DW)]
for task_type, group in dw_tasks.groupby("Task Type"):
    for task in group.Task:
        tag = "builds on testing" if any(w in task.lower() for w in testing_words) else "new"
        print(f"   [{task_type:<12}] [{tag:<17}] {task}")
share = dw_tasks.Task.str.lower().apply(lambda t: any(w in t for w in testing_words)).mean()
print(f"   {share:.0%} of {len(dw_tasks)} tasks build on testing work")

G1 technologies O*NET lists: testers 429, data warehousing specialists 159, in both 116
   warehousing hot technologies also listed for testers: 62 of 64
   not listed for testers: Informatica software, Microsoft Power BI

G2 data warehousing tasks: which build on testing work?
   [Core        ] [new              ] Develop data warehouse process models, including sourcing, loading, transformation, and extraction.
   [Core        ] [builds on testing] Verify the structure, accuracy, or quality of warehouse data.
   [Core        ] [new              ] Map data between source systems, data warehouses, and data marts.
   [Core        ] [new              ] Develop and implement data extraction procedures from other systems, such as administration, billing, or claims.
   [Core        ] [new              ] Design and implement warehouse database structures.
   [Core        ] [new              ] Develop or maintain standards, such as organization, structure, or nomenclature, for the design of data warehouse elements, such as data architectures, models, tools, and databases.
   [Core        ] [builds on testing] Provide or coordinate troubleshooting support for data warehouses.
   [Core        ] [new              ] Write new programs or modify existing programs to meet customer requirements, using current programming languages and technologies.
   [Core        ] [new              ] Design, implement, or operate comprehensive data warehouse systems to balance optimization of data access with batch loading and resource utilization factors, according to customer requirements.
   [Core        ] [new              ] Perform system analysis, data analysis or programming, using a variety of computer languages and procedures.
   [Core        ] [builds on testing] Create supporting documentation, such as metadata and diagrams of entity relationships, business processes, and process flow.
   [Core        ] [new              ] Create or implement metadata processes and frameworks.
   [Core        ] [builds on testing] Review designs, codes, test plans, or documentation to ensure quality.
   [Core        ] [builds on testing] Create plans, test files, and scripts for data warehouse testing, ranging from unit to integration testing.
   [Core        ] [new              ] Select methods, techniques, or criteria for data warehousing evaluative procedures.
   [Core        ] [new              ] Implement business rules via stored procedures, middleware, or other technologies.
   [Core        ] [builds on testing] Prepare functional or technical documentation for data warehouses.
   [Core        ] [builds on testing] Test software systems or applications for software enhancements or new products.
   39% of 18 tasks build on testing work