Skip to main content

What is ETL system testing?


This week let’s talk about what is ETL System testing, and what exactly do we test when we perform this type of test?
But as usual let’s start from the very beginning and understand what ETL type of system is.

What Is ETL System?

ETL – Extract Transfer Load.
This type of a system has a special structure that is usually used in case when we want to either load or extract data in big amounts that cannot be contained in a regular API. In other words we are talking about a components that their main purpose is to move around big amounts of data and manipulate it along the way based on specific logic. ETL system implementations are different; involving different data sources on different platforms; each of those systems can be broken down to a set of interfaces that work together to move the data, but despite the differences we can commonly group them based on the direction the data flows at from data source perspective into two groups:
  •         Inbound interfaces
  •        Outbound interfaces

It is important to understand that ETL system can include very complex logic and calculation that are happening during the data movement process, which is both brings the value and the complexity to such systems and makes the testing effort especially challenging.

What Do We Test in ETL System?

When we talk about ETL testing we talk about testing a process that moves data around, either extracting data from a data source or loading data into data source.
While the specifics of what is being tested is changing based on the system and implementation we can still group all the test in 3 main areas:
  •         Verifying the source data structure and content (file, database etc) before load/extract
  •         Verifying the target data structure and content (file, database etc) after load/extract
  •         Verifying the data manipulation logic during load/extract


 What tools we use?

This type of systems are lacking in proper automation tools that can be used to automate the testing process beyond a dedicated Unit testing code created for specifics of each implementation and hence a lot of testers find them self using half automated half manuals methods to verify. The most common is the use of SQL tools for data verification (I.e SQL developer,  SQL management studio) and powerful file manipulation tools (i.e notepad++, beyond compare etc) 

Comments

Popular posts from this blog

7 Most Popular Test Types in Software Testing

Today we are going to return back to basics of software testing and discuss the 7 most popular test types that are being used in every software testing effort. Those different test types cover all the levels of the software to make sure that the final result matching the expectations from every possible angle. Here is our list: Unit testingSmoke testingRegression testingFunctional testingIntegration testingUser Acceptance TestingPerformance Testing


Now let’s have a deeper dive into each one of those by using a simple example of an imaginary system that was created in order to manage warehouse activity including shipments, inventory and goods receptions from suppliers.
Unit Testing This type of testing is usually performed by the developers and is covering the very basic development component. In this test developers are testing the straight forward functionality of a functional piece of code to make sure that it is performing according to their expectations. In our example: This test w…

Story Points estimation for Scrum with Fibonacci vs Shirt Sizes vs Linear - 7 minute guide

It is all began long time ago when Development Teams were constantly asked to provide estimate and they were having a hard time to properly face the task. Let's admit it, there are so many things that can change, happen, and simply go wrong during the development process that one can hardly expect a proper estimation of hours for each task. That why a relative estimation with Story Points came along.

Story Points Estimation
Its a different way to estimate the effort of the Scrum Development Team with-in Agile methodology, which means that instead of estimating hours of work the team estimates each effort relatively to other efforts in the project.
Let's assume that a developer knows that specific 'Task 1' is much harder than another 'Task 2' it is hard for him/her to quantify that harder feeling in hours of additional work but it is possible to say that it much more work. This situation is being address by Story Points when each story point is representing som…

5 Steps to Install Robot Framework for Web Testing Automation - 7 minute Guide

Lately, I found myself helping several people to begin their automation testing journey. And what a better place to do it then a nice new installation of a Robot Framework with its Selenium2Library on a Windows OS.

Why Robot Framework with Selenium2Library? While you are probably here because you are looking for a quick installation guide for Automation Testing platform for you web project, I still need to do my do diligence and explain in couple of words why we need it.
Robot Framework it's a Python framework that allows for a less development oriented Manual QA professionals to do a quick switch into Test Automation. This can be achieved by allowing for easy English phrases (e.g. Click Element, Wait, etc) to replace a more complex Python syntax.
Selenium2Library it's additional add on on top of Robot Framework that allows easy interaction with web elements during the automation process.

That's it..... DONE with the talking now the installation....

Installation process Ro…