Importance of Non Functional Testing in Data Warehouse

Hi All

In this post I would like to share my knowledge in Non Functional Testing in Data warehouse testing.

There are different types non-functional testing that we do in testing world, some of them is

1. Baseline testing
2. Compatibility testing
3. Compliance testing
4. Documentation testing
5. Endurance testing
6. Load testing
7. Localization testing and Internationalization testing
8. Performance testing
9. Recovery testing
10. Resilience testing
11. Security testing
12. Scalability testing
13. Stress testing
14. Usability testing
15. Volume testing

To me Non Functional testing is something like which will not give any business values; It’s something like dealing with the environment. When we extract the data from heterogeneous source system, we might need to think of handling

Verifying the volume of the data

Any business can’t ensure what could be the volume of the data that they will send. They can say approximately, Our Code should have the capability of pulling the maximum number of data that they source system can send at any point of the time. To manage the Volume of the data, Teradata has the feature called M-Load and T-Pump. When developers designs the system they fix a limit by which data will be loaded into Warehouse.

Example:

• M-Load – If we get a data file with 100 records then the records will be loaded by M-Load functionality
• T-Pump – If we get a data file with less than 100 records then the records will be loaded by T-Pump

What we need to test here is, send a file with 100 records and check records are loaded by M-Load. This can be verified using the Load Job Names.

Verifying Date and Time of the Data file arrival to the Unix Landing directory

Most of the Companies will not function on Week Ends, Public Holidays so our source systems will not send any transactional data on those days. Because of the phenomenon developers will design their jobs to archive any files coming on these days.

Normally, Monday’s transactional data will come to us for loading on Tuesday early morning and it will end on Fridays transactional data will hit us on Saturday early morning.

We as testers need to verify these schedules are working as per the specification. This can be achieved

• sending a file on Week End and check this file is archived
• Sending a file on Public Holiday and check this file is archived
• Verifying Mondays transactional data received on Tuesday morning until on Saturday morning

Verifying Purging and Truncate Loads

I have already mentioned about Purging and Truncate loads in my earlier blogs.

Purging –  The AutoSys jobs will Purge the data leaving the required data in staging table. Suppose if I have loaded 10^th,11^th ,12^th of January data into staging table and when I load 13^th of January data, the 10^th of January data will be purged.

Truncate –  Simple load day_01 data and when you load day_02 data  they Day_01 data will be deleted

We as testers need to verify the Truncate and Purging is happening as per design requirement.

Verifying File Watcher Script

There will be File Watched Script that will look for files until it arrives the Unix Landing directory. Source system is promising us that they will send Day_01 file on 10-01-2013. So we have set the Date in File watcher Script. Source System sent the records on 10-01-2013 , now our File watcher Script will look the date from the file header, if both are matching then it will process the file into Staging table. Source system failed to send the data on 11-01-2013, our file watcher job will look for the file on 11-01-2013 for given time interval if its not arrived then automated Email will be sent to the concern source system saying the file is not arrived

So we as testers needs to verify the File watched job is working as expected.

Cheers – Asik.

Filed under: Data Warehouse Testing - Learners Guide Tagged: Business intelligence, Data warehouse, Database, Extract transform load, Extracts, Non-functional testing, Staging tables, Systems development life-cycle, Test plan