Extract process optimization


Extract process optimization

Description

2017 till now Oracle, SolarWinds, Dedicated Team

Optimization of the client’s general ledger extract process, which works with large datasets from different sources.

Client

The US national leader in customized insurance, claims and patient safety & risk solutions for physicians, surgeons, dentists and other healthcare professionals, as well as hospitals, senior care and other healthcare facilities.

Challenge

The company faced serious problem with their system performance – when the extract process worked in parallel for 8 different sources of data, it captured almost 100% of CPU and took more than 8 hours to complete.

The task was to halve the CPU usage and make the extract process run in less than 1 hour, while the process is working in parallel for 8 different sources of data.

Solution

Having analyzed all steps of the extract process, we found out that the main bottleneck which slowed the system’s performance was the faulty logic implemented to determine the release date of data.

We implemented pre-process to calculate the release date for all sources before the extract process begins. As a result, the extract process was sufficiently speed up: when extracting from all 8 sources in parallel, the execution time became less than 20 minutes with less than 30% CPU usage.


Technologies:

  • Oracle SQL
  • SolarWinds Data Performance Analyzer

Some detailed information not disclosed due to NDA restrictions