How can a petrochemical or process manufacturer speed up BI reporting without rebuilding its data warehouse?
A petrochemical manufacturer's BI reporting had become progressively slower as transaction volumes grew. The underlying data platform was still fit for purpose, but every refresh recalculated the full reporting cube, including data that had not changed. Azati isolated that bottleneck and introduced selective recomputation inside the existing pipeline, improving dashboard response time without a warehouse rebuild or architectural change.
Improvement in BI query response time
From diagnosis to production fix
The optimization stayed inside the existing data pipeline
One Azati AI-augmented Senior Data Engineer, three months, one measurable fix. A key plant downtime and operational analytics dashboard re-executed massive SAP-to-PostgreSQL calculations on every refresh. By implementing selective recomputation inside the existing Airflow pipeline, Azati eliminated compute bottlenecks, stabilized CPU load during peak shift changes, and cut dashboard query times down to 1.5–2.5 seconds without modifying the underlying data warehouse architecture.
The engagement was deliberately narrow: the existing architecture was sound, so the goal was to remove the unnecessary computation rather than replace the platform.
Plant management relied on an Equipment Downtime and Utilization Dashboard to track operational availability, maintenance events, and production losses. As transactional volume grew, dashboard refreshes steadily degraded, creating friction for shift supervisors and operations leads running real-time queries throughout the day.
Source operational data flowed from SAP into a PostgreSQL (v14–16) analytical layer orchestrated via Apache Airflow. Every refresh cycle forced a full re-generation of the entire data mart, recalculating static historical maintenance logs alongside live operational parameters.
The affected reporting suite tracks critical plant performance metrics, including technical readiness, equipment utilization, and unplanned downtime financial losses. Because plant managers and maintenance engineers use these dashboards to make real-time operational decisions, delayed refreshes directly impacted response times during unplanned equipment stoppages.
Operational data flowed continuously from SAP into PostgreSQL (v14–16) via Apache Airflow. On every refresh, the ETL engine re-pulled and re-calculated the entire historical dataset alongside live shift parameters. As transactional logs grew quarter over quarter, this redundant recomputation consumed increasing compute cycles, turning a minor database latency into a recurring operational headache.
Not every data problem needs a platform overhaul. If a specific dashboard, report, or pipeline has been getting slower as your data grows, and nobody's had the time to find out exactly why, that's usually a sign of one fixable inefficiency, not a reason to replace the whole platform.
Azati starts with the actual bottleneck: where the time is going, whether the problem can be fixed inside your existing stack, and what the smallest viable intervention looks like. The assessment helps determine whether targeted optimization is enough or whether a larger modernization effort is justified.
Azati joined the client’s internal data team as an embedded engineer. Rather than introducing third-party tools or parallel infrastructure, the fix was engineered entirely within their existing stack: PostgreSQL, Python, and Apache Airflow.
Combining deep petrochemical software development expertise with pipeline diagnostics, Azati audited the caching logic across the SAP PM (Plant Maintenance) and ERP data flows. The analysis revealed that stable historical parameters were unnecessarily recalculated on every cycle.
Azati refactored the transformation logic to separate static historical maintenance logs from volatile shift parameters. Using structured temporary tables inside PostgreSQL, static data was preserved between refreshes, ensuring only modified records underwent recalculation.
The fix required no schema modifications, external libraries, or complex custom frameworks. The result is a clean, production-grade optimization inside Apache Airflow that the client's internal team can maintain effortlessly.
Dashboard query response dropped from 2–3+ seconds to 1.5–2.5 seconds during peak concurrent access, specifically when plant supervisors and maintenance leads run downtime reviews simultaneously at shift handovers.
Shifting from full cube regeneration to selective incremental processing eliminated transient locks on PostgreSQL tables, allowing parallel queries from other operational systems to execute without lag.
Instead of allowing query latency and server compute costs to compound exponentially with every new SAP batch, the pipeline’s resource footprint remains flat regardless of historical transaction growth.
The optimization was deployed directly inside the live Apache Airflow orchestration pipeline without altering the target schema or disrupting daily plant reporting workflows.
When the underlying architecture is still fit for purpose, rebuilding the platform can introduce more cost and delivery risk than it removes. A targeted performance optimization is often the better first step when the bottleneck is isolated: unnecessary recomputation, inefficient queries, refresh logic, caching, or a specific pipeline stage.
The first question should therefore be where the time is actually going, not whether the platform should be replaced.
This engagement is representative of a broader performance-optimization approach for enterprise data and BI environments. Azati can investigate bottlenecks across data-mart refreshes, ETL/ELT pipelines, query execution, caching, orchestration, and data-processing workloads, starting with the existing architecture and changing only what the evidence shows needs to change.
Azati helps industrial companies improve BI and data-platform performance without defaulting to a rebuild by finding the bottleneck, fixing the unnecessary work, and keeping the optimization inside the architecture that already works.
Find a performance bottleneckFull data-mart rebuilds are justified when the underlying architecture can't support current or future requirements. Here, the architecture was sound. The inefficiency was isolated to one recalculation pattern inside the transformation logic. Identifying that distinction, rather than defaulting to a bigger engagement than the problem required, is itself part of the engineering judgment this kind of fix depends on.
BI performance optimization improves the speed and efficiency of an existing BI environment by identifying unnecessary computation, inefficient queries, refresh bottlenecks, caching issues, or other pipeline constraints. It can be preferable to a rebuild when the underlying data architecture remains fit for purpose.
Yes. When the bottleneck is isolated to query logic, refresh processing, caching, or another specific pipeline stage, performance can often be improved within the existing architecture. In this case, Azati replaced full data-mart recomputation with selective recomputation without changing the underlying data structure.
Targeted optimization is appropriate when the existing architecture can support current requirements and the performance problem has an identifiable bottleneck. A broader modernization is more appropriate when the platform itself creates structural limitations that cannot be resolved through focused optimization.
In petrochemical manufacturing, software solutions must process high-volume operational data from plant equipment (MES, SCADA, and SAP PM). Optimizing existing data pipelines ensures real-time reporting availability for plant operators without introducing the risk or multi-million-dollar overhead of replacing enterprise data platforms.
Explore our successful projects and see how Azati delivers measurable results for our clients.
A large petrochemical operator faced high carrying, maintenance, and disposal costs from unclaimed and duplicate inventory across multiple ERP systems using different coding conventions. The same material was often tracked under different codes, making it impossible to see true inventory levels without manual reconciliation.
Azati ran a seven-year dedicated team engagement building and operating an on-premise inventory management platform with AI-assisted nomenclature normalization across disconnected ERP systems, covering backend development, DevOps, and test automation alongside the core product functionality.
A large chemical enterprise required scalable mechanisms to collect, transform, and deliver trusted analytical data across dozens of interconnected business systems. Growing analytical requirements, a complex data ecosystem with hundreds of flows and thousands of tables, and the need for reliable reporting data created ongoing engineering challenges.
Azati embedded specialists into the client's delivery organization, developing analytical pipelines, data marts, and orchestration workflows using SQL, Python, dbt, Apache Airflow, Apache NiFi, Greenplum, PostgreSQL, and ClickHouse. The engagement focused on continuous platform evolution over 24+ months rather than a one-time implementation.
Last updated
