A trusted but slow reporting system

A homegrown reporting system supports work across the business and produces hundreds of reports each day. Built up over decades, it had become notoriously slow. Users had largely stopped complaining because they assumed the delays were unavoidable.

Replacing it would have been expensive and risky. The system contained 61 report types, each with its own database queries, business logic and presentation. It also lacked tests and documentation, and no one inside the business was prepared to lead a replacement. As one user put it, "each report is like a small application." Despite the delays, users trusted the information it produced.

Initial assessment

The operator wanted faster reports and asked whether to improve the existing system or replace it. After completing a broader systems audit, we examined the timeline of each reporting request.

Logs and metrics showed that each report request took about 14 seconds on average, a figure no one had measured before. During a representative month, users generated 31,792 reports across 53 of the system's 61 report types. Their combined wait was about 126 hours. The three most popular reports accounted for more than half the volume.

Next, we timed each stage of a report request. We separated the launch of the orchestrator, the reporting application's startup, the work unique to each report and the final file handling. One stage dominated: starting the reporting application took 10–14 seconds before report generation could begin. Even a quick report paid the same fixed cost.

The reporting core is an application that has grown with each new report. Its 10–14-second startup cost now exceeds the generation time of the most frequently run reports. The reports themselves were not the bottleneck in most cases.

Removing the startup delay

With the operator, we calculated that removing the startup delay could save users about 100 hours a month. That was enough to justify testing several approaches.

Instead of starting the reporting core as part of each request, the orchestrator keeps a configurable pool of initialized cores ready in the background.

Reusing a core for several requests failed validation because global variables were not always reset between reports. The orchestrator therefore discards each core after one request and replaces it with another initialized instance. This preserves the application's trusted output without changing its untested report logic.

Before release, we ran a representative sample of reports through the original and optimized orchestrators. We found no differences in their generated output.

Impact

Production monitoring showed that average report turnaround fell from roughly 15 seconds to about 3.5 seconds, making reports about four times as fast. At current volume, the measured improvement eliminates more than 100 hours of aggregate report wait time each month.

The improvement required no rewrite and no changes to the 61 reports. Users' workflows did not change.

A practical approach to legacy systems

A replacement may still make sense, but it was not the only way to improve the system. Measuring its performance revealed a smaller change that delivered immediate gains without altering reports or workflows.

The change preserved a trusted source of information and gave the operator time to decide when and how to replace it. The audit separated an urgent performance problem from the longer-term decision about the system's future.