KDDI Improves Observability With Splunk, Reduces Operational Time by Over 90%

In KDDI’s case, unified observability unifies the world.

Japanese telecommunications operator KDDI Corporation strives to run a stable communication infrastructure for building a connected society. This is the realization of the KDDI Vision 2030 — the creation of a society in which anyone can make their dreams a reality, by enhancing the power to connect. To this end, the company is driving growth at full speed for its various business lines.

au PAY, its smartphone payment service, has quickly become a critical social infrastructure since its launch in 2014. By 2023, its global user base reached 33.7 million across 6.26 million locations. To ensure service is available 24/7, au PAY runs critical systems, such as the balance management system which tracks all charging, payments, and transactions.

However, the alarm monitoring and traffic analysis tools that KDDI used required separate analysis of logs and slowed visualization of the service processing status and the response rate of counterpart systems. The staff had to retrieve information from each commercial server and work out an Excel or a PowerPoint report manually, which could easily take a whole day. KDDI needed a solution to unify and improve observability across operations, so it chose Splunk.

Boosting observability with optimized visualization and prediction

KDDI’s balance management system looks after the processing performance of au PAY and is connected to around 30 counterpart systems. After deploying Splunk Enterprise in the main and disaster recovery sites of its cloud infrastructure, KDDI can easily aggregate all data in one central platform. Moreover, KDDI has created 70 types of Splunk dashboards to visualize the health of the log management tool Apache Tomcat, with various daily and monthly reports to monitor usage trends of each outlet, and review the batch operation status.

Splunk Universal Forwarder captures logs once every minute and allows KDDI to trim and format the content. This reduces data size from dozens of GB to a few hundred MB, reducing license fees and smoothing the traffic flow. Splunk’s Machine Learning Toolk it helps detect anomalies and forecast demand for special offers with more than 80% accuracy at major participating outlets.

“Splunk offers an environment that makes it easy for us to deploy its products, including thorough support in the initial implementation phase,” says Ken Saito, Deputy General Manager of DX System Department 1, Information Systems Division, Technology Sector of KDDI Corporation. “We have created one good-looking Splunk dashboard after another, each taking just an hour or so. There’s also a lot of information about Splunk online, and all the details we need are available on Splunk’s website.”

Reducing time in routine operations by more than 90%

Splunk dashboards allow KDDI to customize the format in creative ways, such as color-coding errors and displaying KPIs numerically for more informed decision making. Developers can present information and calculations in a simple way and adjust dashboards created by other colleagues to improve the design. The use of Splunk dashboards has also streamlined routine tasks for KDDI, cutting lead time by more than 90%. To KDDI, Splunk Enterprise is not only a “bridge” that connects the operation and development teams, but also a “third eye” on top of its existing monitoring tools, which improves operational resilience with detailed analysis and trend prediction through log visualization.

KDDI can easily search for information just by writing a few lines of the Splunk Search Processing Language. Splunk Enterprise will respond with hard data that enables KDDI to get to the heart of the issue quickly. “It’s critical that we present data to the business departments in an easy-to-understand way and in real time,” Saito emphasized. “Splunk makes this happen.”

Saito is also impressed that Splunk nurtures the entrepreneurial spirit of the company. “With Splunk, we get the opportunities to interact with those outside the company and spark inspiration internally,” he explains. “Every small success together with the sense of achievement fosters motivation among the team, and Splunk is one of the elements triggering this virtuous cycle.”

From firefighting to fireproofing with proactive IT

Moving forward, KDDI is planning to implement more innovative ideas with Splunk to benefit different operations such as incident investigation, operation management, alarm monitoring, and traffic analysis. While KDDI is currently using Splunk dashboards mainly for IT system monitoring, it’s seeking to extend real-time data analytics capabilities to business users. It is great to see that the IT department can contribute to the business’s success and drive the transformation to a proactive information system.

Saito also spoke of combining artificial intelligence with Splunk. “This would allow us to build an environment for trying new things,” he says. “The challenge is to create actual scenarios for this application, and we will continuously discuss with Splunk as we move forward.”