Company Confuto Systems AS

Case studies


Splunk at Telenor
 Delivering Insights for Continuous Service Improvement




       The Business

Telenor, Norway's largest telecom services provider, believes "growth comes from truly understanding the needs of people to drive relevant change." Considering that this once government-owned, 150-year old organization grew from 15 million mobile subscribers to over 160 million in less than a decade, deeper insight leads to success. Customers rely on Telenor to provide the IT infrastructure that supports their voice, data and content services. And Telenor relies on Splunk to provide the visibility they need to keep their IT infrastructure running at peak performance.


 Limitations

With 160 million customers, thousands of servers and routers, and datacenters located throughout Norway, Europe and Asia, it was impossible for anyone to truly understand the essential operating details of the infrastructure. Communication between far-flung departments was extremely difficult or sometimes
didn’t happen. Some logs were being aggregated, but they
were still difficult to search. Access to single components meant access to everything – a definite security risk. The few people with authorized access faced the impossible task of manually
browsing through more than 100 GB of records a day. No
wonder kernel errors and other issues sporadically slipped by unnoticed.


 The Splunking
The Telenor team uses Splunk for troubleshooting, performance monitoring and
security investigations.

Operations
The operations team generates baseline measurements in Splunk so they can
understand what constitutes normal. They created Splunk alerts to monitor for
error spikes and unfamiliar patterns. This advanced visibility lets them
troubleshoot problems before users notice them or services fail. For example,
the team learned that on average twenty errors occur across all distribution
routers on the IP backbone every fifteen minutes. The day after discovering this,
Splunk detected and alerted on 4000 errors and was used to quickly determine the
root cause.

Security
Once the security team determined the baseline for brute force logins and other
security issues, they used easy-to-compose dashboards to monitor servers and
systems for anomalous activity. By correlating timing and IPs, they now
determine if attacks are coordinated. They also identify vulnerable Web
services.


 Breakthroughs

Affordable Scalability
With Splunk's openness and ability to integrate with Telenor's existing tools,
users continually think up new ways to deploy it. Unlike other appliance-based
solutions Splunk operates on commodity hardware and runs on nearly any operating system, including Windows, Mac, Linux, Unix, AIX or Solaris.

Productivity
Telenor has deployed Splunk in each of its regional datacenters to index data
and support the local staff's searches. They also take advantage of Splunk's
distributed searching capabilities that enable searches across datacenters and
across all Splunk data when needed. Splunk's ability to create ad hoc reports
and dashboards gives them the means to drive efficiency and success.

Responsiveness
Not only can the security and operations teams troubleshoot problems faster than
ever, the understanding gained through Splunk baselines lets Telenor identify a
problem long before it turns into a crisis. These valuable searches are now
saved and run on a schedule providing proactive alerts in front of recurring
issues.

Secure Access
Telenor funnels data to one of three secure Splunk instances. Role-based access
controls ensure users get the access they need without compromising security or
violating customer privacy regulations.

Insights
Over time, the knowledge built into Splunk has enabled the Telenor team to learn more about their IT infrastructure and its potential for the business. Their team is now responding to incidents more proactively and is providing better service as a result.
 
Overview

Industry
  • Telecommunications

Splunk Use Cases
  • IT Operations Management - Server monitoring, Network monitoring

Data Sources
  • Infrastructure logs: Network Switsj, firewall and router logs
  • Server logs (Linux, Windows and Unix)
  • Application logs (Web, email, IPTV)
  • IP backbone logs
  • Storage (RAID controller logs)

The Splunk Moment

When introducing Splunk, the operations team staged a troubleshooting contest.
The Splunk admin was asked to diagnose a particular server problem. With no
prior knowledge of the dataset, the Splunk admin located the error in half the
time of the experts. The rest of the team was converted.



"Today's monitoring tools just tell you when something isn't working. With Splunk, we now proactively manage operations and respond before outage occurs or service erodes."

Henrik Strøm
Security Architect