Imagine that a critical incident — a major outage, cyberattack or disaster — occurs out of nowhere in your company. In such a case, you'll try to minimize the damage and get back to normal operations as quickly as possible.
But how will you do that? You've no idea how to manage such incidents.
This is where incident commanders come in.
They're trained professionals who lead the response to critical incidents. They have the skills and experience to assess the situation quickly, coordinate the response efforts, and make critical decisions under pressure.
Splunk IT Service Intelligence (ITSI) is an AIOps, analytics and IT management solution that helps teams predict incidents before they impact customers.
Using AI and machine learning, ITSI correlates data collected from monitoring sources and delivers a single live view of relevant IT and business services, reducing alert noise and proactively preventing outages.
An incident commander (IC) is a person who's responsible for managing all aspects of incident response. This can range from assessing the situation and developing a plan of action to reaching a successful outcome.
Here's how incident commanders handle critical incidents:
There may be situations when your Plan A doesn't work. So, ICs always keep a backup plan to anticipate unexpected changes like these that may occur during the investigation. This secondary plan outlines different scenarios and how the organization can respond, depending on the situation.
Unofficially, ICs can work with many people across an organization, including SMEs. Since SMEs are experts in their field and have the knowledge and experience to identify the issue's root causes. That's why ICs listen to their input to understand the bigger picture.
More formally, ICs might work in tandem with an incident command system.
An incident command system (ICS) is a standardized approach to managing emergency incidents. It provides a clear chain of command to ensure that all responding agencies and personnel are working together in a coordinated manner to achieve a common goal.
Here are the 5 significant teams within an Incident Command System:
(See how Splunk solutions support the entire incident management practice.)
An unexpected cyberattack can occur in any organization at any time. In such situations, organizations rush toward an Incident Commander. Because they're the ones who can take charge of incidents, assess the risks and coordinate the response.
Without an Incident Commander, incident management teams may work in silos, duplicating each other's efforts or missing critical details.
Instead, the IC guides the commanding teams to work in a coordinated manner, avoiding duplication of work and ensuring that everyone is on the same page. They analyze the situation and determine what actions have been taken and what needs to be done next.
To adjust the response plan, ICs also keep track of what has worked and what hasn't.
(Related reading: Incident Severity Levels 1-5 & Top Incident Response Metrics.)
As the Incident Commander, you have a critical role in coordinating the response effort and leading your team to success. Here are all the duties and responsibilities that it takes to manage a crisis.
As an IC, your primary responsibility is to collect all the information relating to the incident and share it with the team members. For this, you've to be exceptionally good at one thing — communication.
Effective communication ensures you share the right information with team members at the right time. Sometimes, the incident is not easy, so you may also have to train the team members.
When training incident team members, don't forget to give them a generic action plan. Since unexpected incidents can occur at any time, that's why it's always an excellent option to have a general plan that team members can use to move further until you find the right strategy for them to implement.
Incident assessment is a primary response when it comes to understanding the cause of the incident. To carry out an assessment, ICs look into different aspects to test and identify why the incident occurred and what were the weak points.
Once the incident assessment is done, it's time to make crucial decisions to resolve the incident. At this stage, an IC starts working on the incident management process.
Here are some crucial decisions that an Incident Commander makes:
After deciding on essential things, it's time to implement decisions. Incident Commanders create an incident-specific action plan. Here's how you can make it too:
This helps ensure that the most critical tasks are completed first and the flow is maintained.
And let me break one common myth here. As an IC, your job is not limited to creating this plan. You have to stay engaged to actually guide incident command teams about how to follow this structure to reach the end.
After ICs create an incident command plan, they start assigning tasks to the team per expertise levels. There may be tasks that require specific skills and knowledge. So, they pick the right people to put in the right effort in such situations.
For more complex incidents, the IC takes charge and creates multiple teams to tackle different aspects of the incident. By doing this, the process of addressing the problem is sped up.
ICs frequently oversee the progress of the work with a bird's eye view to stay up to date with what's happening. And they also serve as a facilitator for the teams and members to maintain a working flow.
They also communicate with team members and ask for regular updates and reports to ensure they have all the necessary information. It allows the Incident Commander to provide feedback and guidance to the team as needed.
Panic and chaos are common during incidents. In such situations, people can't think clearly and tend to make the worst decisions.
So as an Incident Commander, you should create a calm environment to help people stay focused and produce quality work. Here's how an incident commander creates a calm environment:
After the team has brought the incident to its end, an IC reviews the results. They go through post-mortem reports thoroughly to evaluate the team's performance and determine if the objectives of the incident were met.
Incident Commanders also arrange a post-mortem meeting where they discuss everything about the incident, from why it occurred, what was done to resolve it, and what the team can learn from it. This helps them prepare for handling similar incidents in the future.
OK. So you want to become an IC, great! Becoming an incident commander is a complex process that requires the right combination of knowledge, skills and experience.
Here are the skills you need as an Incident Commander to ace your job:
As an Incident Commander, you must be a problem solver. You need to know how to identify and solve complex problems in high-pressure situations. Side by side, you will also need to think critically and creatively to come up with practical solutions.
Effective communication is vital when it comes to managing an incident. You should be a top-notch communicator to deliver your ideas clearly and concisely to your team.
An Incident Commander makes critical decisions that could impact the safety and well-being of others. They weigh the pros and cons of different options and make confident decisions.
It's not just about talking — ICs are also fantastic listeners. They seek out and understand different perspectives to use that information and make better decisions.
No two incidents are the same. As an Incident Commander, you must adapt to changing circumstances quickly and effectively.
As an Incident Commander, you have to lead a team of responders. You must inspire and motivate the team — while also providing clear guidance and direction to illustrate your leadership.
Time is of the essence when it comes to managing an incident. That's why ICs prioritize tasks and make the most of every minute.
While it's not strictly necessary, having experience dealing with similar incidents is a significant advantage. It’ll help you anticipate potential problems and come up with effective solutions more quickly.
As an Incident Commander, you ensure your organization's safety and success during an incident. While incidents are unpredictable, there are a few best practices to help you navigate them effectively. So, let's dive into them.
The world of incident management is constantly evolving, so staying up-to-date with the latest best practices and trends is essential. A great way to do that is to attend conferences and network with other professionals in the field.
(Related reading: Top Security Conferences To Attend)
Effective incident management starts with a solid plan. An IC should take the time to develop and document an incident response plan, including clear roles and responsibilities for each team member.
As an Incident Commander, you work closely with your organization and IT teams. So, you should understand each team member's different roles and responsibilities, and the overall structure of your organization.
During an incident, it's easy to get overwhelmed by the sheer volume of tasks an IC needs to complete. To stay focused, you should prioritize tasks based on their urgency and importance and break them into smaller, more manageable chunks. This helps in speeding up the work without getting exhausted.
In high-pressure situations, ICs remain calm and focused. This helps them make better decisions and inspires confidence in the team and stakeholders.
Once the incident is over, it's good to conduct a thorough postmortem analysis to identify areas for improvement. So, an IC must take the time to document lessons learned and make any necessary changes to team or incident response plans for the future.
Incident Commanders manage critical incidents, such as cyberattacks or disasters. They lead the response efforts, make critical decisions under pressure and coordinate the response teams.
Effective communication, assessment, action plan creation, delegation and coordination are the key duties of an IC. Without an IC, incident management teams may miss critical details. But an IC ensures everyone is on the same page and that the incident response is efficient and effective.
See an error or have a suggestion? Please let us know by emailing ssg-blogs@splunk.com.
This posting does not necessarily represent Splunk's position, strategies or opinion.
The Splunk platform removes the barriers between data and action, empowering observability, IT and security teams to ensure their organizations are secure, resilient and innovative.
Founded in 2003, Splunk is a global company — with over 7,500 employees, Splunkers have received over 1,020 patents to date and availability in 21 regions around the world — and offers an open, extensible data platform that supports shared data across any environment so that all teams in an organization can get end-to-end visibility, with context, for every interaction and business process. Build a strong data foundation with Splunk.