05 Oct 2015
This is the final part of a multi-part series on how Fastly architects and
builds our support process. You can read part one, part two, and part three to get prior context.
This article dives into our support team's training processes and explains how our lessons learned can help your support team grow.
Training
Many customers come to Fastly’s support seeking advice on Varnish, an open source software that’s at the core of our CDN services. Because of this, our team has to be experts on all things Varnish. We also have an incredibly technical product and customer base, so we have to be equally knowledgeable. Even for team members with prior CDN or Varnish experience, Fastly’s technology requires training and continuing education on a regular basis to maintain meaningful support interactions. If you're running a support team, large or small, proper new hire training and continuing education are essential to keeping your quality and team morale up.
Because of this, Fastly needed to embrace having a comprehensive training and onboarding process for our new and existing support team members in the early days. Our current training programs are not formally completed (yet), but we have everything from advanced technical talks with our subject matter experts to informal computer science classes covering the absolute basics to side-by-side pairing sessions. Fastly has reached the point where nearly any new hire can be brought up to full productivity within a few weeks.
The training I mentioned above focuses on just the technical aspects of the job, not the softer skills that make up support interactions. We have training processes for that, too. Support is a skill, and it takes practice. From spotting related problems from specific wording in a ticket to being able to effectively convey appropriate emotions in writing, language skills are crucial for support.
Timing and tone
With proper technical and tone training in mind, we’ve discovered something surprising as we’ve scaled: using a relatable tone and human voice are extremely important to proper support, but fast initial response and correctness have a much stronger correlation with customer satisfaction in a support interaction.
In practice, there’s a sliding scale effect. If your team immediately responds to customer tickets and solves customer problems, tone is less crucial because you’ve promptly mitigated risks. But if your team’s time to first response is long (24 hours or more), having an approachable and human tone is perhaps the only thing that can improve your relationship with the customer. In the worst possible scenario — taking an extremely long time to respond — you must have both great technical answers and an impeccable tone. This is the hardest response to achieve reliably. Given the option, encouraging and measuring a fast response time is the course of action that benefits most companies today.
Takeaways
It’s hard to scale an effective support organization in the long term. You have to care about support and the customer experience more than you ever thought possible. You need to build tools and processes around support like you would your flagship product. Most importantly, you have to be willing to change your assumptions as your customer base scales and changes. The needs of your early customers will be very different than the needs of your later customers, but the care and respect you show them will be universal.
There’s not a lot of resources on scaling support available, but there is a great deal of writing on scaling operations, capacity planning, and structuring technical teams. Believe it or not, these may be the best resources for your company’s support needs. There are obviously some areas that don’t overlap; you should never treat your support team like interchangeable machines, for example. However, the process of planning based on business growth is a great mindset to use when architecting your support.
Most importantly, there’s a lot of non-advice about support out there. Telling your support team to “care more” or “engineer happiness” or “close more tickets” isn’t support leadership, as it’s not clear or actionable. Support is a process, like any other facet of building a company. You have to work at it, define your customer process effectively, and be pragmatic about what systems you need to scale. It’s not glamorous, but the sooner you can address your customer’s needs, the faster you can build a support organization that’s actually meaningful and helpful.
Scaling support can be difficult, but it’s not impossible. I shared our history at Fastly to give some insight into what it takes to build a great support infrastructure at your company. Fastly’s process will likely not fit your specific business needs, but hopefully our lessons learned can be of some benefit to you.
09 Feb 2015
This is the third part of a multi-part series on how Fastly architects and
builds our support process. You can read part one and part two to get prior context.
Fastly has made some key decisions about how to operate and structure our support team in a way that allows us to deliver quality at scale. Some of the lessons we learned are more specific to issues we encountered as we grew, but most can be applied to any support or customer-facing team.
Don't repeat yourself
The first support pain point Fastly ran into was answering repeated
questions, as every company might experience. There are two major tactics
that can be used in some combination for solving this problem: documentation and
canned scripts in the ticketing system (or a text expander tool). Adding people to
the team is a solution, too, but that isn’t possible for many companies,
as it's by far the most expensive and least efficient way to address the
problem.
In true ops fashion, Fastly embraced DRY (Don't Repeat Yourself) in full force.
If a question was asked twice, someone built the answer into a document on docs.fastly.com. The team didn't use canned scripts or macros at all. The nature of our questions
didn't make reusable scripts practical -- even today there are not a lot of canned
scripts used due to the nature of our support requests.
This presented a few issues. Sometimes the docs were unorganized, as nearly
every employee contributed, and sometimes the docs overlapped. At the time,
there was no one owning or coordinating documentation efforts.
But, it was a net positive, as it tapped into a support truism: many people
would rather not contact support at all. If you can make it easier for your
customers to self service, you're making it easier for everyone. It also makes
it possible for customers to solve problems themselves, which makes them
feel awesome. And who doesn't like feeling awesome?
It's important to note that when a customer contacts support, they are taking a lot of effort to seek assistance and admit they need outside advice. For some people,
that's a normal process. But for others, it can be a huge ego hit. If you can
make self-service easier, either through product design or documentation, your
customers will have a better experience with your product, which directly shapes
their support experience.
The right tools for the job
Another friction area Fastly encountered early on was the need to
communicate through more than just email or support tickets. Since
everyone in the company (and most of our early customer base) was already in
IRC, it made sense to set up an IRC room for customer communication.
The room #fastly on Freenode was born. It's still there. You can ping me in
there right now, most likely (I'm @aspires).
IRC is an amazing way to communicate with customers in real time to either debug
problems or talk shop. You'll get some of your best feature requests and
product insights from the users who take the time to chat with you. Also, Fastly
has reached a critical mass where customers are helping other customers. That's
amazing, but it's not unique to us. It’s happened at every company I've seen with a maintained IRC room.
IRC helps us scale in another way. When we're working closely with
customers, we can set up rooms for the customer quickly, and get down to work
without a lot of overhead. Customer onboarding for major infrastructure changes
needs to be faster and smoother than email, but not as time-consuming and cumbersome as a conference call. IRC offered an ideal, asynchronous communication tool that only
chat can provide.
We later found that when customers helped other customers in IRC, a lot of shared knowledge wasn't getting logged in a reusable way. So,
we've recently launched a Discourse-powered Community forum to help keep the interesting
use cases and shared knowledge in an indexed location.
We also have a lot of unique workflow-improving tools for our specific
circumstances. Some are built right into our ticketing system, and others live
in a central location outside of the ticketing system. Nearly every support
team has them, so this isn't a huge surprise. But I do recommend building these
as early as possible. A great support toolset, combined with comprehensive
documentation, will scale your support organization more than you think. It's
also fun to blow off some steam hacking on an internal tool that will get real
use.
Another tool we use heavily at Fastly is our pager system. That's right, support
carries a pager. It's a follow-the-sun rotation with our London office, which
eliminates the risk of being woken up. It originally started as a way to track
who would field tickets over the weekends, but we've applied the rotation to
different tasks. For example, we have Premium Support customers with an
emergency trigger email, which pages our support team. It also means that if there's an issue that needs support assistance, our engineering teams have the ability to page us.
A dedicated department
I said earlier that Fastly started out with engineering answering tickets
directly. It was a great process that helped fine-tune our initial product set
in the best way possible. Fastly was able to make this scale further than most. With the techniques and tools previously outlined, Fastly engineering was
able to do their own support for almost two years without a dedicated support
organization.
But, as ticket load increased, and as teams across Fastly began to specialize,
there came a time to build a real support team. Fastly has structured support as
part of a larger organization called Customer Engineering, responsible for the
entire customer onboarding and lifecycle process. This includes sales
engineering, support, documentation, and professional services. Some teams call
this "Customer Success," but the result is the same -- support is factored into
the entire business process, rather than operating as an outside team.
Fastly officially started the customer engineering team in mid-2013 with
support being the primary responsibility, but the team took on sales engineering
work that was previously done by the founders and a few early employees. At the time of writing, Fastly's customer engineering team has nearly 20 people divided among our SF, London, and NYC offices.
Team structure
That leads into team structure — not just how our support organization is
structured, but how we interact with the rest of the company.
Customer engineering is a first-class organization at Fastly that actively works
with all teams to ensure a high-quality customer experience. We have the
resources we need to grow and make decisions for the good of the customer, and
because we're structured to address the entire customer cycle, we can have real
impact on customer happiness and product perception.
The customer engineering organization is also included in the product
development process. We don't dictate what's happening, but we do
get copied on projects as they progress. Because we're looped in regularly,
we're set up for success; things don't ship as a surprise, and when things do
ship, we've done the necessary work to be ready for go-live. Because of this,
we're better prepared to manage inevitable bug reports and customer issues after
ship.
This is critical for good support and happy customers -- simply looping your
support team in on the development process early will make your ships
significantly more successful than pushing features without communication.
Fastly's process for solving tickets with other departments is also unique. Typically, support organizations operate on their own island, where they
either cannot escalate a ticket to product engineering for assistance (the
common startup scenario), or where they escalate a ticket to a magical "next
level," never to see it again (the common big company scenario). Neither of these
is ideal. You're either building a system where customers run into barriers and
red tape, or a system where support pushes the buck to other people.
Fastly intentionally structures its support workflow in the following way for
escalating and interacting with other teams:
- New ticket comes in.
- Support claims the ticket and starts investigating.
- Support comes to a point where another team needs to weigh in.
- Support assigns the ticket to the team's group in the ticketing system (support is still copied on the ticket, as well).
- The assigned team is now responsible for providing an answer, update, or response on to the ticket thread.
- Someone from the assigned team weighs in, possibly responding to the customer directly.
- Support takes the ticket back, and continues solving the issue. At this point, if necessary, go to #4 and repeat.
- Support resolves the issue with the customer.
- Support starts the documentation process as needed.
It's a little more involved than other organizations, but the important steps
here are #5 and #7. When support assigns a ticket to another group, such as ops,
finance, legal, neteng, etc., that team works to get an update as soon as they
reasonably can. Once that team weighs in, it’s support's job to take that, relay
it in a meaningful way to the customer, ultimately resolve the issue, and create
documentation (if appropriate).
Our technical teams have been experimenting with an "on call" rotation for
interfacing with other teams on pressing issues. Fastly just happened to use
pager shifts like this to structure other things, so it made sense to build
support (and other teams) into the shift process for communications. You may
need a different system, but the main point is this: great support teams need to have
a codified way to work with the rest of the company in a way that doesn't
distract or derail daily work.
If support isn't included in the final process of shipping products, and
if your support organization doesn't have a respected way to pull in other teams
to help solve customer problems, you're hindering your efforts to build and
scale the quality of support you offer. The people who end up suffering are
your customers and users.
This piece definitely got more into the trenches of planning, team management,
and the actual process of growing support. So if you have any follow-up
questions, let me know at @austinspires on Twitter.
There’s one post more left in this four-part series, so stay tuned.
27 Jan 2015
Lately, I've realized that if I have certain blog post open as a tab for a long time so that I
continuously revisit it, I should probably write it down. Kottke.org's recent piece
about Werner Herzog's life advice is one of these posts.
The original piece is at http://kottke.org/15/01/24-pieces-of-life-advice-from-werner-herzog and definitely
merits a read. But, the list itself is as follows:
- Always take the initiative.
- There is nothing wrong with spending a night in jail if it means getting the shot you need.
- Send out all your dogs and one might return with prey.
- Never wallow in your troubles; despair must be kept private and brief.
- Learn to live with your mistakes.
- Expand your knowledge and understanding of music and literature, old and modern.
- That roll of unexposed celluloid you have in your hand might be the last in existence, so do something impressive with it.
- There is never an excuse not to finish a film.
- Carry bolt cutters everywhere.
- Thwart institutional cowardice.
- Ask for forgiveness, not permission.
- Take your fate into your own hands.
- Learn to read the inner essence of a landscape.
- Ignite the fire within and explore unknown territory.
- Walk straight ahead, never detour.
- Manoeuvre and mislead, but always deliver.
- Don't be fearful of rejection.
- Develop your own voice.
- Day one is the point of no return.
- A badge of honor is to fail a film theory class.
- Chance is the lifeblood of cinema.
- Guerrilla tactics are best.
- Take revenge if need be.
- Get used to the bear behind you.
As much as I dislike the reblog link trafficking that happens all over the internet, I'll betray my own values
for this piece.
12 Jan 2015
Note: this is the second part of a multi-part series on how Fastly architects and
builds our support process. You can read part one here.
Part 2: What the customer experiences, and how that happens.
What customers encounter when evaluating Fastly, and as they onboard, isn't arbitrary.
It's an experience that we've intentionally crafted. This piece will goes into what
customers experience, and what we have in place to make sure this happens every
time, for every customer.
The Customer Process
Before we go too deep into our support scaling process, lets take a quick look at
what happens when a person or company signs up for Fastly. Inevitably, this shapes
the support need and the service we provide, so it's important to keep this in
mind.
Most customers come to us with a scaling or ops need. They may be looking
for a new CDN vendor, they may have a difficult problem to solve such as scaling
a service API, or they may just be growing exponentially. In any case,
there's an obvious operational pain point that we can address.
We intentionally provide a frictionless signup process so customers can begin
configuring and testing without needing to talk to a human first. We're ready
to assist, and we make sure that new customers feel free to reach out, but we
don't force ourselves into the picture unnecessarily.
This is a core value of Fastly's support: we give you as much (or as little)
assistance as you want, exactly when you determine.
Inevitably, customers do reach out with questions during the setup process. This
is infrastructure: communication is to be expected and encouraged. Either over
some email or phone exchanges we discuss fine tuning
the system for the customer's needs. We're intentionally flexible as a service:
you know your application and needs better than we ever will. But that doesn't
mean that there aren't onboarding questions.
Some customers can configure their service on their own, and move traffic
over quickly and easily. We've put a great deal of work into our documentation
and application design to enable self service. But, there are always cases where
our teams need to assist. In that case, we may spin up an IRC room for our team
and yours too coordinate the switchover, or debug any setup issues in real time.
Sometimes chatting for 15 minutes can save days of frustrating email exchange.
We'd rather get to the bottom of things right away.
At this point some portion of a customer's traffic has been moved over. There
may be a slow ramp over time. We may be serving a niche portion of a customer's
traffic. We may get the 100% of a customer's public traffic in one day. In any
case, some traffic is now flowing through the customer network.
In many cases, this is where the interest and support ends. With Fastly,
it's only the beginning.
Because we're a CDN, we need a connection to your origin server. This gives us
the ability to see service issues or error spikes almost immediately. Our teams can
proactively reach out in the event of a service disruption. We then work with
you to as needed to adjust your CDN or troubleshoot issues.
This ability to proactively assist is possibly the most impressive post of our
support. If you're planning support and you have one takeaway from this piece,
it's to find a problem your users have that you can predict, and proactively
reach out when you see issues arise. A fast response to a customer's ticket is
amazing, but if you can spot a problem before it starts that's even better.
At some point a customer may want to put more traffic on to Fastly, fine tune their
current traffic for a higher hit ratio, build in automated purging or set up
detailed logging from the edge. This is a more traditional, consultative support.
We view ourselves as scalability partners for our customers, and this is an
extremely rewarding part of the job. We push the limits of what can be offloaded,
and it's fun to work with customers from an architectural standpoint to help hit
their goals.
This has basically been the support process for every single Fastly customer from
day one, with very few exceptions. This experience is what we work on scaling and
improving. We've absolutely made some changes to how we prepare and carry this
process out on our end, but there have not been major changes to what aim for
when providing support. You should be able to speak with an early Fastly customer
and a recently ramped Fastly customer, regardless of size large or small, and
those two customers should describe roughly the same experience. That's not by
accident.
The Support Process
So what's happening under the hood in the above section? What steps and processes
do we have in place to make customer setup and traffic ramping go smoothly?
Pull up a chair and we'll go right into the thick of things.
Our process starts when the customer contacts Fastly to investigate the product,
or shortly after the customer begins configuring a service after signup. We then
discuss their end goals over email, phone, IRC, or in person. More often than
not, our documentation base can help them get set up with most configurations.
But, in the off case that our documentation doesn't address everything, we work
directly with the customer over the course of a few days or weeks to build up a
configuration that does what they need.
This is configuration is usually done in our web application, but occasionally
we dig in and write some VCL (Varnish Control Language, the scripting language
that powers our caching system) to get things done. We may help with setting up
logging and analytics, but those are easy to self-provision.
Now we're ready to start testing traffic with the customer. In prep for this we
work with our ops and neteng teams to forecast traffic. As we're now serving a
consistent 1.8 million requests per second worldwide, individual sites and
customers don't cause the network delta they once did, but we still loop in
everyone. When it's time for a customer to finally flip the switch, we have a
group of folks standing by from ops, neteng, product engineering, and support.
Of course, this process always doesn't happen exactly as I describe here. The
process is as unique as their own application, so we help iteratively as needed.
Every few weeks, customers may come back and file a ticket or two, or bump us in
IRC, but the bulk of the work is up front.
The actual coordination of these processes is jointly run by the customer's
account manager and our support team. Our account managers typically will get
the relevant information regarding a turn up, and plan with our support and
engineering teams for a successful traffic switchover. The main takeaway is that
the entire company contributes to the customer success. It's the majority of what
our support team does on a daily basis, but every team at Fastly is partially
involved with and responsible for customer success.
Next Time
The next piece in this series will dive down into the structure of Fastly's support
team, and how the team interacts with the rest of the company. This has been the
most requested topic so far, so I'm excited to publish it. All things equal, this
team and cross-company interaction is possibly the most unique aspect of our
support architecture. Follow me on twitter
to hear when updates are live
05 Jan 2015
Note: What started as a quick blog post about support inspired by this tweet
quickly ballooned into a novella. The initial draft of this "quick blog post"
was enormous, so I've greatly edited down the content, and split the piece
into multiple parts
Part 1: Our Standards
One of Fastly's greatest product advantages is our support infrastructure. We
get compliments and positive feedback nearly every day through tickets, tweets,
and old fashioned conversation. When we run case study interviews with customers,
support is a common factor in why people choose Fastly over other providers.
This is something we take pride in, and something we love to brag about.
This culture of support and service isn't an accident. Our support processes
span the entire company, and are baked into every aspect of the way we do
business. We worked hard to build our support infrastructure (our processes are
so broadly encompassing and involved that calling our support an "infrastructure"
is the only term that feels right) to scale without sacrificing quality,
timeliness, or the soul that got Fastly where it is today.
In the time I've been at Fastly, we've had enough customers and friends ask us about
how we do support, and what's going on under the hood that it seems appropriate
to give a high level overview of how we build, what we've learned, and what other
teams can borrow/steal from our setup.
The Context
When you look at Fastly's support, you have to start with looking at the context
surrounding the company.
Infrastructure vendors historically have bad support experiences, unless
you're spending something huge: think budgets in the tens of millions. Even then, support
is viewed as a cost center -- the vendor's goal with support is for you to go
away. If you going away is the result of them solving a problem, that's a plus.
But the goal is to make you go away, solution or not. If you don't believe
me, talk to your head of ops.
Because we're a company founded by operations engineers, for operations engineers,
having par-for-the-course (ie: bad) support wasn't acceptable. Fastly's founding
team set out to create the support experience they always wanted from a vendor.
The catch is: every startup begins with this goal. I've never met a business owner who
feels like their company should have bad, or even mediocre, support. They all claim
they want the best. That said, we've all seen cases where companies grow, become
successful, and their support inevitably goes down the tubes. That pursuit of
"the best" has been sacrificed, either intentionally or as an unfortunate side
effect of other decisions.
I can't comment on other companies and their circumstances, but I can go into
detail about the culture and processes that Fastly has in place to prevent this.
The Early Days of Yore
In the earliest days of Fastly, the team made some very crucial decisions about
how they would approach support as a company.
From the outset, support was considered a profit center, rather than a cost of
doing business. Fastly theorized that great support would generate more
business than any sales force could in the earliest days, and approached setting up
support with that in mind. It turns out, that theory was right, and customers
flocked.
In the operations world, reputation is everything. Fastly already had respect in
the community through the early team's technical reputation, but their support
reputation became world renowned in quick fashion.
So, what actual decisions were made to set the foundation?
In the early days of Fastly, everyone joined in on support. Engineering, Ops,
Neteng: they all took tickets they could answer, helped as quickly as possible,
and communicated as honestly as they could.
Everyone shouldered the support burden, and took pride in delivering a great
experience. The experience was so valued that people even "competed" to take
tickets first. Great support was baked into the culture. Fastly's early
employees had all run into horrible vendor support in their previous jobs, and
were determined to do the opposite in their time at Fastly.
The people who shipped product answered the tickets about their features, wrote
docs as needed, and made sure customers got the full benefit of Fastly. When
new customers came in, everyone pulled together to make sure the migration went
smoothly. It wasn't farmed out to a specific individual, the customer wasn't
just left to their own devices. Infrastructure migration often needs extra help to
make the process go well, so everyone chipped in. The customer's success was
viewed as Fastly's success.
A buzzphrase in the startup community these days is "do things that don't scale."
Fastly did this with support. It's not scalable to have your entire team onboard
single customers. It's not scalable to have your CEO assist in real time over IRC.
It's not scalable to give your everything to make sure the customer has a great
experience.
The catch is: during those same early days, we did start scaling, and we did
so very successfully. We started building the tools, resources, and everything
else necessary to make sure this quality could be extended indefinitely into the
life of Fastly.
It was said earlier: Fastly is a company by operations engineers, for operations
engineers. Ops scales applications. Ops plans capacity. Fastly scaled and planned
for capacity with support in this same way.
As an aside, many support teams are avoiding the concept of "scaling support"
outright, as "scaling support" often implies cutting corners and removing the
human touch that makes so many organizations successful. This should more
accurately be considered "scaling support poorly." When Fastly scales support, we
make the conscious effort to preserve the most essential facing aspects. If those
aren't preserved, we view the scaling efforts as a failed (but valuable) test
and roll back.
The next part of this piece will go into the decisions we made early on as an
organization, the lessons we learned in the process, and what we've done to expand our quality
of support and customer service. Follow me on twitter
to hear when updates are live!