Continuous Dashboarding - Your DevOps Airbag

name: inverse
class: center, middle, inverse
layout: true
<div class="header"><span>@unixbigot — Christopher Biggs — Continuous Dashboarding</div>
<div class="footer"><span>Linux.conf.au Sysadmin Miniconf Jan 2017</span></div>
---
layout: true
<div class="header"><span>@unixbigot — Christopher Biggs — Continuous Dashboarding</div>
<div class="footer"><span>Linux.conf.au Sysadmin Miniconf Jan 2017</span></div>
---
class: center, middle
template: inverse

# Continuous Dashboarding - Your DevOps Airbag

.green[Building and shipping Agile dashboards using your existing data silos.]

![](node-red-title-flow.png)

.bottom.right[
Christopher Biggs
<br>.blue[@unixbigot]
]

???
Gday everyone.

Today I want to talk about things that I've
learned over the last dozen years working to
keep web-commerce and software-as-a-service businesses
alive, with varying degrees of success.

I want to tell you about why I think you should
treat your business intelligence systems as seriously
as your customer-facing products, and I'm going to
cover some techniques and tools to help you do this.

---
.left-column[
# Agenda
]
.right-column[
### **Strategy**:
* .red[**Who**] are your dashboards for?
* .orange[**What**] to monitor
* .yellow[**When**] to build your dashboards

### **Tactics**:
* .green[**How**] to develop dashboards
* .blue[**Where**] to fit dashboards into your DevOps pipeline
* .purple[**With**] some suggested tools 
]

???

First, I'll speak about why its important to consider that
different users of data have unique needs, and
I'll go into what kinds of things you can
do with your data beyond just projecting it on the wall.

Next I want to stress why it's valuable to have
the same people who write code thinking about data, and
working on both at the same time.

After this we'll get more practical and look
at how to apply DevOps principles to dashboards and
data.

---
class: center, middle
template: inverse

# Who?

## Dashboards and Business Intelligence (BI) are .red[Product]

### Think about who your .green[customer] is

???

First I want to consider the different consumers
of data, and how you need to serve them according to
their needs and inclinations.

---
.left-column[
#Strategy
## Who
]
.right-column.autodim[
## Who are your dashboards for?

#### Dashboards are traditionally targeted at ops.
 * Measure **all** the things!
 * Wake me if it's on **fire**
 * Little feedback goes back **upstream**

.img-800w[![Munin](munin-boring.png)]
]

???
Traditionally dashboards have been for operations staff,
they've been highly technical and detailed, measuring
a huge number of things but at a very low level,
and often just not very pleasant to look at.

Monitoring and alerting has focused on servers and services,
rather than the application domain.

The other thing I've noticed is that operations
people tend to learn a lot about the way that
users and systems behave, but don't often have
an opportunity to pass that back up the development
pipeline.

---
.left-column[
#Strategy
## Who
]
.right-column.autodim[
## Who are your dashboards for?

#### Traditionally targeted at ops, but
#### DevOps is a thing.
 * Development is a different **point of view**
 * **Instrument** your applications
 * Measure **trends** and **levels** in your KPIs
 * Look for **out-of-range** metrics
 * What does normal **look** like?
 * Regression-test your **bottom line**
]

???

So, the first group of people who who aren't being
served well are developers.

Disk io-ops per second is not the most important thing
to a developer.  What they really need to know is whether
their application is working.  When I say application I'm
abbreviating the term "applications software", that is, software
that *does* something.   They key question here is did I, as a
developer, achieve what I set out to do.   Does my feature
work, is my user interface working the way I planned,
have I positively or negatively affected the experience
of using this product, did I advance the business plan.

Businesses make money by providing services to customers,
you need to understand what are your true services and
points of interaction, understand what normal looks like
and ensure that when something that's *not* normal happens,
the right people  know about it.

Startups learn all the time that their understanding
of what their users wanted was all wrong, the key
here is to learn this while you're still in business.

---

.left-column[
#Strategy
## Who
]
.right-column.autodim[
## Who are your dashboards for?

#### Traditionally targeted at ops, but
#### DevOps is a thing
#### Support and Sales need to know everything's OK, too
 * Is the first time you know of problems when support escalates?
 * Do support call ops and ask "Is everything OK, we're getting a lot
of calls"?
 * "Well Mrs Arbuckle, I see other customers are having problems too"
]

???
Your customer facing staff have data needs too.

Often the first time a business learns that normal has
left the building is when the support phones start ringing.
I've found that for every person that calls there's 99 who
just went somewhere else, so if you let your customers
tell you about faults that's far too late.

Wouldn't it be better if your support staff can know the
probable cause of a customer issue before the customer
has to explain it?

---
.left-column[
#Strategy
## Who
]
.right-column.autodim[
## Who are your dashboards for?

#### Traditionally targeted at ops, but
#### DevOps is a thing
#### Support and Sales need to know everything's OK too
#### Management want reassurance (and to know you deliver)
 * The morning report for "How long till my next **Porsche**"
 * Shipped a great new feature?  You now have a **hotline** to the CEOs eyeballs.
]

???
Dashboards might be the only regular communication you have with
upper management.

It's a cliche that every executive loves a big fat
meaningless pie chart for breakfast, but if you
give your management a steady diet of interesting
and varied intelligence, then you have a way to
draw a direct connection between your existence and the bottom
line.

Your upper management probably don't see the day to day business
of development, but this is your chance to show the sprint-to-sprint
value added by your team.

---
.left-column[
#Strategy
## Who
]
.right-column.autodim[
## Who are your dashboards for?

#### Traditionally targeted at ops, but
#### DevOps is a thing
#### Support and Sales need to know everything's OK too
#### Management want reassurance (and to know you deliver)
#### Customers (do you dare?)
 * What are you **ashamed** of them seeing?
 * What could you **do** about that?

]
???
Okay, now this one might be scary.

Would you show your customers your performance data,
or your error rates?

Why not, what are you ashamed of?  Just asking
yourself that question might teach you something
valuable.

---
.left-column[
#Strategy
## Who
]
.right-column.autodim[
## Give **customers** a weather report

#### Take notice of load spikes
 * "We are processing your order.   This hour that
takes around 25 seconds but could be up to two minutes"
]

???

Let's look this another way.  Data lets you soothe
your customers.   Every piece of software has a bad
day, so if things aren't going well, and you know
about it, you can let the customers down gently,
give them your sympathy along with realistic
expectations.

People appreciate sincerity.

---
.left-column[
#Strategy
## Who
]
.right-column.autodim[
## Give **customers** a weather report

#### Take notice of load spikes
#### Respond to temporary outages
 * "Sorry, it looks like PayPal has been experiencing high error rates
 for the last 15 minutes.  May we suggest you try AliPay or Credit
Card?"
   * This could head off a flood of support calls or bad reviews
]
???
If you work with banks you know they're a bit like
elephants, inscrutable, difficult to win an argument with, and a source
of mind boggling amounts of excrement when you least
expect it.

If your third party interfaces are on the fritz you can
respond to that, head off frustration before it happens.

---
.left-column[
#Strategy
## Who
]
.right-column.autodim[
## Give **customers** a weather report

#### Take notice of load spikes
#### Respond to temporary outages
#### Assure customers that errors are being acted on
 * "We're sorry, this page experienced a problem.  Bug #3342 has
already been forwarded to our Dev Team.   You'll get an email when
it's fixed, or check our .fakelink[change log]"
   * again, turn your failures into goodwill
]
???
Finally, why do customers have to call to report errors.
You know the error happened, you threw an exception.

Why don't you turn it around, tell the customer you're
on it, and give them a way to follow up.   You could
even promise to notify them as soon as the fault is
rectified.

This could prevent them taking their money elsewhere.

---
.left-column[
#Strategy
## Who
]
.right-column.autodim[
## What's relevant to this **support** call?

#### Notice recent errors affecting this customer
 * "Mr Lee encountered .fakelink[Bug #3342] six minutes ago"

]
???
Lets look at how data can help your support team.

If you've instrumented your application, and you have a
the ability to query that data, then you can have the ability
to reconstruct what that particular user did.

This turns the act of supporting a customer from palm reading
to brain scanning.

---
.left-column[
#Strategy
## Who
]
.right-column.autodim[
## What's relevant to this **support** call?

#### Notice recent errors affecting this customer
#### Notice recent outages affecting this customer
 * "Currently experiencing elevated transaction declines for Westpac
cards"

]
???
If there is a cluster of failures in your system thats
important information for your support staff and customers
to know.

---
.left-column[
#Strategy
## Who
]
.right-column.autodim[
## What's relevant to this **support** call?

#### Notice recent errors affecting this customer
#### Notice recent outages affecting this customer
#### Remember recent behaviour of this customer
 * "Dr Patel had her password reset yesterday, and presented her old
 password three times today"

]
???
Finally, sometimes customers are not reliable witnesses.

There's a whole class of common software problems that
we can fix if we store more data, even if only for a few
hours.  Don't be parsimonious in recording analytic data,
record as much as you can, and if storage becomes a problem thin it
out later.

---
class: center, middle
template: inverse

# What?

## Not just **things**, also .green[patterns] and .red[trends]

???
All right, lets look at what kinds of data you should be shepherding.
---
.left-column[
#Strategy
## Who
## What
]
.right-column.autodim[
## What should I be monitoring?

#### Traditionally - **Things**
  * system load indicators
  * rates of key events (pageloads, signups, checkouts etc.)
  * service status
  * error alerts ("pages")
]
???
Going back again to the traditional operations monitoring,
what most people record is the vital statistics of their
servers.   A bunch of statistics that you have to understand
intimately to even know if there is a problem or not.

Nowadays we have elastic servers and multi-tier architectures.
A simple graph of operations per second implicitly assumes that
resources are fixed, which is rarely true any more.