Use Custom Charts in CockroachDB to Troubleshoot Issues Faster

I was working on a follow-up to load balancer comparison post.  This required a series of tests and examining numerous time-series graphs.  Initially, I spent more than the expected amount of time correlating the disparate graphs manually.  This post will describe CockroachDB Admin UI Custom Charts I ended up using. The steps should take just a couple of minutes to follow and should yield tremendous productivity.

As a background, CockroachDB Admin UI has prearranged grouping of times series graphs.  These dashboards are on separate pages and as a result, the graphs I needed were spread across many different pages. This caused some inefficiencies.  Flipping back and forth the dashboards and choosing the right criteria from pulldown menus each time was time-consuming. Remembering the info from different dashboards became difficult.   In the end, it was hard to draw conclusions from the various tests.

What would be better is to have the relevant graphs in one place correlated with each other. This way, it is easy to correlate graphs.  Easy to compare expected vs actual results. Easy to describe and validate the results.  Luckily, CockroachDB Admin UI provides functionality to customize graphs. This feature is called Customized Charts. I will describe simple steps to:

  • Find the available times series data
  • Create customized charts
  • Use the customized charts
  • Share and reuse the customized charts

Step 1: Find the available times series metrics

CockroachDB Admin UI Overview pages list over 200 time-series metrics that can be used in a Customized Chart.  For for the load balancer follow up post, I need to examine:

  • CPUs utilization is in the normal range to ensure correct results
  • Total number of connections match the test scenario
  • Connections are spread evenly among the CockroachDB nodes
  • TPS increases with the increased concurrency

The following are the Metrics Names of interest for this post.  These metrics can also be viewed on the standard CockroachDB Admin UI dashboards:  Overview Dashboard, Hardware Dashboard, and SQL Dashboard as a reference.

  • CPU information will be provided by sys.cpu.combined.percent-normalized
  • Connection information will be provided by sql.conns
  • TPS information will be provided by sql.select.count and sql.update.count

Step 2: Create Customized Charts

Follow the documentation to land at Custom Chart debug page in the Admin UI and follow the steps below.  The number below (1 thru 4) represents the area on the graphs.

  1. Click Add Chart four times, for each custom graph.  
  2. Pick the METRIC NAME from the pulldown menu for the first graph
  3. Pick the METRIC NAME  from the pulldown menu for the second graph.  Note how the same metrics can be used multiple times
  4. Choose PER NODE

custome chart.png

Step 3: Use Customized Charts

As the metrics are added, the following charts will be displayed in real-time.  The below example chart shows a 10-minute sample. The graphs are manually annotated for illustration purposes showing 4 test scenarios (app, JDBC, HAProxy, app w/ 1 thread) and the four criteria defined from Step 1: Find the available times series metrics.  With this customized chart, the 16 different outputs can be compared easily.

custom chart.png
Step 4: Share and Reuse the Customized Charts

To share and reuse the dashboard, save the URL of the customized chart.  The definition of the chart is encoded in the URL. Below is the encoded URL of the above chart.  Simply change the address highlighted in the Admin UI URL. (ie: change localhost:26258 to the target IP:Port)

http://localhost:26258/#/debug/chart?charts=…

Summary

There we have it. We showed it was easy to create a customized chart. We consolidated 3 dashboards down to a single chart. Used the chart to quickly validate a test. We saved the customized chart definition for share and reuse.

This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s