Tableau Love

With a new release of TabJolt hitting github and several TabJolt-related sessions about to kick off at TC16, I have TabJolt on the brain.

While TabJolt is incredibly easy to use compared to “high end” tools such as LoadRunner, you can still make mistakes. It actually is fairly common for new TabJolters to fall into one or more “newbie traps” . Once you’ve taken a few of the lumps detailed below, you will likely (unnecessarily) freak yourself out. You will lose weight, grow very pale, and your dog will cease to recognize you. Don’t be this guy. 

In the spirit of TabJolt happiness here are some things to keep in mind:

Your workload should be real-world

Your goal is to understand how Tableau will behave at scale. Fine…it should be “your scale”, however. When you test with TabJolt, attempt to recreate a test that you think will be reasonably close to how your users will leverage Tableau:

Don’t test with a single workbook

You need the same range of simple, moderately complex, and “ugly” workbooks that your users will run.  You should use your workbooks, not Tableau Samples.

Don’t test with workbooks you know are slow

You wouldn’t take a test when you’re hung over, so don’t do the same thing to Tableau. All workbooks in your workload should be reasonably fast. If not, remove them from your workload or make them fast before you begin testing them.

Don’t use the wrong Test

It is common for folks to think 100% of their users will interact with each and every viz that gets rendered. This means Tableau Server is doing much more work on a per-user basis. In the real world, many (perhaps most) of your users use a Dashboard to view some KPIs quickly then they go back to work without laying a finger on a filter. I’d recommend you use the TabJolt test which splits your viewing and interacting into 50% / 50% buckets.

Do add think time

Your users do not render a dashboard and then launch a different one 3 milliseconds later. They take time to view what they just rendered. Your TabJolt test should mirror this behavior.

Do know how many concurrent users you need to support

Then, focus your initial efforts on hitting that number.  If you don’t know the number, you’re not doing real world testing. A little bit of guesswork is fine, but you need to have a goal, and that goal should be realistic. FYI- Most customers I talk with tend to overestimate their concurrency numbers. 10% of your user population is generally the top end you should consider. I typically see ranges between 3-6%.

Do prove out your concurrency needs

..but don’t overdo it: Are you really going to have all 500 of your users hit Tableau Server at the same time, ever? No, you’re not. Really. I’ve done this over and over again…and they’re not. You may THINK this will happen first thing in the morning, but it won’t. Trust me. If you want to try this once for kicks anyway, cool! But it shouldn’t be your PRIMARY test.

Response Time is NOT the Be-All and End-All

New TabJolt users nearly always spend too much time worrying about “the number”. For example, you’ll hear someone say “I need to see an average of 10 second response time”. In my opinion, this is a mistake. While response time is clearly important, it should not be your #1 focus. It probably belongs down at around #3 or #4 on your list of “important things”

Do focus on ALL criteria: TPS, Error Rate, Response time

TabJolt’s goal is to help you see how all three of these metrics together expose Tableau Server’s health. Get good at watching how all three metrics change together as you add more users to your system. Over-focusing on a single metric is like ignoring someone’s body language and tone of voice when you ask a question. The words which are spoken as a response give you some context, but the tone of voice, body language, etc. carry much more useful info.

If you choose to over-focus on Response time anyway, at least do it right

So, you’re going to ignore my sage advice? Fine. Be that way. However there are some things you should know. The response time number you’re so worried about includes any think time you add to the test. That’s right. If you add 5 seconds to thinkTimeBetweenTest, then your 3 second response time will look like 8 seconds in the sample workbook and in the TabJolt console. If you leverage thinkTimeBeforeInteraction, then an (unknown fraction) of that time gets added to the overall response time, too.

Since you really should add think time (thinkTimeBetweenTest)  to your test to make it real-world and typically users DO pause and look at viz results before they interact (thinkTimeBeforeInteraction) , you’re essentially in a no-win situation: The response time value reported by TabJolt will generally be wrong  (too high).  I find response time as measured by TabJolt to be useful for comparison purposes between tests only. If you really want to eyeball “actual” render time, I prefer to use the Stats for Load Times viz that is part of Tableau Server’s Site Status page.

Don’t assume response time will stay at the same place throughout your test

Response time will and should go up slowly as you increase the number of virtual users on Tableau. The trick is to watch for the point when it starts going up sharply. Typically, this will be the same point at which the TPS measure starts leveling off as you add more users and error rates starts increasing .

How to Test with TabJolt

Once you know what to test and some of what to focus on, there’s still more…

Do run your tests for at least 5-10 minutes

The default runtime of a TabJolt test (as displayed in help) is 60 seconds. That’s way, way, way too short to get any meaningful information. Just like it would be silly to measure my job performance as I’m rolling out of bed first thing in the morning, you need to give TabJolt time to “ramp up”. Typically, you’ll want to run a test at least 5-10 minutes and pretty much ignore the first and last minute of the test as TabJolt ramps the workload up and back down again. The product team generally runs their tests for at least 60 minutes in order to really exercise the machine and try to “break stuff”.

I repeat: Do Watch TPS, Error Rate, and Response Time together

When we test at Tableau, we know a machine is saturated when TPS no longer continues to rise as more virtual users are added. At this point (which is typically about when the machine hits 80-90% CPU utilization), your response time will start rising rapidly and your may start seeing some errors.  Look for this pattern. When you see it occur, your machine is “done”.

Don’t freak out when you see errors

You can expect to see some errors getting generated from time to time. If you’ve chosen to run slow vizzes (don’t do this!) that take longer than 3 minutes to execute, you will see errors which simply mean “this viz took so long to come back, something must be wrong”. The viz WILL actually complete, but TabJolt will assume it’s broken and ignore it. You can use the requestTimeout setting to modify this behavior.

You will also occasionally see error “spikes” that come and go in a semi-regular pattern. MANY viz requests will fail all the same time, and then things are fine again. This is mostly likely one of your VizQLServer processes cycling and is expected.

That’s it! Hopefully these words of wisdom will be useful and defuse any fear, uncertainly and doubt you feel when using TabJolt for the first time. Test On!