Monitor maxQueryThreads

Questions & Answers
1

Hi,

We would like to assess Mondrian’s maximum number of concurrent queries; that is, the maxQueryThreads advanced setting. We use eazyBI for Jira Server with child process. What we would like to know is either when the maximum has been reached or some statistics report on the number of concurrent queries during a period of time. We would appreciate community’s suggestions.

If this study drived us to increase maxQueryThreads, which resource usage statistics should we care of? In our opinion, when Mondrian was working at its maximum load because maxQueryThreads was reached, it would probably also reach a maximum usage of child process JVM memory, Linux CPU and database resources. So we should monitor those resources at the points in time of the maximum concurrent queries; if some happened to be overused we should increase them… or even give up increasing maxQueryThreads in order to avoid a new bottleneck! Which statistics had we better check? We reviewed database statistics and assessed the database was OK. We thought we should have monitored child process JVM memory too, but (lazy us) we had no memory monitoring in place and just searched for OutOfMemory exceptions in the logs.

For the time being, we are plotting the number of concurrent queries by this poor man’s trick. It’s easy to see why we ask: there must be a better way! We assumed that every new query should report its start in the log with a line like this:
Started GET "/eazy/accounts/54/cubes/Issues" for nnn.nnn.nnn.nnn at 2019-05-19 07:03:56 +0000
and its completion with
Completed 200 OK in 23069.0ms (Views: 54.0ms | ActiveRecord: 30.0ms)

So it was a matter of counting starts and ends with this Linux awk:

sudo grep 'INFO: ' eazybi-child.log | \
egrep '\] Completed |\] Started ' | \
awk 'BEGIN{N=0} { if ($6 == "Started" ) N++; else N--; \
  print $1 " " $2 " " N }'

2019-05-27 03:20:27 1
2019-05-27 03:20:27 2
2019-05-27 03:20:27 3
2019-05-27 03:20:37 2
2019-05-27 03:20:38 3
2019-05-27 03:20:38 2
2019-05-27 03:20:51 1
2019-05-27 03:20:51 2
2019-05-27 03:20:52 1
2019-05-27 03:21:04 0
2019-05-27 03:21:04 1
2019-05-27 03:21:05 0
2019-05-27 03:21:36 1
2019-05-27 03:21:36 0

From this report, the maximum was three and was reached twice.

Thanks,
Mariano

2

Hi Mariano,

In eazybi-web.log and eazybi-child.log you see the full stack not only Mondrian queries. Also, the DB information logged in those files or not about Mondrian engine, but only eazyBI. To view what Mondrian is executing and you can enable Mondrian debug logging (use it temporarily, it can impact performance).

Another way to debug performance would be to enable Mondrian request profiling, and then you could see SQL queries generated by each report and maybe find some ways to speed them up.

There is also usage statistics which could help you to find those slow reports.

I think maxQueryThreads should not exceed 2x CPU count, so you should increase it only if you see that not all CPU cores are utilized when eazyBI is working. Everything depends on the report, usually slow report most time is spent after the data is retrieved from the DB - all those custom calculations.

1 Like