Question: how do I know that one user is waiting for a lock and another user is burning CPU on MySQL?
i.e. how can I create active session history on MySQL like on Oracle?
Two simple cases I’m looking at
- session waiting on lock
- The wait event is ” wait/io/table/sql/handler” which happens for a number of potential reasons
- -> I can join to information_schema views INNODB_LOCK_WAITS & INNODB_TRX but this is heavy, and I’d rather only access those after I know there is a lock waiter.
- session burning CPU
- How do I know from performance schema views that the connection is active and burning CPU?
- It has a wait event that shows up wait/synch/mutex/sql/THD::
LOCK_thd_data - on Oracle to determine CPU, we just look at connections that are active and not waiting.
Also what is NESTING_EVENT_TYPE? I see the same thread sometime twice in events_waits_current with NESTING_EVENT_TYPE STAGE and WAIT.
On some systems for the lock test below the NESTING_EVENT_TYPE shows up as NULL and others as STAGE.
I basically want to monitor user activity and collect who is active and if they are waiting for something or if they are on CPU.
Row Lock
First user
start transaction;
update toto set id=1;
Second user does
update toto set id=4; -- hangs
How do I see second user is waiting for a row lock? Here is output from
- performance_schema.threads
- performance_schema.events_
waits_current
For a connection waiting on a lock:
select * from performance_schema.threads where thread_id=5329\GTHREAD_ID: 5329NAME: thread/sql/one_connectionTYPE: FOREGROUNDPROCESSLIST_ID: 5223PROCESSLIST_USER: rootPROCESSLIST_HOST: localhostPROCESSLIST_DB: kylePROCESSLIST_COMMAND: QueryPROCESSLIST_TIME: 782PROCESSLIST_STATE: updatingPROCESSLIST_INFO: update toto set id=4PARENT_THREAD_ID: NULLROLE: NULLINSTRUMENTED: YESHISTORY: YESCONNECTION_TYPE: SSL/TLSTHREAD_OS_ID: 2088RESOURCE_GROUP: USR_defaultselect * from performance_schema.events_waits_current where thread_id=5329\G *************************** 1. row ***************************THREAD_ID: 5329EVENT_ID: 740239187END_EVENT_ID: NULLEVENT_NAME: wait/io/table/sql/handlerSOURCE: handler.cc:2991TIMER_START: 45606173945401364TIMER_END: 46413477812683064TIMER_WAIT: 807303867281700SPINS: NULLOBJECT_SCHEMA: kyleOBJECT_NAME: totoINDEX_NAME: NULLOBJECT_TYPE: TABLEOBJECT_INSTANCE_BEGIN: 139872327141152NESTING_EVENT_ID: 740239186NESTING_EVENT_TYPE: STAGEOPERATION: fetchNUMBER_OF_BYTES: 61
CPU
Have user running a procedure that just burns CPU running “call minute_rollup(100000000)”. Here is output from
- performance_schema.threads
- performance_schema.events_
waits_current
select * from performance_schema.threads where thread_id= 6273 \G*************************** 1. row ***************************THREAD_ID: 6273NAME: thread/sql/one_connectionTYPE: FOREGROUNDPROCESSLIST_ID: 6167PROCESSLIST_USER: rootPROCESSLIST_HOST: localhostPROCESSLIST_DB: kylePROCESSLIST_COMMAND: QueryPROCESSLIST_TIME: 668PROCESSLIST_STATE: Opening tablesPROCESSLIST_INFO: call minute_rollup(100000000)PARENT_THREAD_ID: NULLROLE: NULLINSTRUMENTED: YESHISTORY: YESCONNECTION_TYPE: SSL/TLSTHREAD_OS_ID: 2087RESOURCE_GROUP: USR_defaultselect * from performance_schema.events_waits_current where thread_id=6273\G THREAD_ID: 6273EVENT_ID: 403685579END_EVENT_ID: 403685579EVENT_NAME: wait/synch/mutex/sql/THD::LOCK_thd_data SOURCE: sql_class.h:4142TIMER_START: 46300974764007954TIMER_END: 46300974764070510TIMER_WAIT: 62556SPINS: NULLOBJECT_SCHEMA: NULLOBJECT_NAME: NULLINDEX_NAME: NULLOBJECT_TYPE: NULLOBJECT_INSTANCE_BEGIN: 139871988613512NESTING_EVENT_ID: 403685578NESTING_EVENT_TYPE: STATEMENTOPERATION: lockNUMBER_OF_BYTES: NULLFLAGS: NULL
Procedure to burn CPU:
DELIMITER $$ DROP PROCEDURE IF EXISTS minute_rollup$$ CREATE PROCEDURE minute_rollup(input_number INT) BEGIN DECLARE counter int; DECLARE out_number float; set counter=0; WHILE counter <= input_number DO SET out_number=SQRT(rand()); SET counter = counter + 1; END WHILE; END$$ DELIMITER ; call minute_rollup(100000000);
I think the trick to see if the connection is on CPU is performance_schema.events_waits_current.EVENT_ID= END_EVENT_ID then we are on CPU
select
t.thread_id,
IF(w.EVENT_ID=w.END_EVENT_ID,'CPU', w.EVENT_NAME) AS wait_event,
REPLACE( REPLACE(substr(t.PROCESSLIST_info ,1,80), '\r', '') , '\n','') sql_text
from
performance_schema.threads t
left join performance_schema.events_waits_current w on w.thread_id = t.thread_id
where
t.PROCESSLIST_STATE IS NOT NULL
and ( t.PROCESSLIST_COMMAND != 'Sleep' and t.PROCESSLIST_COMMAND != 'Daemon')
and t.PROCESSLIST_ID != CONNECTION_ID();
+-----------+-------------------------------------------+----------------------------------------------------------------------------------+
| thread_id | wait_event | sql_text |
+-----------+-------------------------------------------+----------------------------------------------------------------------------------+
| 518002247 | CPU | SELECT anon_1.anon_2_anon_3_customer_id_ AS anon_1_anon_2_anon_3_customer_id_, a |
| 518007382 | CPU | COMMIT |
| 518006817 | wait/io/table/sql/handler | SELECT anon_1.anon_2_customer_id_ AS anon_1_anon_2_customer_id_, anon_1.anon_2_c |
| 518007025 | wait/io/table/sql/handler | update customer set name=name where id_ in (2, 1); |
| 518007027 | wait/synch/cond/sql/Item_func_sleep::cond | select sleep(20) |
| 518007028 | wait/io/table/sql/handler | update customer set name=name where id_ in (8, 4); |
| 518007029 | wait/io/table/sql/handler | update customer set name=name where id_ in (6, 3); |
| 518007032 | wait/synch/cond/sql/Item_func_sleep::cond | select sleep(20) |
| 518007034 | wait/io/table/sql/handler | SELECT anon_1.anon_2_customer_id_ AS anon_1_anon_2_customer_id_, anon_1.anon_2_c |
| 518007035 | wait/synch/cond/sql/Item_func_sleep::cond | select sleep(20) |
| 518007039 | wait/io/table/sql/handler | update customer set name=name where id_ in (5, 3); |
| 518005220 | wait/io/table/sql/handler | SELECT anon_1.customer_id_ AS anon_1_customer_id_, anon_1.customer_name AS anon_ |
| 518007058 | wait/io/table/sql/handler | update customer set name=name where id_ in (4, 2); |
| 518004406 | wait/io/table/sql/handler | SELECT anon_1.anon_2_customer_id_ AS anon_1_anon_2_customer_id_, anon_1.anon_2_c |
| 518003240 | wait/io/table/sql/handler | update toto set id=4 |
+-----------+-------------------------------------------+----------------------------------------------------------------------------------+
One unusual thing is that the thread can show up more than once in events_waits_current.
select
t.thread_id,
w.NESTING_EVENT_TYPE,
IF(w.EVENT_ID=w.END_EVENT_ID,'CPU', w.EVENT_NAME) AS wait_event,
REPLACE( REPLACE(substr(t.PROCESSLIST_info ,1,80), '\r', '') , '\n','') sql_text
from
performance_schema.threads t
left join performance_schema.events_waits_current w on w.thread_id = t.thread_id
where
t.PROCESSLIST_STATE IS NOT NULL
and ( t.PROCESSLIST_COMMAND != 'Sleep' and t.PROCESSLIST_COMMAND != 'Daemon')
and t.PROCESSLIST_ID != CONNECTION_ID()
+-----------+--------------------+-------------------------------------------+----------------------------------------------------------------------------------+
| thread_id | NESTING_EVENT_TYPE | wait_event | sql_text |
+-----------+--------------------+-------------------------------------------+----------------------------------------------------------------------------------+
| 518010010 | STATEMENT | wait/io/table/sql/handler | SELECT customer.id_ AS customer_id_, customer.name AS customer_name, customer.de |
| 518010010 | WAIT | wait/io/file/sql/hash_join | SELECT customer.id_ AS customer_id_, customer.name AS customer_name, customer.de |
| 518010011 | STATEMENT | wait/io/table/sql/handler | SELECT customer.id_ AS customer_id_, customer.name AS customer_name, customer.de |
| 518010011 | WAIT | wait/io/file/sql/hash_join | SELECT customer.id_ AS customer_id_, customer.name AS customer_name, customer.de |
In the case of NESTING_EVENT_TYPE of type WAIT, I want that WAIT. IF there is no NESTING_EVENT_TYPE of type WAIT and only of type STATEMENT , then I want that wait (AFAICT) leading to a bit of a convoluted statement
select
t.thread_id,
COALESCE( ww.NESTING_EVENT_TYPE, ws.NESTING_EVENT_TYPE) NESTING_EVENT_TYPE,
IF(ws.EVENT_ID=ws.END_EVENT_ID,'CPU',
COALESCE( ww.EVENT_NAME, ws.EVENT_NAME) ) AS wait_event,
REPLACE(REPLACE(substr(t.PROCESSLIST_info,1,80),'\r',''),'\n','') sql_text
from
performance_schema.threads t
left join performance_schema.events_waits_current ww on ww.thread_id = t.thread_id
and ww.NESTING_EVENT_TYPE = 'WAIT'
left join performance_schema.events_waits_current ws on ws.thread_id = t.thread_id
and ws.NESTING_EVENT_TYPE = 'STATEMENT'
where
t.PROCESSLIST_STATE IS NOT NULL
and ( t.PROCESSLIST_COMMAND != 'Sleep' and t.PROCESSLIST_COMMAND != 'Daemon')
and t.PROCESSLIST_ID != CONNECTION_ID();
+-----------+--------------------+-------------------------------------------+----------------------------------------------------------------------------------+ | thread_id | NESTING_EVENT_TYPE | wait_event | sql_text | +-----------+--------------------+-------------------------------------------+----------------------------------------------------------------------------------+ | 518009610 | STATEMENT | wait/io/table/sql/handler | SELECT anon_1.anon_2_anon_3_customer_id_ AS anon_1_anon_2_anon_3_customer_id_, a | | 518010002 | STATEMENT | CPU | SELECT current_schema, sql_text, digest, digest_text, timer_start, @startup_time | | 518003240 | STATEMENT | wait/io/table/sql/handler | update toto set id=4 | | 518010412 | STATEMENT | wait/io/table/sql/handler | SELECT anon_1.anon_2_customer_id_ AS anon_1_anon_2_customer_id_, anon_1.anon_2_c | | 518010010 | WAIT | wait/io/file/sql/hash_join | SELECT customer.id_ AS customer_id_, customer.name AS customer_name, customer.de | | 518010417 | WAIT | wait/io/file/sql/hash_join | SELECT anon_1.customer_id_ AS anon_1_customer_id_, anon_1.customer_name AS anon_ | +-----------+--------------------+-------------------------------------------+----------------------------------------------------------------------------------+
Get the last statements for thread with blocking transactions
select
t.thread_id,
s.EVENT_ID,
REPLACE(REPLACE(substr(s.SQL_TEXT,1,40),'\r',''),'\n','') thread_proc_info_short,
s.TIMER_WAIT/100000000000 time
from
performance_schema.events_transactions_current t
join performance_schema.events_statements_history s on s.NESTING_EVENT_ID = t.EVENT_ID
where t.thread_id in ( select BLOCKING_THREAD_ID from performance_schema.data_lock_waits )
and s.NESTING_EVENT_TYPE='TRANSACTION'
order by t.thread_id, s.EVENT_ID;